CockroachDB consistency check failed: checksum mismatches and confirmed corruption

When the background consistency checker detects a checksum mismatch between replicas of a range, CockroachDB logs a fatal error and the node terminates. This is not a transient condition. The database is telling you that data integrity is compromised.

The definitive message: “consistency check failed with N inconsistent replicas” or a Pebble-level “checksum mismatch” error. Both originate from CockroachDB’s consistency checker subsystem, not from unrelated log noise. This article covers the failure mechanism, the recovery path, and the signals that help you scope the damage.

What this means

CockroachDB’s consistency checker runs as a background loop on every node. For each range where the node is the Raft leader, it computes a SHA-512 checksum of a snapshot and compares it against follower replicas. The default cycle is once per 24 hours per range, controlled by server.consistency_check.interval. The checking rate is throttled by server.consistency_check.max_rate (default 8.0 MiB/s).

When a mismatch is detected:

1. The node logs a fatal error: “consistency check failed with N inconsistent replicas.”
2. The node terminates. Modern CockroachDB panics on consistency check failure. There is no flag or environment variable to disable this behavior.

Before terminating, the checker runs a second pass with an advanced diff setting. This publishes the leader’s full snapshot to follower replicas so they can log a key-level diff of where the data diverges. That diff output is your primary forensic evidence.

The consistency checker cannot tell you which replica has the corrupt data. It only knows that checksums do not match. The leader might hold the bad data, or a follower might. This asymmetry is what makes recovery difficult.

flowchart TD
    A[Consistency checker runs
on Raft leader for each range] --> B{SHA-512 checksum
match across replicas?}
    B -->|Yes| C[Range is healthy
next check in ~24h]
    B -->|No| D[Second check with diff
logs key-level divergence]
    D --> E[Node logs fatal error
and terminates]
    E --> F[Operator must isolate,
assess scope, recover]

Common causes

Quick checks

Read-only diagnostics. Run on the affected node if still running, or on surviving nodes. Disk-level commands require root.

# Check if the node process is still running
ps aux | grep '[c]ockroach'

# Check for consistency check failure messages in logs
grep -i "consistency check failed" /path/to/cockroach-data/logs/cockroach.log

# Check for Pebble-level corruption errors
grep -i "checksum mismatch" /path/to/cockroach-data/logs/cockroach.log

# Check for blob file corruption (newer Pebble versions)
grep -i "blob file footer checksum" /path/to/cockroach-data/logs/cockroach.log

# Check for disk-level errors in kernel ring buffer (requires root)
sudo dmesg | grep -iE 'error|fail|corrupt' | tail -20

# Check disk SMART health (requires root)
sudo smartctl -a /dev/sdX | grep -iE 'reallocated|pending|uncorrectable|error'

# Check for recent IMPORT or RESTORE jobs on surviving nodes
cockroach sql --host=<surviving-node>:26257 -e "SELECT job_id, job_type, status, created FROM crdb_internal.jobs WHERE job_type IN ('IMPORT','RESTORE') ORDER BY created DESC LIMIT 10;"

# Check under-replicated and unavailable ranges on a surviving node
curl -s http://<surviving-node>:8080/_status/vars | grep -E 'ranges_underreplicated|ranges_unavailable'

# Check current CockroachDB version
cockroach sql --host=<surviving-node>:26257 -e "SELECT version();"

How to diagnose it

1. Confirm the failure is from the consistency checker. The message must be a definitive “consistency check failed” or “checksum mismatch” from the consistency checker subsystem, not a loose match on log noise containing the word “corrupt” in some other context. Grep specifically for the checker’s output.

2. Preserve the logs before anything else. The second-pass diff output is your primary forensic evidence. Copy the entire log directory from the terminated node before taking any recovery action.

3. Identify the affected range(s). The failure message includes range IDs. Record them.

4. Determine the scope. Check whether the corruption is isolated to one replica or spread across multiple. Storage-level corruption on a single disk typically affects one replica. Software bugs that cause divergence may produce different results on different replicas. Note: if a corrupted write were committed through Raft normally, all replicas would agree and the consistency checker would not fire. The checker detects divergence between replicas, not corruption per se.

5. Check for hardware causes. Run dmesg, check SMART data, and review WAL fsync latency history for the affected disk. Disk corruption is the most common root cause. If hardware is failing, corruption will recur on that disk regardless of database-level recovery.

6. Check for software causes. Review recent operations: IMPORT jobs, schema changes, CockroachDB version upgrades. Check whether your version has known consistency check or checksum mismatch bugs. Historical examples include v20.2.x (iterator destruction during range splits) and v22.1.0 (elevated “pebble/table: invalid table” errors).

7. Assess cluster safety. Check ranges_underreplicated and ranges_unavailable on surviving nodes. If the cluster has lost quorum on critical ranges, the recovery path is different from a single-corrupt-replica scenario.

Metrics and signals to monitor

Fixes

Step 1: Isolate the affected node

Do not restart the terminated node. If the corruption is on disk, restarting will trigger the same consistency check failure and the node will terminate again.

Keep the node’s data directory intact for forensic analysis. Copy logs and any Pebble debug output before modifying anything.

Step 2: Determine if quorum is intact

If the failed node was one of three replicas and the other two are healthy, the range still has quorum. The corrupt replica can be removed and rebuilt from healthy replicas.

If quorum is lost on affected ranges, you need loss-of-quorum recovery. The cockroach debug recover command family exists for this scenario, but it is not a general corruption repair tool.

Step 3: Decommission or remove the corrupt node

If quorum is intact on affected ranges:

1. Decommission the terminated node (cockroach node decommission <node-id> --host=<surviving-node>:26257) to remove its replicas from the cluster.
2. The cluster will up-replicate affected ranges from healthy replicas to other nodes.
3. Monitor ranges_underreplicated to confirm the cluster is healing.

If the node holds the only copy of some data (unlikely in a properly replicated cluster), decommissioning is not safe. Restore from backup.

Step 4: Restore from backup if corruption scope is unclear

If corruption has spread to multiple replicas, or if you cannot determine which replica is healthy, restore the affected ranges or tables from your most recent backup. This is the safest recovery path when the scope of corruption is unclear.

CockroachDB’s termination message explicitly warns: “It is not necessarily safe to replace this node; cluster data may still be at risk of corruption.” Take this seriously.

Step 5: Replace hardware if disk failure is the cause

If dmesg, SMART data, or WAL fsync latency history indicate a failing disk, replace the hardware before reintroducing a node to the cluster. A new node on the same failing disk will reproduce the failure.

Step 6: Report the bug if no hardware cause exists

Consistency check failures without a hardware root cause are likely CockroachDB or Pebble bugs. Collect logs, Pebble debug output, and version information. File a report with CockroachDB support. Known corruption-class bugs have been tracked for IMPORT-induced SSTable corruption and Pebble blob file footer checksum errors.

Prevention

• Monitor disk health proactively. WAL fsync latency, disk stall detection, and SMART data provide early warning before corruption occurs.
• Keep CockroachDB patched. Several versions have had elevated checksum mismatch rates from software bugs. Track release notes for corruption-related fixes.
• Rate-limit bulk operations. IMPORT jobs with high concurrent AddSSTable requests can produce SSTable corruption during range splits. Tune concurrency settings conservatively.
• Verify backups are restorable. Test restores periodically. Your backup is your last line of defense.
• Watch the consistency queue backlog. If server.consistency_check.max_rate is too low for your replica count, the consistency queue can back up, delaying detection of corruption.
• Alert on SQL XX000 internal errors. These can indicate storage corruption or internal bugs before the consistency checker runs its next cycle.

How Netdata helps

• Per-second disk health metrics (WAL fsync latency, disk stall counters, I/O error rates) give early warning before corruption propagates. Correlating disk health with consistency check timing helps determine whether hardware is the root cause.
• Node liveness and range availability metrics (ranges_unavailable, ranges_underreplicated) provide immediate visibility into cluster state after a node terminates.
• SQL error rate monitoring by error code class surfaces XX000 internal errors that may precede or accompany corruption, giving broader signal coverage than log scanning alone.
• Log-based alerting on definitive consistency checker output ensures you are paged on confirmed corruption events without false positives from unrelated log noise.
• Cross-node correlation lets you determine whether a consistency failure on one node correlates with disk anomalies, version upgrades, or bulk operation timing on adjacent nodes.

For details, see CockroachDB monitoring with Netdata.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB certificate expired: TLS handshake failures and online rotation
• CockroachDB changefeed lag: changefeed_max_behind_nanos and the GC time bomb
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule