ZooKeeper data tree digest mismatch: detecting corruption before it spreads

When zk_digest_mismatches_count increments on a ZooKeeper node, the in-memory data tree on that node has diverged from the checksum ZooKeeper expects. This is a data-integrity alarm, not a performance signal. Clients reading from that node may be receiving wrong answers, and if the divergence came from a ZAB replication bug rather than local corruption, the same divergence may be propagating to other ensemble members.

The digest feature (enabled by default since ZooKeeper 3.6.0) computes a running checksum of the entire data tree using an incremental adHash scheme. On every write, ZooKeeper updates the hash and compares it against the expected digest attached to the transaction. When the comparison fails, the counter ticks up and ZooKeeper logs a “First digest mismatch on txn” message. A single increment is enough to page.

What this means

ZooKeeper holds its entire data tree in JVM heap. Every znode, its data, its ACL, its children list: all live in memory and replicate across the ensemble via ZAB. There is no per-read checksum validation and no on-disk integrity scan at runtime. The adHash digest is the only continuous in-process integrity check ZooKeeper runs.

The digest is incremental: when a znode changes, ZooKeeper computes current_hash = current_hash + hash(new_data) - hash(old_data). The hash function is CRC-based: fast, not collision-resistant, but adequate for detecting bit flips and dropped transactions. The expected digest for each committed transaction is computed by the leader and attached to the proposal. Followers compare their locally-computed digest against this expected value when they apply the commit. Nodes under /zookeeper (the internal subtree, including quota stats) are excluded from the calculation because their bookkeeping can legitimately diverge.

When the comparison fails:

1. The DIGEST_MISMATCHES_COUNT counter (exposed via mntr as zk_digest_mismatches_count) increments.
2. ZooKeeper logs the mismatched zxid and the expected vs actual digest values.

The node keeps running. It does not crash, does not stop serving reads, and does not drop out of the ensemble on its own. Without an explicit alert on the delta, a diverged node can serve stale or corrupted data indefinitely while appearing healthy to shallow liveness checks like ruok.

One logging gotcha: the mismatch log line prints the TxnDigest object as the “expected digest” (for example, “expected digest is 2,10124071835”) while the actual digest is a single number, making the two values hard to compare by eye. This is tracked as ZOOKEEPER-4845 and is still open.

Common causes

Quick checks

These are all read-only. Run them on every ensemble member, not just the one that alerted.

# Check the mismatch counter on each node
echo mntr | nc localhost 2181 | grep zk_digest_mismatches_count

# Check for the other integrity signals that travel with digest mismatches
echo mntr | nc localhost 2181 | grep -E 'zk_(unrecoverable_error|snapshot_error|restore_error)_count'

# Confirm which node is the leader (leader-only metrics only appear there)
echo mntr | nc localhost 2181 | grep zk_server_state

# Pull the digest history associated with zxids (3.6.0+)
echo hash | nc localhost 2181

# Look for the mismatch log lines on the affected node
grep -i "digest" /var/log/zookeeper/zookeeper.log | tail -50

# Check for hardware memory errors on the host
edac-util -v     # or: ipmitool sel list | grep -i memory

# Check for disk-level errors that could have corrupted snapshot or txnlog
dmesg -T | grep -iE 'error|medium|failed|ecc|mce' | tail -30

# Compare the current zxid across all ensemble members to see if one is behind
# (mntr does not expose the zxid; the Zxid line comes from srvr)
echo srvr | nc localhost 2181 | grep -E 'Zxid|Mode'

If hash is not whitelisted you will get “hash is not executed because it is not in the whitelist.” Add it to 4lw.commands.whitelist if you need it, or use the AdminServer /commands/hash endpoint on port 8080 instead. The same whitelist caveat applies to mntr since ZooKeeper 3.5.3.

How to diagnose it

The single most important diagnostic question is: how many nodes incremented?

flowchart TD
    A["zk_digest_mismatches_count incremented"] --> B{"Which nodes incremented?"}
    B -->|"One node"| C{"Did it increment on startup?"}
    B -->|"Multiple nodes"| D["Stop. Escalate immediately."]
    C -->|"Yes, during load or replay"| E["Disk corruption in snapshot or txnlog"]
    C -->|"No, at runtime"| F{"Recent follower sync or restart?"}
    F -->|"Yes"| G["ZAB replication bug or stale on-disk state"]
    F -->|"No"| H["Memory error. Check ECC counts."]
    E --> I["Rebuild node from healthy snapshot"]
    G --> I
    H --> I
    D --> J["Likely ZAB bug. Stop writes, preserve data, file issue with zxids."]

1. Collect the counter from every node. Do not trust a single-node scrape. A digest mismatch on one follower is a local problem. A digest mismatch on the leader, or on multiple followers simultaneously, is an ensemble problem.

2. Note whether the increment happened at startup or at runtime. A mismatch during snapshot or txnlog replay points to on-disk corruption: the file was written correctly by ZK but the bytes on disk no longer match. A mismatch at runtime, after the tree was already loaded, points to either a memory error or a ZAB-layer inconsistency introduced during sync.

3. Correlate with sync events. If the mismatch appeared immediately after a follower rejoined the ensemble or after the leader restarted, check the ZooKeeper version against the known ZAB bugs. ZOOKEEPER-4352 affects 3.6.3 and 3.7.0: when the leader has minCommittedLog=0 (empty in-memory committed log), it sends only a DIFF to a rejoining follower instead of a full snapshot, leaving the follower with an inconsistent tree. ZOOKEEPER-4444 affects 3.6.3 through 3.8.1: a follower killed and restarted twice fails to synchronize and produces “Digests are not matching” while serving stale or deleted znodes.

4. Check the hardware. If only one node is affected and there is no recent sync or restart, treat it as a suspected memory error until proven otherwise. Check ECC error counts, check dmesg for machine-check exceptions, and check whether the host is bare metal with faulty DIMMs or a VM on a noisy host.

5. Capture the mismatched zxid from the log before you touch anything. The log line contains the transaction at which the divergence was detected. If you end up filing a ZooKeeper issue or doing forensic comparison of snapshots, you need this number.

Metrics and signals to monitor

Fixes

There is no in-place repair for a diverged data tree. The correct response is to remove the node from the ensemble and rebuild its state from a known-good source.

Single-node mismatch: eject and rebuild

If exactly one node incremented and the rest of the ensemble is clean:

1. Stop the ZooKeeper process on the affected node. This prevents it from serving further stale or corrupted reads.
2. Preserve the data directory for forensics. Copy dataDir and dataLogDir aside before you wipe them. If this turns out to be a ZAB bug rather than local corruption, the maintainers will want the corrupted snapshot and txnlog.
3. Wipe the data directories. Remove the contents of dataDir/version-2/ and dataLogDir/version-2/ (or wherever your dataDir and dataLogDir point). This is destructive. Confirm you have a forensic copy from step 2 before proceeding.
4. Restart the node. It will rejoin the ensemble and perform a full sync from the leader (SNAP sync). The leader serializes its complete data tree and sends it. The rebuilt node’s tree will match the leader’s, and the digest will be correct.
5. Verify. After sync completes, confirm zk_digest_mismatches_count is zero on the rebuilt node and that its zxid matches the leader.

Do not attempt to patch the diverged znodes by hand. The node’s state is already known to be wrong; rebuilding from a healthy peer is the only reliable path.

If the node that mismatched is the leader, force a leader election first (stop the leader, let the ensemble elect a new one from the clean followers), then rebuild the former leader as a follower.

Multi-node mismatch: stop and escalate

If the mismatch appears on the leader, or on more than one follower, do not rebuild individual nodes. The divergence is likely coming from the replication layer itself, and rebuilding nodes from a leader that is producing bad digests will propagate the corruption.

1. Stop writes if you can. If dependent systems (Kafka, HBase) can be paused or failed over to a secondary coordination path, do so. Every additional transaction on a diverged ensemble risks wider inconsistency.
2. Preserve all data directories on all nodes. This is your forensic evidence.
3. Identify the most-trusted node. Usually this is the node with the highest committed zxid that has not logged a digest mismatch. If you cannot identify one with confidence, do not pick arbitrarily.
4. Escalate. Multi-node digest divergence is a ZAB-level event. File a ZooKeeper issue with the mismatched zxids, the versions, and the sync history. If you have a support relationship with a vendor distribution, engage it.

Version-specific bugs

If your version matches a known ZAB bug, the rebuild procedure above is the workaround, but you should also plan an upgrade:

• ZOOKEEPER-4352 (3.6.3, 3.7.0): Restarting the leader to rebuild its in-memory committed log before the follower rejoins avoids the empty-committedLog DIFF path.
• ZOOKEEPER-4444 (3.6.3 through 3.8.1): Deleting the data directory on the affected follower and letting it re-sync is the documented workaround. No fix version has been assigned for this issue.
• ZOOKEEPER-4845 (open): The confusing log format is not a data-integrity bug, but it makes forensics harder. The “expected digest” field in the log is a TxnDigest object representation, not a single comparable number.

Prevention

• Alert on the delta of zk_digest_mismatches_count, not the absolute value. The counter is monotonic within a process lifetime and resets on server restart; any increment is a PAGE.
• Monitor zk_unrecoverable_error_count, zk_snapshot_error_count, and zk_restore_error_count alongside the digest counter. A digest mismatch with no other error counters moving suggests memory or ZAB; a digest mismatch with snapshot or restore errors suggests disk.
• Run ZooKeeper on hosts with ECC memory. The adHash digest catches single-bit flips that non-ECC memory will not report on its own. Without ECC, the digest mismatch may be your first and only indication of a failing DIMM.
• Keep dataLogDir on dedicated, healthy storage. Disk corruption in the txnlog or snapshot is a root cause most preventable by operational hygiene.
• Stay current on patch releases. The ZAB sync bugs that produce false-positive digest mismatches (ZOOKEEPER-4352, ZOOKEEPER-4444) are not yet resolved in any released version. Running old 3.6.x or 3.7.x in production means carrying known data-divergence bugs.
• Whitelist the hash four-letter word, or use the AdminServer equivalent. When a mismatch fires, you want the digest history available immediately, not after a config change and restart.

How Netdata helps

• Correlation across ensemble members in a single view lets you answer the critical triage question (one node vs multiple nodes) without manually scraping each node and comparing outputs. Per-second collection means a single increment is visible within a second of occurring.
• Co-display of zk_server_state, zk_zxid, and zk_digest_mismatches_count lets you see whether a mismatch followed a sync event, a leader election, or a restart, which is the fastest path to distinguishing a ZAB bug from local corruption.
• Integration with host-level signals (ECC errors from kernel logs, disk error counters, dmesg patterns) means the hardware-cause hypothesis can be confirmed or eliminated from the same dashboard, without switching tools mid-incident.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper autopurge not configured: snapshots and logs filling the disk over months
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper transaction log disk full: the crash with no graceful degradation
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls