ZooKeeper snapshot errors: recovery safety at risk

zk_snapshot_error_count (new in ZooKeeper 3.9.0) increments when a snapshot-related AdminServer command fails, and zk_restore_error_count (also 3.9.0+) counts restore-command failures. On 3.5.x through 3.8.x these counters do not exist; there the equivalent signals are the server log and zk_unrecoverable_error_count. A failed scheduled snapshot is not silent: since 3.5, an IOException while writing a scheduled snapshot makes the server log Severe unrecoverable error, exiting and stop the JVM.

A single increment (from the AdminServer snapshot command) is often transient: a backup job, a brief disk-full condition, or I/O contention from a colocated batch process can make one snapshot fail, and the next snapshot cycle (taken every snapCount transactions, default 100,000) succeeds. If you page on every increment, you burn through your team’s attention on events that self-resolve. A failure of a scheduled snapshot is different: the node exits immediately, which is itself the alarm.

This metric (on 3.9.0+) is the only counter-based early warning for a corrupt snapshot that will block the next startup. The playbook: escalate to PAGE only when zk_snapshot_error_count is accompanied by zk_unrecoverable_error_count or zk_restore_error_count also incrementing. Both zk_restore_error_count and zk_unrecoverable_error_count are exposed via mntr (3.9.0+; unrecoverable_error_count since 3.6.0). Multiple error counters moving together means the failure is systemic, not transient.

This article covers what to check when zk_snapshot_error_count increments, how to distinguish transient I/O stress from corruption, and how to validate that the node can actually recover before the next time it restarts.

What this means

ZooKeeper persists state in two complementary files: a write-ahead transaction log appended on every write, and a periodic fuzzy snapshot of the entire data tree. On restart, the server loads the latest snapshot and replays any transaction log records with a zxid higher than the snapshot’s. If the snapshot is missing, incomplete, or fails checksum validation, the recovery path is broken.

zk_snapshot_error_count increments when a snapshot AdminServer command fails (ZooKeeper 3.9.0+). The most common causes are the destination filesystem being unable to accept the write — disk full, I/O error, or permission denied — and, less commonly, an OutOfMemoryError partway through serializing the data tree. Scheduled snapshots are written directly to snapshot.<zxid> without an atomic rename when fsync is disabled, so an interrupted write can leave a truncated file that only fails when it is loaded; an OutOfMemoryError during serialization normally terminates the JVM outright.

For an AdminServer snapshot-command failure the node continues serving traffic because the in-memory data tree is unaffected, and the next snapshot cycle can succeed normally. For a scheduled-snapshot failure the node does not keep serving: it logs Severe unrecoverable error, exiting and stops. In either case the recovery safety net is broken, and a node that cannot write snapshots will not come back cleanly after a restart.

A healthy ensemble takes snapshots regularly. Snapshot frequency is governed by snapCount (default 100,000 transactions), randomized slightly per server so all ensemble members do not snapshot simultaneously. If zk_snapshot_error_count keeps incrementing at the snapshot cadence, every snapshot is failing. If it increments once and stops, the cause likely resolved.

flowchart TD
    A[zk_snapshot_error_count increments] --> B{Other error counters moving?}
    B -- "unrecoverable or restore also up" --> C[PAGE: systemic failure]
    B -- "Only snapshot_error_count" --> D{Disk space on dataDir?}
    D -- "Below 20% free" --> E[Disk full: clear space or grow volume]
    D -- Adequate free --> F[Check ZK log for snapshot write errors]
    F --> G{Error is recurring?}
    G -- "Yes, every snapshot cycle" --> H[Ticket: recovery safety broken]
    G -- "No, single event" --> I[Correlate with backup or batch job]
    C --> J[Validate snapshot with zkSnapShotToolkit.sh]
    H --> J

Common causes

Quick checks

All read-only and safe to run on a production node.

# Confirm which error counters are moving
echo mntr | nc localhost 2181 | grep -E "zk_snapshot_error_count|zk_restore_error_count|zk_unrecoverable_error_count"

# Confirm there is at least one recent, non-empty snapshot
ls -lhS <dataDir>/version-2/snapshot.* | head -5

# Check disk space on both dataDir and dataLogDir
df -h <dataDir> <dataLogDir>

# Snapshot and transaction log file counts vs. autopurge.snapRetainCount
ls <dataDir>/version-2/snapshot.* | wc -l
ls <dataLogDir>/version-2/log.* | wc -l

# Disk I/O health on the dataDir device
iostat -x 1 5

# Look for snapshot write failures in the ZK log
grep -iE "snapshot|IOException|No space left|Permission denied" <zk_log_dir>/zookeeper.log | tail -50

# Look for kernel-level I/O errors on the dataDir device
dmesg -T | grep -iE "i/o error|read-only|filesystem" | tail -20

# Confirm the snapshot.trust.empty escape hatch is not set (checks JVM args; also grep zoo.cfg)
ps -ef | grep -o "zookeeper.snapshot.trust.empty=[a-z]*"

# Heap pressure around snapshot time (Full GC or OOM is suspicious)
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 5

Replace <dataDir>, <dataLogDir>, and <zk_log_dir> with the paths from your zoo.cfg.

How to diagnose it

1. Establish scope. Run mntr on every ensemble member and compare zk_snapshot_error_count. A single node incrementing usually means a node-local problem (its disk, its permissions, its heap). All nodes incrementing at the same time points at something systemic: a shared storage backend, a correlated backup job hitting every node, or a data tree size that has outgrown heap on every member.

2. Capture the timestamp. Note the wall-clock time of the increment. Snapshot creation is logged. Cross-reference the increment timestamp with the ZK server log around that moment. The error message (IOException text, OutOfMemoryError, Permission denied) tells you the cause class directly.

3. Correlate with other error counters. The single most important triage step:

   • zk_snapshot_error_count alone: TICKET. The node is running; recovery safety is at risk but the cluster is serving traffic.
   • zk_snapshot_error_count plus zk_restore_error_count: PAGE. The server has failed to load state during recovery, not just write it.
   • zk_snapshot_error_count plus zk_unrecoverable_error_count: PAGE. A critical internal error compounds the snapshot failure; the server’s integrity is in question.
   • zk_snapshot_error_count plus zk_digest_mismatches_count: PAGE. The data tree diverged from its checksum, which means in-memory state may already be corrupted, not just at risk.

4. Inspect the snapshot directory. The latest snapshot file should be roughly the size of zk_approximate_data_size plus per-znode overhead. A file much smaller than the prior snapshot, or a snapshot file with a timestamp older than snapCount transactions worth of writes, indicates the last successful snapshot was a while ago. Look for .tmp or partially written files that indicate a write interrupted mid-stream.

5. Validate the latest snapshot before relying on it. zkSnapShotToolkit.sh (bundled with ZooKeeper) reads a snapshot file and dumps its contents. If it parses cleanly, the snapshot is structurally valid. If it throws an EOFException or Unreasonable length error, the snapshot is corrupt and the node will fail to recover from it. The toolkit’s flags are -d (dump each znode’s data) and -json (json output).

6. Check whether snapshot.trust.empty is set. This property (zookeeper.snapshot.trust.empty, introduced in 3.5.6 and 3.6.0 as an upgrade escape hatch, ZOOKEEPER-3056) tells ZooKeeper to start even when no snapshot exists but transaction logs are present. Leaving it set after upgrade defeats the snapshot integrity check entirely. A node with this property set may appear to recover but with stale or empty state. Remove it as soon as a valid snapshot exists.

Metrics and signals to monitor

Fixes

Disk full on dataDir

Free space first. The safest immediate action is to confirm autopurge is enabled and let it run: autopurge removes old snapshots and transaction logs down to autopurge.snapRetainCount (default 3). Do not manually delete snapshots or logs by hand unless you understand which ones are needed for recovery. The minimum set is the latest snapshot plus all transaction logs with a zxid greater than that snapshot’s. Removing the wrong file breaks recovery.

If autopurge is not configured (autopurge.purgeInterval = 0), set it now. See ZooKeeper autopurge not configured: snapshots and logs filling the disk over months.

If the partition is chronically undersized, grow the volume. Do not use this as a permanent fix without also addressing why snapshot sizes are growing. See ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern.

I/O errors or device degradation

Move dataDir to a healthy device. If the device is throwing kernel-level errors, the filesystem is likely already damaged. After moving the directory, the node will perform a SNAP sync from the leader on restart to rebuild its snapshot and log state. See ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius.

Permission or ownership

If a recent chown, chmod, or restore from backup changed ownership of the version-2 directory, the ZooKeeper user can no longer write snapshots. Restore the correct owner and mode. The directory should be owned by the user running QuorumPeerMain and writable by that user.

JVM OOM during snapshot

This is the most dangerous cause because the snapshot file is written but truncated. Heap pressure comes from one of three sources: data tree bloat (zk_znode_count and zk_approximate_data_size climbing), watch table growth, or session table bloat. Check heap usage with jstat -gcutil and the GC log. Increase heap (after the rolling restart) and address the underlying growth. See ZooKeeper avg_latency hides write stalls: why the headline number lies for related heap-pressure patterns.

Corrupt snapshot

If zkSnapShotToolkit.sh cannot parse the latest snapshot, you have two options:

1. Force a SNAP sync from the leader. Stop the node, move the entire version-2 directory aside (do not delete it; keep it for forensics), and restart. The node will request a full snapshot from the leader. This is the standard recovery path for a single corrupt node. See ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius.

2. Repair the transaction log with zkTxnLogToolkit.sh. If the failure is a CRC error in the transaction log rather than the snapshot, the bundled zkTxnLogToolkit.sh can rewrite the log with recalculated CRCs in recovery mode (-r, with -y for non-interactive fix-all). It writes a .fixed file next to the original (-r for recover mode, -y for non-interactive fix-all). This is recovery work, not first-line triage.

Do not move the version-2 directory aside on more than one node at a time. Recovery depends on at least one healthy member having a valid snapshot.

Prevention

• Monitor zk_snapshot_error_count on every member, every scrape. Treat any increment as a TICKET. Escalate to PAGE only when paired with zk_unrecoverable_error_count or zk_restore_error_count.
• Track snapshot recency. If your monitoring can read the filesystem, alert when the newest snapshot file is older than snapCount worth of writes plus a safety margin. A node that stops snapshotting is a node whose recovery safety has quietly gone stale.
• Enable autopurge. autopurge.purgeInterval (hours) and autopurge.snapRetainCount (default 3) prevent disk exhaustion and recovery time creep. See ZooKeeper autopurge not configured: snapshots and logs filling the disk over months.
• Keep dataDir and dataLogDir on separate devices. Snapshot writes and transaction log fsyncs competing for the same disk is a common root cause of write stalls that compound into snapshot failures.
• Remove snapshot.trust.empty after upgrade. It exists to bridge 3.4.x to 3.5.x+ upgrades. Leaving it set disables the snapshot integrity check. Verify it is unset on every node.
• Size heap to the data tree. Snapshot serialization allocates buffers proportional to tree size. A heap sized for steady-state reads but not for snapshot time will OOM at the worst moment.
• Track snapshot size over time. Growing snapshots mean a growing data tree, longer recovery times, and higher OOM-during-snapshot risk. See ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern.
• Patch against AdminServer snapshot/restore CVEs. If you are running ZooKeeper 3.9.x before 3.9.4, patch two AdminServer issues: CVE-2024-51504 (IP-authentication bypass in the AdminServer via X-Forwarded-For header spoofing, fixed in 3.9.3) and CVE-2025-58457 (insufficient permission check on the AdminServer snapshot and restore commands, fixed in 3.9.4). Upgrade to 3.9.4 or later, or disable the endpoints via admin.snapshot.enabled=false and admin.restore.enabled=false (admin.enableServer=false disables the whole AdminServer).

How Netdata helps

• Per-second scraping of zk_snapshot_error_count catches single increments that slower scrapers miss entirely. A minute-level scrape can fold a transient backup-induced failure and a real corruption event into the same data point.
• Correlating zk_snapshot_error_count with zk_unrecoverable_error_count, zk_restore_error_count, and zk_digest_mismatches_count in one view makes the PAGE-versus-TICKET decision immediate. Multi-counter increments are the systemic-failure signature.
• ML anomaly detection on zk_approximate_data_size, zk_znode_count, and heap metrics surfaces data-tree-bloat and heap-pressure conditions that produce OOM-during-snapshot before the snapshot actually fails.
• Disk usage and disk I/O metrics on the same dashboard as the ZooKeeper metrics let you confirm disk-full or I/O-error causes without switching tools.
• Cold-start suppression keyed off zk_uptime prevents brief snapshot failures during recovery from a planned restart from firing as a real incident.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper autopurge not configured: snapshots and logs filling the disk over months
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper KeeperErrorCode = ConnectionLoss: the transient disconnect every client hits
• ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper transaction log disk full: the crash with no graceful degradation
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection