ZooKeeper "Unable to load database on disk": corrupt snapshot on startup

The startup error Unable to load database on disk (logged by QuorumPeer when the database load fails) means ZooKeeper cannot reconstruct its in-memory data tree from the on-disk snapshot and transaction log. The node refuses to join the ensemble and exits before serving traffic. You typically see this only on the next restart after the corruption happened, often days later.

The failure is nasty because the running process looks fine until it does not. The corruption was already on disk; the restart made it impossible to ignore. A node that was serving requests an hour ago can refuse to come back after an unclean shutdown, an OOMKill, or a full dataDir disk.

Recovery requires either restoring the corrupt file from a healthy peer or removing the local state so the node falls back to an earlier snapshot or rebuilds via a full sync. Both paths assume at least one healthy peer exists in the ensemble. If every node is corrupt at once, you need an external backup.

What this means

On startup, every ZooKeeper server loads the most recent snapshot from dataDir/version-2/snapshot.<zxid> and replays the transaction log forward from dataLogDir/version-2/log.<zxid> to reconstruct the in-memory data tree. If the snapshot cannot be deserialized, or the transaction log cannot be replayed, the node fails fast and writes Unable to load database on disk to its log, usually followed by a Java exception such as EOFException, IOException, KeeperException$NoNodeException, or a “Unreasonable length” error.

ZooKeeper fails closed here rather than serving from partial state, because a corrupt tree would silently diverge from the ensemble and propagate bad data to clients. The fix is to repair local state, not to force the node up with corrupted data.

The corruption itself almost always happened before the restart. Classic triggers:

• Disk filled mid-write, leaving a truncated snapshot or transaction log.
• Unclean shutdown (power loss, OOMKill, kill -9) that interrupted an fsync.
• Transaction that grew beyond jute.maxbuffer (default 0xfffff, just under 1 MB) during processing and was written to the log but cannot be deserialized on replay.

None of these produce a visible outage while the node is still running. The zk_snapshot_error_count counter is the only forward signal, and it must be observed before the next restart.

flowchart TD
    A[Running ZK node] --> B{Trigger event}
    B -->|Disk fills mid-write| C[Truncated snapshot or txnlog]
    B -->|OOMKill / SIGKILL / power| D[Interrupted fsync]
    B -->|Txn exceeds jute.maxbuffer| E[Unreadable log entry]
    C --> F[Corrupt file on disk]
    D --> F
    E --> F
    F --> G[Node keeps serving from in-memory tree]
    G --> H[Next restart: Unable to load database on disk]
    H --> I{Healthy peer exists?}
    I -->|Yes, single node corrupt| J[Delete version-2, SNAP sync from leader]
    I -->|No, all nodes corrupt| K[Restore from external backup]

Common causes

Quick checks

# Confirm the error and capture the trailing exception
grep -A 20 "Unable to load database on disk" /var/log/zookeeper/zookeeper.log | tail -40

# Check snapshot and txnlog partition fullness now
df -h /var/zookeeper/data /var/zookeeper/txnlog

# Inspect the latest snapshot and transaction log files
ls -lt /var/zookeeper/data/version-2/snapshot.* | head -5
ls -lt /var/zookeeper/txnlog/version-2/log.* | head -5

# Check whether any peer is still healthy
echo ruok | nc peer-host 2181
echo srvr | nc peer-host 2181 | grep Mode

# Confirm the local process is not running (the load failed)
echo ruok | nc localhost 2181

# Pull integrity counters from any peer that is still up
echo mntr | nc peer-host 2181 | grep -E "zk_snapshot_error_count|zk_digest_mismatches_count|zk_unrecoverable_error_count|zk_restore_error_count"

How to diagnose it

1. Read the exact exception trailing the error. EOFException usually means truncation. IOException with “Unreasonable length” points at jute.maxbuffer. KeeperException$NoNodeException mid-replay indicates the snapshot and log are inconsistent with each other.

2. Confirm which disk filled, if any. Pull disk usage history for dataDir and dataLogDir. A node that crossed 100% on the txnlog partition and then restarted is the textbook disk-full case. Clearing space now does not heal the truncated log.

3. List the most recent snapshot and transaction log on the failing node. A snapshot file with an implausibly small size, or a log.<zxid> far shorter than the typical 64 MB pre-allocation, is the prime suspect.

4. Verify that at least one peer is healthy and current. echo ruok | nc peer 2181 returning imok, srvr showing Mode: leader or Mode: follower, and a mntr scrape showing a non-zero zk_zxid confirm the ensemble still has a source of truth to repair from.

5. Check whether the corruption is cluster-wide. If multiple nodes refuse to start with the same exception, the corrupt transaction log is on every node, typically from a single oversized transaction. Standard “delete version-2 and re-sync” does not work because there is no healthy peer to re-sync from. Workarounds are raising jute.maxbuffer past the offending transaction size or restoring from external backup.

6. If you recently upgraded from 3.4.x, check whether the trailing message is actually “No snapshot found, but there are log entries.” That is the upgrade-time validation introduced in 3.5.x. The documented escape hatch is zookeeper.snapshot.trust.empty=true in zoo.cfg until a real snapshot has been created. Remove the property afterward.

Metrics and signals to monitor

Fixes

Remove the corrupt local state and re-sync from a healthy peer

The recovery documented in the ZooKeeper admin guide is to delete every file under datadir/version-2/ and datalogdir/version-2/ on the failing node, then restart it. On restart the node performs a SNAP sync from the leader: it receives the full current data tree as a snapshot and applies it locally.

Use this when at least one healthy peer exists, the ensemble has quorum, the corruption is isolated to one node, and you can tolerate the node being unavailable for the duration of the full snapshot transfer.

Destructive command. Only run on the node that is already failing to start.

# DESTRUCTIVE: removes all local ZooKeeper state. Only safe when at least
# one healthy peer exists in the ensemble and quorum is intact.
systemctl stop zookeeper
rm -rf /var/zookeeper/data/version-2/*
rm -rf /var/zookeeper/txnlog/version-2/*
systemctl start zookeeper

Tradeoff: the node does a full SNAP sync, which is I/O- and bandwidth-intensive on the leader. During the sync the leader has reduced capacity. For large data trees, redirect client reads away from the leader during the transfer if you can.

Restore from a peer snapshot via the Admin Server API (ZooKeeper 3.9.x)

ZooKeeper 3.9.x exposes an HTTP snapshot and restore API on the Admin Server, rate-limited to one restore per five minutes. Pull a snapshot from a healthy peer and push it to the recovering node:

# Pull a streaming snapshot from a healthy peer
curl http://healthy-peer:8080/commands/snapshot?streaming=true --output /tmp/zk-snapshot.bin

# Push it to the recovering node
curl -X POST http://recovering-node:8080/commands/restore --data-binary "@/tmp/zk-snapshot.bin"

This can be faster than waiting for a full SNAP sync in topologies where pulling from a non-leader peer spares the leader’s serialization cost. The five-minute rate limit makes it unsuitable for fast recovery of many nodes at once.

Fall back to an earlier snapshot

If only the most recent snapshot is corrupt and the txnlogs are intact, removing just that single snapshot.<zxid> file rather than all of version-2/ lets ZooKeeper fall back to the previous snapshot and replay forward. This accepts whatever transaction-replay gap exists between the two snapshots and is only safe if the previous snapshot is consistent with the surviving logs.

This is a narrower intervention than wiping version-2/, but more fragile: you are betting that the second-newest snapshot is intact and that the replay does not trip over the same oversized transaction or NoNodeException that broke the newest one. Prefer this only when you have a specific reason to avoid a full re-sync.

Cluster-wide corruption: there is no clean fix

If every node fails to start with the same exception, standard recovery does not work because there is no healthy peer to re-sync from. Documented workarounds:

• Increase jute.maxbuffer past the size of the offending transaction so the log can be replayed.
• Restore the dataDir and dataLogDir of at least one node from external backup, bring the ensemble up from that node, then let the others re-sync.

If you have no external backup and cannot raise jute.maxbuffer enough, there is no automated recovery. Treat this as a data-loss incident.

Upgrade-time “No snapshot found” failures

If the error appears immediately after upgrading from 3.4.x to 3.5+ or 3.6+ and the trailing message is “No snapshot found, but there are log entries,” set zookeeper.snapshot.trust.empty=true in zoo.cfg, restart the node, let it create a real snapshot, then remove the property and restart again. Leaving zookeeper.snapshot.trust.empty=true set permanently defeats the validation that catches this class of bug.

Prevention

• Alert on zk_snapshot_error_count. It is the only signal that surfaces this failure while the node is still running, before the next restart turns it into an outage. Any increment warrants investigation.
• Keep dataDir and dataLogDir below 80% full. A full dataLogDir truncates txnlog writes; recovery requires manual file removal.
• Configure autopurge. autopurge.purgeInterval (in hours) must be non-zero; autopurge.snapRetainCount (default 3) controls retention. The default purgeInterval of 0 disables autopurge entirely, which is how disks fill silently over months.
• Monitor zk_digest_mismatches_count and zk_unrecoverable_error_count. A digest mismatch is the strongest indicator that data on a node has already diverged from what the rest of the ensemble believes.
• Keep jute.maxbuffer at its default unless you have a documented reason to raise it. A too-large buffer combined with the integer-overflow bug fixed in 3.9.3 and 3.10.0 can itself produce false “Unreasonable length” errors. If you must raise it, configure zookeeper.jute.maxbuffer.extrasize so the sum stays below Integer.MAX_VALUE.
• Take external backups of dataDir/version-2/ and dataLogDir/version-2/ from at least one node. They are the only recovery path when corruption hits the entire ensemble at once.
• Treat any unclean shutdown (OOMKill, SIGKILL, host power loss) as a trigger to check snapshot integrity before the next planned restart, not after.

How Netdata helps

• Per-second collection of zk_snapshot_error_count, zk_restore_error_count, zk_unrecoverable_error_count, and zk_digest_mismatches_count surfaces the only forward signals for this failure mode. A counter increment is visible while the node is still running, not on the next restart.
• Host-level disk usage on the dataDir and dataLogDir partitions can be correlated against zk_snapshot_error_count increments on the same timeline, confirming the disk-full root cause without a separate investigation.
• zk_uptime resets reveal the unclean shutdown that produced the corrupt file. Correlating an uptime reset against a subsequent snapshot-error increment reconstructs the causal chain.
• ML-based anomaly detection on the integrity counters surfaces a single increment as anomalous even when absolute values are tiny, which matters because these counters should be exactly zero in steady state.
• Composite dashboards let you place mntr integrity counters alongside JVM GC pause metrics and disk latency, which is what you need to reconstruct the moment the corruption happened rather than just the moment the node refused to start.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun
• ZooKeeper “Detected pause in JVM or host machine (e.g. GC)”: the pause-monitor warning
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper GC pause cascade: how a Stop-the-World freeze expires sessions and re-elects the leader
• ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once