ZooKeeper transaction log disk full: the crash with no graceful degradation

ZooKeeper has no graceful degradation path for a full dataLogDir partition. When the WAL append fails, the server throws an IOException and dies. There is no read-only fallback, no throttling, and no mntr warning that precedes the crash. The same applies to the snapshot directory when the next snapshot write or pre-allocation fails.

The most common root cause is broken or disabled autopurge. With autopurge.purgeInterval defaulting to 0 (disabled) and autopurge.snapRetainCount defaulting to 3, an ensemble that has never been explicitly configured will accumulate transaction logs and snapshots forever. Disk consumption is silent and cliff-edge. By the time ruok fails, the process is already gone.

What this means

Every write to ZooKeeper is synchronously appended to a write-ahead transaction log under dataLogDir and fsync’d before the proposal is broadcast to followers. If the append fails, the integrity guarantee is broken and the server stops.

ZooKeeper pre-allocates transaction log files in 64MB chunks (preAllocSize, default 64M) to avoid seeks during writes. A partition that appears to have tens of megabytes free can fail on the next log rotation, before the actual transaction data would have fit. The error message is identical to a true full-disk error.

The crash is the beginning of the problem, not the end. After you free space and restart ZooKeeper, recovery can fail in three ways:

• Crash-loop on restart: The last log file is truncated mid-transaction. ZooKeeper reads the header, gets an EOFException, and exits again. Freeing space alone does not fix this. ZOOKEEPER-1621 documents this behavior; operators on 3.4.x and early 3.5.x branches are most likely to encounter it.
• “Unreasonable length” IOException: A full disk can corrupt the middle of a log, not just the end. ZOOKEEPER-3975 captures this pattern.
• Silent data loss on quorum rejoin: If the node restarts multiple times and autopurge runs each time, valid snapshots can be replaced by snapshots taken during the disk-full window. The node then joins the quorum with only the transactions from the last orphaned log file. ZOOKEEPER-2745 documents this as a still-open critical bug. ZOOKEEPER-2325 mitigates part of it on 3.5.x+ by refusing to restore from txn logs when no valid snapshot exists.

flowchart TD
    A[txnlog partition fills] --> B[IOException: process dies]
    B --> C[operator frees space, restarts]
    C --> D{recovery path}
    D -->|last log clean| E[rejoins quorum]
    D -->|log truncated or corrupted| F[crash-loop: EOFException or Unreasonable length]
    F --> G[repair with TxnLogToolkit -r or delete last log]
    G --> E
    D -->|valid snapshot purged| H[rejoins with orphaned txnlogs only]
    H --> I[silent state divergence; watch digest mismatch counter]

Common causes

Quick checks

All read-only. Run them on the affected node first, then on the rest of the ensemble.

# Free space on the txnlog and snapshot partitions
df -h /var/zookeeper/txnlog /var/zookeeper/data

# Process and quorum state - is ZK actually up and writable?
echo ruok | nc -w 2 localhost 2181
echo isro | nc -w 2 localhost 2181
echo mntr | nc -w 2 localhost 2181 | grep -E 'zk_server_state|zk_uptime'

# Count and size of transaction logs
ls -la /var/zookeeper/txnlog/version-2/log.* | wc -l
du -sh /var/zookeeper/txnlog/version-2/

# Count and size of snapshots
ls -la /var/zookeeper/data/version-2/snapshot.* | wc -l
du -sh /var/zookeeper/data/version-2/
ls -lhS /var/zookeeper/data/version-2/snapshot.* | head -5

# Newest and oldest txnlogs - gauge growth rate
ls -lt /var/zookeeper/txnlog/version-2/log.* | head -3
ls -lt /var/zookeeper/txnlog/version-2/log.* | tail -3

# ZK log for the actual IOException that killed the process
grep -E 'IOException|No space left|Unreasonable length|fsync-ing' /var/log/zookeeper/zookeeper.log | tail -30

# Confirm autopurge config
grep -E 'autopurge|dataLogDir|snapCount' zoo.cfg

How to diagnose it

1. Confirm the partition is the cause. Run df -h on both dataLogDir and dataDir. A full txnlog partition produces an immediate crash; a full snapshot partition produces failures at the next snapshot. Either can fail during pre-allocation before they look completely full.
2. Confirm ZK actually died from the IOException. Grep the ZK log for IOException, No space left on device, or Unreasonable length. The line will be the last entry before the process exited.
3. Determine whether the disk filled from txnlogs, snapshots, or something else. Compare du of the version-2 directories against other top-level entries on the same partition. If something else is consuming the space (app logs, core dumps), do not assume autopurge is the root cause.
4. Verify whether autopurge is enabled and how it is configured. A missing or zero autopurge.purgeInterval is disabled. The interval is in hours and the minimum effective value is 1.
5. On the leader, check whether the crashed node is counted out. Run echo mntr | nc <leader> 2181 | grep -E 'zk_learners|zk_synced_followers'. A 3-node ensemble tolerates one loss; the next loss breaks quorum.
6. Before restarting, plan recovery. If the log was being written when the disk filled, expect either a clean recovery or a crash-loop. Identify the most recent valid snapshot and the log files that will be replayed.

Metrics and signals to monitor

The page rule is df below 10% or 2GB on either partition, whichever is reached first. The headroom target is dataLogDir above 20% free and dataDir above 30% free.

Fixes

Free space before restarting

Stop ZooKeeper if it is still running and remove old logs and snapshots by hand. Keep at least the most recent valid snapshot and the log files that pair with it. Do not delete the current log file until you have decided whether to repair it.

# Identify the most recent valid snapshot
ls -lt /var/zookeeper/data/version-2/snapshot.* | head -3

# Identify txnlogs newer than the most recent snapshot - these will be replayed
ls -lt /var/zookeeper/txnlog/version-2/log.*

Warning: Deleting or moving the wrong log or snapshot files can make the node unrecoverable. Always identify the most recent valid snapshot first. Prefer moving files out of the directory over deleting them so you can restore if recovery fails. If the ensemble has no other healthy peer, copy the entire version-2 directory aside before touching anything.

If autopurge was simply disabled, removing old snapshot and log files beyond the retained set often frees tens of gigabytes. Always keep at least snapRetainCount (default 3) of the newest snapshots.

Recover from a crash-loop

If the node enters a restart loop after the disk is freed, the last log file is likely truncated. Two recovery paths:

• TxnLogToolkit ships with ZooKeeper and supports a -r (recover) mode that recalculates CRCs for broken transaction log entries. Try it on the suspected log file first. Note that “Unreasonable length” corruption (ZOOKEEPER-3975) stems from a corrupted record length field, not just a bad CRC, so TxnLogToolkit -r may not be able to repair that specific damage.
• Manual deletion: If TxnLogToolkit cannot repair the file, move the corrupted log out of the directory and let ZK recover from the previous valid snapshot plus the prior logs.

After either path, restart ZK and watch the log for Snapshotting, Loading snapshot, and the server entering FOLLOWING or LEADING state.

Confirm no silent data loss

Before allowing the recovered node back into the quorum, compare its zxid against the leader and the other followers. The values should be identical, or within a few transactions during catch-up.

# Compare zxid across the ensemble
echo mntr | nc <node> 2181 | grep zk_zxid
echo mntr | nc <leader> 2181 | grep zk_zxid

If the recovered node’s zxid is far behind or appears to come from an orphaned log (no matching snapshot), stop it and rebuild from a known-good snapshot taken from a healthy peer. Watch zk_digest_mismatches_count after the node rejoins.

Fix the root cause

Re-enable autopurge if it was disabled. Minimum sane defaults:

autopurge.snapRetainCount=3
autopurge.purgeInterval=1

If your write rate is high enough that logs accumulate faster than the hourly purge can keep up, autopurge alone cannot solve this. You need a larger partition, a reduction in write rate, or both. Consider enabling snapSizeLimitInKb (3.6+) so log rotation triggers on size as well as count.

If dataLogDir was not set, set it to a separate volume. This also fixes the snapshot-vs-txnlog I/O contention problem covered in the fsync warning guide.

Prevention

• Monitor df on both partitions and page at <10% or <2GB free. This is the single highest-value check and the only early warning before the crash.
• Maintain at least 3x the largest snapshot size free, plus 24 hours of txnlog growth at peak write rate. This is the headroom rule from the playbook.
• Confirm autopurge is configured on every node. Default purgeInterval=0 is disabled. The interval is in hours with a minimum of 1.
• Place dataLogDir on its own volume. Snapshot growth can no longer consume the txnlog partition.
• Track snapshot size growth. Growing snapshots mean a growing data tree, which means growing txnlog volume between snapshots.
• Track txnlog file count. Anything far above snapRetainCount + 5 with autopurge enabled means purge is broken or stuck.
• Test recovery on a staging node. Simulate a full partition by filling it with dd and walk through the recovery flow before you have to do it at 3 a.m.

How Netdata helps

• Disk space on dataLogDir and dataDir: page at <10% or <2GB free with per-second collection.
• zk_uptime resets: detect the crash immediately without polling ruok.
• zk_server_state: confirm whether quorum survived the node loss.
• zk_snapshot_error_count and zk_unrecoverable_error_count delta: leading indicators before the process exits.
• zk_digest_mismatches_count delta: the only signal for the silent data loss variant on rejoin.
• Correlate disk space, ZK log events, and zk_uptime on one timeline: identify “txnlog filled at 03:14, autopurge disabled, recovery crashed at 03:17” instead of just “ZK crashed”.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper GC pause cascade: how a Stop-the-World freeze expires sessions and re-elects the leader
• ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once
• ZooKeeper heap usage climbing: catching the GC death spiral before it starts