ZooKeeper autopurge not configured: snapshots and logs filling the disk over months

autopurge.purgeInterval defaults to 0, meaning snapshots and transaction logs accumulate forever. On a quiet ensemble the growth is slow enough that nobody notices for months, then the dataLogDir partition hits 100%, ZooKeeper cannot fsync the next write, and the process dies. The leader throws an IOException on the transaction log and the ensemble loses a member, or quorum if more than one node fills simultaneously.

This article covers how to confirm autopurge is the cause, clean up safely without forcing followers into expensive SNAP syncs, and set autopurge.purgeInterval and autopurge.snapRetainCount so it does not recur.

What this means

ZooKeeper persists state in two on-disk artifacts:

• Transaction logs (log.<zxid>) in dataLogDir (or dataDir if dataLogDir is not set). Every mutation is appended here and fsync’d before acknowledgment.
• Snapshots (snapshot.<zxid>) in dataDir. The full in-memory data tree serialized to disk roughly every snapCount transactions (default 100,000).

Without autopurge, neither artifact is ever deleted. Snapshots pile up proportional to write throughput, and transaction logs accumulate between snapshots. Transaction log files are pre-allocated in 64MB chunks (zookeeper.preAllocSize, default 65536KB), so even modest write traffic accumulates full 64MB files at every rotation.

Two config keys control cleanup:

• autopurge.purgeInterval - interval in hours between purge runs. 0 (the default) disables it.
• autopurge.snapRetainCount - minimum snapshots (and associated logs) to retain. Default 3, which is also the minimum supported value.

The diagnostic signature: snapshot file count far exceeds snapRetainCount, log.* file count far exceeds what one retention window would need, and disk usage trends monotonically upward.

flowchart TD
  A[autopurge.purgeInterval = 0] --> B[Snapshots never deleted]
  A --> C[Txn logs never deleted]
  B --> D[Disk usage climbs over months]
  C --> D
  D --> E[Partition hits 100%]
  E --> F[fsync fails on next write]
  F --> G[IOException, ZK process dies]
  G --> H[Ensemble loses a member, or quorum]

Common causes

Quick checks

All safe read-only operations.

# Confirm whether autopurge is configured at all
grep -E 'autopurge\.(purgeInterval|snapRetainCount)' /path/to/zoo.cfg

# Disk usage on the two partitions that matter
df -h /var/zookeeper/txnlog /var/zookeeper/data

# Snapshot file count and total size (dataDir/version-2)
ls /var/zookeeper/data/version-2/snapshot.* 2>/dev/null | wc -l
du -sh /var/zookeeper/data/version-2

# Transaction log count and total size (dataLogDir/version-2)
ls /var/zookeeper/txnlog/version-2/log.* 2>/dev/null | wc -l
du -sh /var/zookeeper/txnlog/version-2

# Newest and oldest snapshot, to see the time span being retained
ls -lt /var/zookeeper/data/version-2/snapshot.* | head -3
ls -lt /var/zookeeper/data/version-2/snapshot.* | tail -3

# Verify the running process start time (autopurge is read at startup)
ps -o lstart= -p $(pgrep -f QuorumPeerMain)

# Confirm ZK is still healthy (4lw must be whitelisted in 3.5.3+)
echo isro | nc localhost 2181
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_znode_count|zk_approximate_data_size'

# Check ZK logs for purge activity or disk write failures
grep -E 'Purge task|IOException|No space left on device' /var/log/zookeeper/zookeeper.log | tail -30

The single most telling signal is the relationship between snapshot count and autopurge.snapRetainCount. A healthy node with autopurge running has roughly snapRetainCount snapshots plus a small number of transaction logs. A node with autopurge disabled has dozens or hundreds of snapshots and a transaction log count that grows without bound.

How to diagnose it

1. Confirm the symptom is accumulation, not data tree growth. If zk_approximate_data_size is also climbing fast, the root cause may be application misuse of ZooKeeper as a database. See the related guide on data size growth. If zk_approximate_data_size is stable but disk usage climbs, this is a retention problem.
2. Check whether dataLogDir and dataDir share a partition. If they do, snapshot growth compounds transaction log growth, and you also have fsync contention waiting to surface. Separate them when you fix this.
3. Verify the config is present and non-zero. Empty output from the grep above is the most common finding. A purgeInterval of 0 is equivalent to absent.
4. Verify the process is running with the config loaded. autopurge.* keys are read at startup. If you added them but never restarted, they are not in effect.
5. Check for Windows or Solr embedded-ZK edge cases if the config looks correct but accumulation continues.
6. Rule out external writers. A colocated backup job, log shipper, or monitoring agent writing to the same partition produces the same disk symptom. du -sh on sibling directories confirms.

Metrics and signals to monitor

Fixes

Enable autopurge

Add or uncomment both keys in zoo.cfg:

autopurge.purgeInterval=1
autopurge.snapRetainCount=3

purgeInterval is in hours, accepts only positive integers. 1 is reasonable for ensembles with steady write traffic. snapRetainCount of 3 is the default and minimum supported value. Do not set it below 3.

Restart ZooKeeper on each node, one at a time, confirming quorum between restarts. Autopurge parameters are read at startup; the running process will not pick them up from a config reload.

Why snapRetainCount must be at least 3

A follower that was offline briefly (maintenance, crash, GC pause) catches up by replaying transaction logs from its last snapshot forward. If the logs it needs have been purged because snapRetainCount is too low, the follower cannot catch up incrementally. The leader must send the entire data tree as a SNAP sync: the leader serializes and transmits the full tree, the follower is unavailable for reads during transfer, and write latency on the leader can degrade while the snapshot is sent.

snapRetainCount=3 keeps enough history that brief follower outages resolve with incremental log replay instead of a full snapshot transfer. See the related guide on SNAP sync blast radius.

Clean up accumulated files safely

If the disk is already critically full, do not wait for autopurge. Use the bundled PurgeTxnLog utility (also wrapped by bin/zkCleanup.sh in some distributions) to do a controlled purge respecting the same retention rules:

# DESTRUCTIVE: deletes old snapshots and transaction logs.
# Stop the ZK process on this node first, or confirm autopurge is disabled,
# to avoid races with the in-process purge task.
java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.PurgeTxnLog \
  /var/zookeeper/txnlog /var/zookeeper/data -n 3

The -n argument is the retain count. Use the same value as autopurge.snapRetainCount. Run it one node at a time, rolling, so the ensemble never loses redundancy.

Avoid manual rm. PurgeTxnLog understands which logs are referenced by which snapshots and will not orphan a snapshot from the transaction log entries it needs for consistent recovery.

Address configuration management

Check all ensembles. Any deployment not explicitly configured is vulnerable. Common gaps:

• Helm charts that do not expose autopurge in values.yaml
• Container images that set ZOO_AUTOPURGE_PURGEINTERVAL=0 explicitly
• Ansible/Chef/Puppet roles copied from a template that never included the keys
• Embedded ZooKeeper inside Solr or another product, where the parent product’s docs do not surface the ZK-level config

Edge cases that survive correct configuration

• Windows. ZOOKEEPER-2844 documents autopurge logging success without deleting files on Windows Server. File handle locking is the suspected cause. The JIRA remains open and unresolved through current 3.8.x and 3.9.x releases. On Windows, schedule an external PurgeTxnLog run.
• Solr embedded ZooKeeper. Solr’s SolrZKServer does not honor autopurge.snapRetainCount or autopurge.purgeInterval. Move to an external ensemble, or run PurgeTxnLog externally on a schedule.

Prevention

• Treat autopurge config as required. Every production ensemble should have both keys explicitly set. A missing key is a deployment defect, not a neutral default.
• Alert on snapshot count, not just disk percentage. Disk percentage alerts get snoozed because they climb slowly. A snapshot count above snapRetainCount + 5 for more than one purge interval is a low-noise signal that purge is broken.
• Alert on dataLogDir free space in absolute terms. The failure mode is pre-allocation failure when the partition is nearly full. Alert at 20% free, not 10%, because the 64MB pre-allocation can fail before df shows zero.
• Separate dataLogDir and dataDir on different volumes. This improves fsync latency and prevents snapshot accumulation from triggering transaction log write failures.
• Track zk_approximate_data_size independently. If disk growth is from data tree bloat rather than missing purge, fixing autopurge will slow but not stop the problem.
• Include autopurge verification in deployment checklists. After every install or manifest change, confirm the keys are present, non-zero, and the process has been restarted since they were added.

How Netdata helps

• Per-second disk usage metrics on the dataDir and dataLogDir partitions surface the slow monotonic climb that percentage-based alerts miss. Long-term retention makes the “growing over months” pattern visible at a glance.
• Correlation with zk_approximate_data_size, zk_znode_count, and disk I/O latency lets you confirm in one view whether disk growth is a retention problem (autopurge) or a workload problem (data tree bloat).
• Alerts on partition free space fire before the partition is full, giving you the window needed to enable autopurge and restart safely.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper GC pause cascade: how a Stop-the-World freeze expires sessions and re-elects the leader
• ZooKeeper OutOfMemoryError: Java heap space - the OOM that kills the whole ensemble at once
• ZooKeeper heap usage climbing: catching the GC death spiral before it starts
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning