ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius

A follower that fell too far behind the leader does not catch up transaction by transaction. Once its last-seen zxid is older than the leader’s retained transaction log, the leader ships the entire data tree as a snapshot. This is a SNAP sync, the most expensive recovery path a healthy ensemble runs short of a leader election.

The blast radius is not limited to the recovering follower. While the leader serializes and transmits the full tree, its request pipeline competes for CPU and network. Clients writing to the leader see elevated latency for the duration. The recovering follower is meanwhile unavailable for reads: it is loading a snapshot, not serving requests. In a three-node ensemble, a SNAP sync temporarily cuts read capacity by a third and stretches write latency on the only node that handles writes.

SNAP syncs are not bugs. They are the recovery path ZooKeeper uses when DIFF sync (replay the missing transactions) is no longer possible because the leader has already purged those transactions. The operator’s job is not to prevent every SNAP sync, but to make them rare, short, and non-cascading.

What this means

ZAB Phase 2 synchronization has three modes the leader picks between when a follower reconnects:

• DIFF: the leader still holds the missing transactions in its in-memory committedLog or on-disk transaction log. It replays them. Cheap, common, fast.
• TRUNC: the follower has transactions the leader does not (for example from a brief split-brain). The leader tells the follower to roll back. Also cheap.
• SNAP: the leader no longer has what the follower needs. It serializes the entire in-memory DataTree and streams it as a snapshot. Expensive on both ends.

The leader compares the follower’s last processed zxid against its retained transaction log. If the gap exceeds what the log covers, SNAP is the only option.

flowchart TD
  F[Follower zxid behind retained log] -->|triggers| L[Leader selects SNAP sync]
  L --> S[Leader serializes full DataTree]
  S -->|CPU contention| LD[Leader write latency rises for ALL clients]
  L -->|streams snapshot| FR[Follower receives snapshot]
  FR --> FL[Follower deserializes into heap]
  FL -->|during load| NR[Follower serves NO reads]
  LD --> Q[Outstanding requests grow on leader]

The leader-side cost is twofold: serializing the data tree is CPU work that competes with the single-threaded request pipeline, and transmitting the snapshot is sustained network output that competes with PROPOSE, ACK, and COMMIT traffic. The follower-side cost: it must deserialize the snapshot into heap before it can participate in the ensemble, and during that window it serves no reads.

Common causes

Quick checks

All safe, read-only operations. On ZooKeeper 3.5+, the 4lw commands (mntr, srvr) require 4lw.commands.whitelist to be set in zoo.cfg.

# Identify the leader and followers
echo mntr | nc localhost 2181 | grep zk_server_state
echo srvr | nc localhost 2181 | grep Mode

# Compare zxids across the ensemble. A large gap is the SNAP sync trigger.
for host in zk1 zk2 zk3; do
  echo "$host: $(echo srvr | nc $host 2181 | grep '^Zxid')"
done

# On the leader: see how many followers are fully synced
echo mntr | nc leader 2181 | grep -E 'zk_(followers|synced_followers|pending_syncs)'

# SNAP sync log signature on the leader and the follower
grep -E "Sending snapshot|SNAP" /var/log/zookeeper/zookeeper.log | tail -20
grep -E "Loading snapshot|Snapshotting" /var/log/zookeeper/zookeeper.log | tail -20

# Confirm autopurge and snapshot retention configuration
grep -E "autopurge|snapRetainCount|snapCount|dataLogDir" /etc/zookeeper/zoo.cfg

# Confirm initLimit and syncLimit; SNAP syncs can blow past initLimit
grep -E "^(tickTime|initLimit|syncLimit)" /etc/zookeeper/zoo.cfg

# Watch leader-side disk and network output during the transfer
iostat -x 1
sar -n DEV 1

How to diagnose it

1. Confirm a SNAP sync is actually in progress. The leader log should show “Sending snapshot” and the follower log should show “Loading snapshot” or “Snapshotting”. If neither appears, you may be looking at slow DIFF sync or a follower stuck in LOOKING.

2. Confirm the follower is the laggard. Compare the last-processed zxid across all ensemble members (the srvr four-letter-word command reports Zxid; mntr does not expose zxid). The follower in SNAP sync will have a zxid frozen at an old value while the leader continues advancing.

3. Assess the blast radius on the leader. While the SNAP is in flight, watch zk_outstanding_requests, zk_avg_updatelatency (or zk_avg_latency on pre-3.6), and zk_fsynctime on the leader. Any sustained climb that begins when “Sending snapshot” appears and resolves when the follower rejoins is the SNAP sync degrading leader performance.

4. Confirm the recovering follower is not serving reads. Its zk_num_alive_connections may still be non-zero because clients reconnect to it, but reads will fail or stall until the snapshot is loaded.

5. Estimate remaining sync time. There is no built-in progress metric. The signal is the follower’s zxid (from srvr) jumping to the leader’s current value when NEWLEADER is processed. Until that jump, assume the sync is ongoing.

6. Check for cascading SNAP syncs. If maxConcurrentSnapSyncs (default 10) allows multiple simultaneous SNAP syncs and several followers restart at once, the leader can be saturated by serialization work alone.

Metrics and signals to monitor

Fixes

During an active SNAP sync

The first lever is traffic shaping on the leader. maxConcurrentSnapSyncs (default 10) caps how many followers the leader will fully sync at once; additional followers wait. Lowering this value protects a leader with constrained I/O or network during mass-restart scenarios, at the cost of slower ensemble convergence. The companion knob maxConcurrentDiffSyncs (default 100) does the same for DIFF syncs.

The second lever is client routing. If the leader’s zk_outstanding_requests and write latency are climbing during the transfer, shift read traffic away from the leader and away from the recovering follower. The remaining healthy followers should absorb reads.

Do not restart the leader to “fix” a SNAP sync in progress. You will abort the sync, force a new election, and the new leader will start the snapshot over.

Preventing recurrence

Tune autopurge carefully. autopurge.snapRetainCount defaults to 3 and the minimum is 3. Lower is not permitted, and going higher than needed keeps more logs on disk without operational benefit. Pair it with a sane autopurge.purgeInterval (in hours); leaving it at 0 disables purge, which is the path to disk-full incidents.

Size initLimit for your data tree. initLimit (in ticks) bounds how long followers have to connect and sync to a newly-elected leader. If a SNAP transfer takes longer than initLimit * tickTime, the leader abandons leadership and the ensemble re-elects. With defaults (initLimit=10, tickTime=2000ms), the budget is 20 seconds. A large snapshot with hundreds of thousands of znodes can exceed that on slower disks. Increase initLimit proportionally to snapshot size; do not change tickTime to compensate.

Watch snapCount and transaction log size. The default snapCount of 100,000 triggers snapshots based on transaction count. Two related knobs help when log files balloon: zookeeper.snapSizeLimitInKb (default 4,194,304 KB, i.e. 4 GB; a non-positive value disables it) triggers snapshots based on log size, and zookeeper.txnLogSizeLimitInKb (disabled by default) caps individual log file size. The admin guide explicitly warns that larger transaction logs slow follower sync.

Handling the oversized-transaction-log trap

If you have raised snapCount substantially (for example to 10,000,000) to reduce snapshot frequency, you may have fallen into ZOOKEEPER-3757 (still open): the leader underestimates retained transaction log size and chooses DIFF replay over SNAP even when SNAP would be cheaper. The symptom is sync times jumping from a few seconds to 20+ seconds, with the leader replaying a very large transaction log instead of shipping a smaller snapshot. The workaround is zookeeper.forceSnapshotSync=true, a testing-only system property (present in current source, not in the admin guide).

Version-specific sync hazards

Several recent ZooKeeper versions have SNAP and DIFF sync bugs that affect data integrity. Before treating a SNAP sync as routine, confirm you are not running a vulnerable version.

• 3.9.3 only: A DIFF sync bug (ZOOKEEPER-4925) introduces a hole in committedLog, leading to data loss. Fixed in 3.9.4 and 3.10.0.
• 3.9.0 through 3.9.3: Lock contention between snapshotting and the sync operation on follower servers (ZOOKEEPER-4858) slows SNAP syncs. Fixed in 3.9.4.
• 3.8.x before 3.8.4, 3.9.x before 3.9.2: A race in DIFF sync (ZOOKEEPER-4785) where the follower updates its epoch and ACKs NEWLEADER before persisting uncommitted transactions. A crash at that point can lose data.
• 3.5.x before 3.5.10, 3.6.x before 3.7.0: Data inconsistency when the leader crashes after sending SNAP but before sending NEWLEADER (ZOOKEEPER-3642). The follower advances lastProcessedZxid without persisting the snapshot.

If you observe zk_digest_mismatches_count incrementing after a sync on any of these versions, treat it as a symptom of the bugs above, not transient corruption.

Prevention

• Keep transaction logs sized for DIFF sync. Use snapCount, snapSizeLimitInKb, and txnLogSizeLimitInKb together so the leader can almost always satisfy a returning follower with DIFF rather than SNAP.
• Size initLimit for your worst-case snapshot. Measure actual snapshot load times in pre-prod and set initLimit * tickTime comfortably above the p99.
• Avoid long follower downtimes during maintenance. Drain and restore followers quickly; the longer one is out, the more likely its zxid falls off the retained log.
• Upgrade off vulnerable versions. The current stable releases are 3.8.6 and 3.9.5; both carry the sync-related data integrity fixes above.
• Run with a separate dataLogDir. Shared txnlog and snapshot disks make SNAP syncs slower on both ends and increase the chance the sync blows past initLimit.
• Instrument leader-side latency during any planned follower restart. A routine rolling restart that triggers SNAP syncs is the most common way teams discover their snapRetainCount is too low or their transaction logs are oversized.

How Netdata helps

• Per-second zk_server_state, zk_synced_followers, and zk_pending_syncs let you see the SNAP sync window with tight time resolution, instead of discovering after the fact that the leader’s write latency degraded for two minutes.
• zk_follower_sync_time and zk_zxid per node, collected from every ensemble member and filterable to the leader, show exactly when a follower fell behind and how long catch-up took.
• Correlating leader write latency (zk_avg_updatelatency, zk_p99_updatelatency) with zk_outstanding_requests during a SNAP sync tells you whether the leader’s pipeline is saturated by serialization work or merely busy.
• Host-level disk and network metrics on the leader and follower, alongside ZooKeeper metrics, isolate whether the bottleneck is leader CPU, leader network output, follower disk, or follower deserialization.
• Anomaly detection on zk_fsync_threshold_exceed_count and zk_digest_mismatches_count surfaces the integrity risks described in the version-specific bugs above without waiting for a human to notice a counter moving.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals