ZooKeeper follower sync time climbing: a follower approaching ejection

When zk_avg_follower_sync_time or zk_max_follower_sync_time starts climbing, a follower is taking longer to process proposals from the leader. The metric measures how close a follower is to being ejected from the quorum.

The hard ceiling is syncLimit x tickTime. With defaults of syncLimit=5 and tickTime=2000ms, that ceiling is 10 seconds. When a follower’s sync time approaches that limit, the leader closes the connection, stops pushing updates, and the follower must re-enter leader discovery. In a 3-node ensemble, that drops you to minimum quorum with zero remaining fault tolerance.

Follower sync time tracks the full round-trip: leader sends PROPOSE, follower writes to its transaction log, follower fsyncs, follower ACKs, leader counts the ACK toward quorum. A climb means one of those steps is getting slower on the follower side. The usual suspects are follower disk I/O, inter-node network degradation, and follower GC pauses.

Correlate the follower’s sync time with the leader’s zk_quorum_ack_latency (elevated too, because quorum ACK includes the slow follower) and with the follower’s own zk_fsynctime. The combination tells you which subsystem is guilty.

What this means

ZooKeeper’s ZAB protocol requires the leader to get quorum acknowledgment before it commits a transaction. Every follower that participates in quorum must receive the proposal, fsync it to its transaction log, and send an ACK back to the leader within syncLimit x tickTime. If a follower cannot do that within the window, the leader declares it unsynchronized.

flowchart TD
    A["Follower sync time rises"] --> B["Quorum ACK latency rises on leader"]
    B --> C["zk_synced_followers drops"]
    C --> D["syncLimit x tickTime exceeded"]
    D --> E["Leader ejects follower"]
    E --> F["Follower re-enters discovery"]
    F --> G["Quorum tolerance reduced"]
    G --> H["One more failure = quorum loss"]

zk_follower_sync_time should stay well below syncLimit x tickTime. The warning zone is a meaningful fraction of that limit. Do not wait until the metric approaches 10 seconds to act. By the time sync time is in the multiple-second range, ejection is imminent.

On the leader’s side, zk_synced_followers drops when a follower is struggling. In a 3-node ensemble, zk_synced_followers should be 2. If it drops to 1, the leader is at minimum quorum and any additional failure breaks the cluster. Page when zk_synced_followers equals floor(ensemble_size/2), because the next failure causes a complete outage.

After ejection, the follower must reconnect and resync. If it fell far enough behind, the leader may require a SNAP sync (full data tree transfer), which is expensive for both nodes and degrades leader performance for all clients during the transfer.

Common causes

ZOOKEEPER-4858 removes lock contention between snapshotting and the sync operation and is fixed in 3.9.4, so 3.9.0-3.9.3 are the affected releases when running under snapshot load.

Quick checks

All read-only and safe to run during an incident.

# Confirm which node is the leader (leader-only metrics require this)
for host in zk1 zk2 zk3; do echo "$host: $(echo srvr | nc -w 2 $host 2181 | grep Mode)"; done

# Check follower sync time on the leader
echo mntr | nc localhost 2181 | grep zk_.*follower_sync_time

# Check synced followers count (leader-only)
echo mntr | nc localhost 2181 | grep -E 'zk_(followers|synced_followers|pending_syncs)'

# Check quorum ACK latency on the leader
echo mntr | nc localhost 2181 | grep zk_.*quorum_ack_latency

# On the suspect follower, check fsync time
echo mntr | nc localhost 2181 | grep zk_.*fsynctime

# On the suspect follower, check JVM pause time
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# Compare zxids across the ensemble (replication lag)
for host in zk1 zk2 zk3; do echo "$host: $(echo mntr | nc -w 2 $host 2181 | grep zk_zxid)"; done

# Check follower disk I/O saturation
iostat -x 1 5

# Check follower heap usage
jcmd $(pgrep -f QuorumPeerMain) GC.heap_info

Note: in ZooKeeper 3.5.3+, four-letter-word commands (mntr, srvr, etc.) must be whitelisted via the 4lw.commands.whitelist property in zoo.cfg. If mntr is blocked, you will get “mntr is not executed because it is not in the whitelist.” Whitelist at least mntr and srvr for monitoring.

JVM pause time (jvm_pause_time_ms) has been exposed through mntr since ZooKeeper 3.6.2 as zk_avg_jvm_pause_time_ms, zk_max_jvm_pause_time_ms, zk_p99_jvm_pause_time_ms, etc.; on earlier versions it is JMX-only.

How to diagnose it

1. Identify the leader. Run srvr on each node. Only the leader reports zk_followers, zk_synced_followers, zk_pending_syncs, and the quorum ACK latency metrics. If you query a follower, you will not see these and may miss the problem entirely.

2. Confirm the follower is falling behind. On the leader, check zk_synced_followers against ensemble_size - 1. If it is lower, a follower is not in sync. Check zk_pending_syncs: sustained non-zero means followers cannot keep up with the write rate.

3. Pinpoint which follower is slow. Compare zk_zxid across all ensemble members. A follower with a zxid meaningfully behind the leader is the one struggling. The zxid is a 64-bit number where the upper 32 bits are the epoch and the lower 32 bits are the transaction counter. Within the same epoch, a lower counter means the follower is behind.

4. Check the slow follower’s fsync time. On that follower, run echo mntr | nc localhost 2181 | grep zk_.*fsynctime. If zk_avg_fsynctime or zk_p99_fsynctime is elevated (above single-digit milliseconds on SSD), the follower’s disk is the bottleneck. The follower must fsync each proposal before it can ACK, so slow fsync directly inflates sync time.

5. Check the slow follower’s JVM pause time. Run echo mntr | nc localhost 2181 | grep zk_.*jvm_pause. If GC pause times are spiking, the follower is freezing during collection and cannot process proposals or send ACKs. GC pauses approaching syncLimit x tickTime will cause ejection.

6. Check network between leader and the slow follower. If fsync and GC look normal on the follower, the network path is the next suspect. Check for packet loss, retransmits, and latency. The Apache Troubleshooting wiki documents cases where NIC misconfiguration caused high TCP packet loss that manifested as replication lag.

7. Check whether a SNAP sync is in progress. If the follower recently rejoined after downtime or restart, it may be receiving a full snapshot transfer. During SNAP sync, the leader serializes and sends the entire data tree, which is I/O and network intensive. This is expected but degrades leader performance for all clients during the transfer. Confirm by checking the ZooKeeper log for “Sending snapshot” on the leader or “Loading snapshot” on the follower.

8. Check the version. If you are running ZooKeeper 3.9.0 through 3.9.3, a known bug (ZOOKEEPER-4858, fixed in 3.9.4) caused lock contention between snapshotting and the sync operation, which could manifest as higher follower sync times under snapshot load.

Metrics and signals to monitor

Fixes

Follower disk I/O

If the follower’s zk_fsynctime is elevated, the transaction log disk is the bottleneck.

Check dataLogDir configuration. If dataLogDir is not set, the transaction log shares a disk with snapshots (dataDir). This is the most common single-cause misconfiguration. Set dataLogDir to a dedicated, low-latency device and do a rolling restart.

Check for I/O contention. If another workload is on the same disk (backups, logs, a colocated service), move it. Cloud burst credit exhaustion on EBS gp2/gp3 volumes can cause abrupt fsync latency cliffs. Check IOPS limits and burst credits.

Measure the disk directly. Run iostat -x 1 5 on the txnlog disk. If %util is near 100 or await is high, the disk cannot keep up. Measure raw fsync latency:

# Measure write+fsync latency on the txnlog disk (-W does O_DIRECT writes)
ioping -c 10 -s 4k -W /path/to/dataLogDir/

Inter-node network degradation

If fsync and GC look normal but sync time is still climbing, the network path between the leader and the slow follower is the problem.

Check packet loss and retransmits. Run ping between the leader and the slow follower. Check ss -ti or netstat -s for TCP retransmit counts. Even a few percent packet loss on a high-throughput replication link can inflate sync time significantly.

Check for NIC misconfiguration. The Apache Troubleshooting wiki documents a case where misconfigured NICs caused high TCP packet loss that affected a subset of clients. Verify duplex and speed settings, and check switch port error counters.

Check cross-datacenter latency. If the ensemble spans datacenters or availability zones, the network RTT sets a floor on quorum ACK latency. zk_quorum_ack_latency should stay well below syncLimit x tickTime. Cross-AZ deployments have higher baselines than single-rack ensembles.

Follower GC pauses

If zk_jvm_pause_time_ms on the follower is elevated, GC is freezing the process during collection.

Check heap usage. If the follower’s heap is above 85% sustained, the GC runs frequently and pauses grow. Use jcmd $(pgrep -f QuorumPeerMain) GC.heap_info to inspect.

Check the GC algorithm. ZooKeeper’s own scripts set no GC default; the JVM’s default applies (G1GC on JDK 9 and newer, Parallel GC on JDK 8). Older deployments on CMS or Parallel GC produce longer stop-the-world pauses. Consider ZGC (production-ready from JDK 15+) for sub-millisecond pauses.

Check heap sizing. If the data tree has grown (check zk_znode_count and zk_approximate_data_size), the heap may be undersized. Increase heap or address znode accumulation.

Check for Transparent Huge Pages. THP on Linux can inflate GC pauses through khugepaged compaction and memory-charging stalls; JVM latency guidance (Oracle, Azul, database vendor runbooks) recommends disabling it on latency-sensitive hosts. Check cat /sys/kernel/mm/transparent_hugepage/enabled. If it is not never, disable it for ZooKeeper hosts.

Version-specific issues

If you are running ZooKeeper 3.9.0 through 3.9.3, ZOOKEEPER-4858 (fixed in 3.9.4) caused lock contention between snapshotting and the sync operation. This could manifest as higher follower sync times under snapshot load. Upgrading to 3.9.4 or later eliminates this contention.

ZOOKEEPER-4925 (diff sync introduced a hole in a stale follower’s committedLog, causing data loss in the leader; fixed in 3.9.4) is triggered by any stale follower doing a diff sync after rejoin, not by ejection specifically. Operators on 3.9.3 who see follower re-syncs should treat this as urgent.

Prevention

• Put the transaction log on a dedicated disk. Set dataLogDir to a separate, low-latency device. This is the single most impactful configuration change for write-path stability and prevents snapshot I/O from competing with fsync.

• Monitor fsync latency, not just disk space. Track zk_fsynctime p99 on every ensemble member. Fsync latency is the leading indicator for both write stalls and follower sync degradation.

• Monitor GC pause times. Track zk_jvm_pause_time_ms p99 on every ensemble member. GC pauses approaching syncLimit x tickTime will cause follower ejection.

• Separate leader and follower monitoring thresholds. The leader and followers have fundamentally different workloads. The leader handles all writes and tracks follower sync state. Monitor the leader’s quorum ACK latency and synced follower count with different thresholds than follower-level metrics.

• Keep ZooKeeper current. The 3.9.x line has had multiple sync-related fixes. If you are running 3.9.0 through 3.9.3, plan an upgrade to 3.9.4 or later.

• Test failover. Deliberately kill a follower in a controlled environment and verify the ensemble re-elects and resyncs within acceptable timeframes. This validates both the system and your monitoring.

How Netdata helps

Netdata collects these signals per second, which is the resolution at which ejection events actually unfold.

• Per-second follower sync time and quorum ACK latency. The relationship between zk_follower_sync_time on the leader and zk_quorum_ack_latency is the primary diagnostic pair. Per-second resolution shows the moment sync time starts diverging from baseline, before it reaches the ejection threshold.

• Follower-side fsync and JVM pause metrics. Netdata collects zk_fsynctime and zk_jvm_pause_time_ms with percentile breakdowns on every node. When a follower’s sync time climbs, switch to that node’s view to see whether the follower’s disk or GC is the cause.

• Leader-only metrics with automatic role detection. zk_followers, zk_synced_followers, and zk_pending_syncs only appear on the leader. Netdata identifies the leader automatically and surfaces these metrics without manual filtering.

• Cross-node zxid comparison. Replication lag is visible as a zxid delta between the leader and followers. Netdata’s per-node collection makes this comparison immediate.

• Anomaly detection on sync-related metrics. Netdata flags unusual patterns in follower sync time, quorum ACK latency, and fsync time before they cross hard thresholds, which catches gradual degradation that static alerts miss.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals