ZooKeeper quorum ack latency high: followers slow to acknowledge proposals

zk_p99_quorum_ack_latency is climbing on the leader and every write in the ensemble is paying for it. This metric measures the time from the leader sending a PROPOSE message to receiving quorum acknowledgments from followers. It is a leader-only signal.

The metric ships in the metrics framework introduced in ZooKeeper 3.6.0; on earlier versions it does not exist.

Every ZooKeeper write blocks until a quorum of followers ACK. An ACK means the follower has written the proposal to its transaction log and fsync’d it to persistent storage. So quorum ack latency bundles two things into one number: the inter-node network round trip, and the follower’s fsync plus processing time. When this metric is elevated, the slowest follower in the quorum is bottlenecking writes for every client connected to the ensemble.

On a healthy LAN, p99 quorum ack latency should be under 5ms. The hard ceiling is syncLimit x tickTime (10 seconds with the sample settings tickTime=2000 ms and syncLimit=5). If a follower takes longer than that window to acknowledge, the leader drops it from the quorum. In a 3-node ensemble, dropping one follower puts you at the quorum threshold, where a single additional failure means total write unavailability. The goal is to isolate which follower is slow and whether the cause is disk, GC, or network, before the follower gets ejected.

What this means

ZAB is a two-phase commit. The leader sends PROPOSE, followers append the proposal to their transaction log and fsync, then return ACK. Once a quorum of followers ACK, the leader sends COMMIT and applies the change to its in-memory data tree. The zk_quorum_ack_latency metric captures the PROPOSE-to-quorum-ACK window.

What sits inside that window:

• Network transit: PROPOSE travels from leader to follower, and ACK travels back. On a LAN this is sub-millisecond and rarely the bottleneck.
• Follower fsync: the follower must append to its transaction log and call fsync before ACKing. This is the dominant component on most deployments and the most common cause of elevated ack latency.
• Follower processing: request thread scheduling, commit processor queue depth, JVM safepoints, and GC pauses.

If the slowness persists past syncLimit x tickTime, the leader ejects the follower. Watch for zk_synced_followers declining and zk_looking_count incrementing as downstream signals of that ejection.

flowchart TD
    A["zk_p99_quorum_ack_latency elevated on leader"] --> B{"Per follower:
zk_max_fsynctime elevated?"}
    B -->|"Yes, one follower"| C["Single follower disk I/O
check iostat on that host"]
    B -->|"Yes, multiple followers"| D["Shared storage tier problem
or correlated co-located load"]
    B -->|"No"| E{"zk_p99_jvm_pause_time_ms
spiking on a follower?"}
    E -->|"Yes"| F["Follower GC pauses
stalling ACKs"]
    E -->|"No"| G["Network degradation
between leader and follower"]
    C --> H["Without intervention:
follower approaches syncLimit x tickTime
and gets dropped from quorum"]
    F --> H
    G --> H

Common causes

Quick checks

Run these on the leader first, then on each follower as needed. All are read-only.

# Identify the current leader
echo srvr | nc localhost 2181 | grep Mode

# Quorum ack latency percentiles (leader-only metric)
echo mntr | nc localhost 2181 | grep zk_.*quorum_ack_latency

# Follower count, synced followers, pending syncs (leader-only)
echo mntr | nc localhost 2181 | grep -E 'zk_(followers|synced_followers|pending_syncs)'

# Follower sync time percentiles
echo mntr | nc localhost 2181 | grep zk_.*follower_sync_time

# Fsync latency (max/avg; fsync has no percentile variant in mntr) (run on each follower host)
echo mntr | nc localhost 2181 | grep -E 'zk_(avg|max)_fsynctime'

# JVM GC pause time percentiles (run on each follower host)
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# Split write vs read latency to confirm write path is the bottleneck
echo mntr | nc localhost 2181 | grep -E 'zk_p99_(update|read)latency'

# OS-level disk latency on the transaction log device
iostat -x 1 5

# Confirm the node is read-write, not stuck in read-only after quorum loss
echo isro | nc localhost 2181

If mntr returns empty, four-letter commands may not be whitelisted. Check 4lw.commands.whitelist in zoo.cfg (required since ZK 3.5.3+).

How to diagnose it

1. Confirm you are on the leader. Quorum ack latency is leader-only. Run echo srvr | nc localhost 2181 | grep Mode and ensure it returns leader. On a follower the metric will be absent.

2. Determine whether one follower or all followers are slow. On the leader, check zk_synced_followers and zk_pending_syncs. If synced_followers is below ensemble_size - 1, a follower has already been ejected or is lagging. Sustained non-zero pending_syncs means followers cannot keep up with the write rate.

3. On each follower, check fsync latency. Run echo mntr | nc localhost 2181 | grep -E 'zk_(avg|max)_fsynctime'. On dedicated SSD, max under 2ms is a reasonable baseline. If one follower’s fsync is elevated but the others are normal, that follower’s disk is the bottleneck. If all followers show elevated fsync, the problem is a shared storage tier or correlated co-located I/O.

4. On each follower, check JVM pause time. Run echo mntr | nc localhost 2181 | grep zk_p99_jvm_pause_time_ms. GC pauses freeze the follower and prevent ACKs. If ack latency spikes are rhythmic and correlate with GC pause spikes, the follower’s heap or GC algorithm is the issue.

5. If fsync and GC are both normal, check the network. Measure RTT between the leader and the slow follower. Look for TCP retransmits on the inter-ensemble connection. Cross-datacenter deployments will have a higher ack latency baseline proportional to RTT, which may be expected rather than pathological.

6. Check whether a follower was recently restarted. zk_follower_sync_time spikes during catch-up after rejoin. This is transient and should resolve on its own. If it does not, the follower may be doing a full SNAP sync, which is I/O-intensive on both sides and can degrade leader performance.

7. Check correlated write latency. zk_p99_updatelatency should track quorum ack latency closely. If update latency is elevated but quorum ack latency is normal, the bottleneck is on the leader’s side (leader fsync or commit processing), not the followers.

Metrics and signals to monitor

Fixes

Slow follower disk (fsync bottleneck)

This is the most common cause. The follower cannot fsync proposals fast enough, so every ACK is delayed.

• Separate dataLogDir from dataDir. If the transaction log and snapshots share a disk, snapshot writes compete with fsync. Put dataLogDir on a dedicated low-latency device. This requires a rolling restart to take effect.
• Check for co-located I/O. Other processes writing to the same disk (backups, batch jobs, monitoring agents) steal IOPS. Identify the culprit with iostat -x and iotop.
• Cloud storage throttling. On EBS gp2/gp3, burst credit exhaustion causes sudden fsync cliffs. Increase provisioned IOPS or switch to a higher-performance tier.
• Disk hardware degradation. Check SMART errors and sector reallocation rates. A degrading SSD shows progressively worse fsync times.

Follower JVM GC pauses

GC pauses freeze the follower entirely: no heartbeats, no request processing, no ACKs.

• Check heap sizing. If the data tree has grown, the follower may need more heap. Monitor zk_znode_count and zk_approximate_data_size against heap capacity.
• Switch GC algorithm. ZooKeeper ships no GC flags: the JVM default applies (G1 has been the default since JDK 9).

On JDK 15+, ZGC provides sub-millisecond pauses. CMS and Parallel GC are problematic for large heaps and should be avoided.

• Enable GC logging. Without -Xlog:gc* output, you are guessing. GC logs confirm whether pauses correlate with ack latency spikes.

Network degradation between leader and followers

If fsync and GC are normal on all followers but ack latency is elevated, the network is the suspect.

• Measure RTT between the leader and each follower. On a LAN, expect sub-millisecond.
• Check for TCP retransmits on the inter-ensemble ports using netstat -s | grep retransmit or ss -ti.
• Cross-datacenter deployments have inherently higher ack latency proportional to RTT. If your ensemble spans datacenters, recalibrate your baseline. The 5ms LAN guideline does not apply.
• Check for bandwidth saturation. If replication traffic fills the link, SNAP sync events (full data tree transfer to a lagging follower) create a feedback loop that makes it worse.

Metrics provider lock contention

In some ZooKeeper versions, the built-in PrometheusMetricsProvider can cause lock contention under heavy write load, artificially inflating latency metrics. Thread dumps show CommitProcWorkThread blocked inside the metrics quantile computation.

Reported as ZOOKEEPER-4741: JIRA lists the 3.6.x line (3.6.0 through 3.6.4) as affected, and the fix landed only in 3.10.0.

• Check the ZK version. If running a 3.6.x version and the symptom appeared after enabling Prometheus metrics, test whether the issue resolves on a patched release.
• Upgrade. If the version is affected, upgrade to a release that includes the fix (3.10.0+).

Prevention

• Dedicated transaction log disk. dataLogDir on its own device, separate from snapshots. This is the single highest-impact configuration change for write latency stability.
• Monitor fsync per follower, not just the aggregate. A single slow follower is invisible if you only watch the leader’s averages.
• Track zk_p99_quorum_ack_latency against syncLimit x tickTime. Alert before the metric approaches the ejection threshold, not after the follower is already gone.
• Size heap for the data tree. Monitor zk_znode_count and zk_approximate_data_size growth. GC pressure from an undersized heap produces rhythmic ack latency spikes.
• Enable autopurge. autopurge.purgeInterval and autopurge.snapRetainCount prevent transaction log accumulation that slows recovery and competes for I/O.
• Test follower ejection and rejoin. Know how your ensemble behaves when a follower is dropped and rejoins, before it happens during a real incident.

How Netdata helps

• Netdata collects zk_quorum_ack_latency percentiles at per-second resolution, so you see the exact moment latency departs from the healthy LAN baseline rather than waiting for a slow scrape interval.
• Because the metric is leader-only, correlating it with zk_server_state labels lets you filter to the active leader automatically, without manually switching scrape targets during a failover.
• Pairing zk_quorum_ack_latency on the leader with zk_fsynctime and zk_jvm_pause_time_ms on each follower isolates whether the bottleneck is a single follower’s disk, a single follower’s GC, or the inter-node link.
• zk_synced_followers and zk_pending_syncs on the leader show whether the slow follower is about to be ejected from quorum, elevating a latency degradation into an availability risk.
• ML-based anomaly detection flags the gradual upward drift in fsync or ack latency that precedes a follower drop, before the metric crosses a fixed threshold.

Related guides

• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper unexpected leader election: finding why the leader dropped
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum
• ZooKeeper leader election storm: an ensemble that keeps re-electing