ZooKeeper proposals not committing: proposal_count outpacing commit_count

In a healthy ZooKeeper ensemble, every proposal the leader broadcasts is committed a few milliseconds later, after a quorum of followers acknowledges it. The two counters zk_proposal_count and zk_commit_count track the front and back of that pipeline, and on a cluster with active writers their rates should track each other closely.

When zk_proposal_count keeps climbing but zk_commit_count stalls, the leader is generating proposals but cannot reach quorum ACK. Writes are not landing. Clients with operation timeouts start failing, ephemeral nodes are not being created, and dependent systems (Kafka controller, HBase master, Solr overseer) start logging “operation timeout” or “session expired”. A worse signal is both metrics staying flat while writers are active: the write pipeline is fully blocked, or the leader is isolated from the quorum.

The signal pairs you need are zk_quorum_ack_latency, zk_synced_followers, zk_pending_syncs, and per-follower zk_fsynctime and zk_jvm_pause_time_ms. Both counters are leader-only; on ZK 3.6.0+ they are part of the standard mntr output, and on 3.4.x / 3.5.x they are not available via mntr (JMX only).

What this means

The ZAB protocol only commits a proposal after the leader receives an ACK from a quorum of servers, including itself. The leader’s proposal stage and commit stage are separate, so under contention the two rates can briefly diverge. A sustained divergence means one of:

• A quorum of followers cannot ACK fast enough (slow disk, GC, or network).
• The quorum has shrunk because followers dropped out.
• The leader believes it has quorum but is not actually committing (version-specific bugs).
• The leader is isolated from the writers (writes are not even reaching the proposal stage).

The shape of the divergence narrows the diagnosis:

A divergence for less than a second is normal burst behavior. Anything sustained beyond a few seconds during active write load is a problem.

Common causes

Quick checks

Run these on the leader first, then on each follower. They are all read-only.

# Identify the leader (run on every node)
echo srvr | nc localhost 2181 | grep Mode

# Confirm divergence is real (run twice, a few seconds apart)
echo mntr | nc localhost 2181 | grep -E 'zk_(proposal|commit)_count'

# Check quorum health on the leader
echo mntr | nc localhost 2181 | grep -E 'zk_(followers|synced_followers|pending_syncs)'

# Check quorum ACK latency on the leader
echo mntr | nc localhost 2181 | grep -E 'zk_.*quorum_ack_latency'

# Check write-mode status (rw = healthy, ro = quorum lost)
echo isro | nc localhost 2181

# Check pipeline saturation
echo mntr | nc localhost 2181 | grep -E 'zk_(outstanding_requests|sync_processor_queue_size)'

# Compare zxids across nodes (run on every node); mntr does not expose zxid
echo srvr | nc localhost 2181 | grep Zxid

# Check fsync latency on followers (the ACK bottleneck is often here)
echo mntr | nc localhost 2181 | grep -E 'zk_.*fsynctime'

# Check JVM pauses on followers
echo mntr | nc localhost 2181 | grep -E 'zk_.*jvm_pause'

zk_proposal_count, zk_followers, zk_synced_followers, zk_pending_syncs, and zk_*quorum_ack_latency* are leader-only. Querying a follower returns nothing for these. Either target the leader explicitly, or query all nodes and filter for the one whose Mode is leader.

How to diagnose it

1. Identify the leader. Run echo srvr | nc <host> 2181 | grep Mode on each node. If no node reports leader, you do not have a divergence problem, you have quorum loss. See ZooKeeper quorum loss.

2. Confirm the divergence is sustained. Sample zk_proposal_count and zk_commit_count at least three times over 15 seconds. Compute rates from deltas. The raw counters are monotonically increasing since process start, so absolute values are not useful.

3. Check quorum size and follower sync status. On the leader:

   • If zk_followers < N - 1, a follower is disconnected. Investigate the network or the missing follower’s process.
   • If zk_synced_followers < zk_followers, a connected follower has not caught up. Check zk_pending_syncs and the lagging follower’s disk and GC.
   • If zk_synced_followers equals floor(N/2), the ensemble is one failure away from going read-only.

4. Check quorum ACK latency. If zk_quorum_ack_latency is elevated (above single-digit milliseconds on a LAN, or trending upward), followers are ACKing slowly. mntr exposes it as avg/min/max/cnt/sum plus p50/p95/p99/p999 (3.6+). Cross-reference with per-follower zk_fsynctime and zk_jvm_pause_time_ms to find the slow follower.

5. Compare zxids across the ensemble. Run echo srvr | nc <host> 2181 | grep Zxid on every node (mntr does not expose zxid). The leader and synced followers should report identical zxids. A follower with a lower zxid is behind. zxid is a 64-bit value where the upper 32 bits are the epoch (incremented on each election) and the lower 32 bits are the transaction counter, so compare within the same epoch first.

6. Check for version-specific bugs if the previous steps do not localize the problem:

   • ZK 3.5.8 or 3.6.1 with a recent ensemble expansion: check the leader log for “Have quorum of supporters” without subsequent commits. This is the allowedToCommit=false bug, ZOOKEEPER-3830, fixed in 3.5.9, 3.6.2, and 3.7.0.
   • ZK 3.6.4 in environments where DNS may not be ready at startup (notably Kubernetes): check the log for “Couldn’t bind to /”. This is ZOOKEEPER-4728, fixed in 3.8.4, 3.9.2, and 3.10.0 (ZOOKEEPER-5026, a related unresolved issue, remains open with no fix version).

7. Check whether writers can reach the leader. If both proposal_count and commit_count are flat while clients are reporting errors, the leader may be isolated from clients even though it has internal quorum. That is a different failure from the leader being unable to reach followers.

flowchart TD
    A[proposal rate > commit rate on leader] --> B{synced_followers below expected?}
    B -- Yes --> C[Degraded quorum]
    B -- No --> D{quorum_ack_latency elevated?}
    D -- Yes --> E[Follower disk, GC, or network]
    D -- No --> F{Both rates zero with active writers?}
    F -- Yes --> G[Leader isolated from clients]
    F -- No --> H[Check version-specific commit bugs]
    C --> C1[Network partition or follower crash]
    E --> E1[Follower fsync, JVM pause, or RTT]

Metrics and signals to monitor

Fixes

Triage by the cause the diagnosis pointed at. None of these should be the first action during an outage; the first action is to identify which subsystem is failing so you do not mask the root cause.

Slow follower disk

If a follower’s zk_fsynctime is the bottleneck for quorum ACK, the txnlog disk is saturated. Short term, identify and stop whatever is competing for I/O on that disk (backups, co-located workloads). Structural fixes:

• Configure dataLogDir on a dedicated low-latency device, separate from dataDir.
• On cloud, increase provisioned IOPS or move the txnlog to a higher tier.
• Confirm autopurge.purgeInterval and autopurge.snapRetainCount are set so log accumulation is not contributing.

Restarting the slow follower will not fix the disk; it will only trigger a SNAP sync when it rejoins, which loads the leader further. See ZooKeeper fsync warning for the related write-stall pattern.

Follower GC pauses

If zk_jvm_pause_time_ms (3.7+) on a follower is high, the follower’s JVM is freezing during ACKs. Address the heap: enable GC logging, confirm heap is sized for the data tree, and consider moving from CMS or Parallel to G1GC, or to ZGC on JDK 15+. Confirm Transparent Huge Pages is set to never on the host. GC pauses are also a common trigger for unnecessary leader elections, which compounds the original divergence.

Degraded quorum

If zk_followers or zk_synced_followers is below expected, the fix is to restore the missing follower: restart the process, fix the network, free disk, or address whatever caused it to drop. Do not force a leader re-election hoping it will help; re-election loses in-flight proposals and lengthens the outage. See ZooKeeper server stuck in LOOKING when a follower cannot rejoin.

Network degradation between leader and followers

If zk_quorum_ack_latency is high but no single host shows disk or GC problems, suspect the network. Check inter-node RTT, packet retransmissions on the quorum ports, and any recent security group or firewall change. Both the follower-to-leader port and the election port must be reachable; a blocked election port is silent until the next election.

Version-specific commit bugs

If the diagnosis points at allowedToCommit=false (ZK 3.5.8 or 3.6.1, ZOOKEEPER-3830) or DNS binding stuck (ZK 3.6.4, ZOOKEEPER-4728), the only durable fix is to upgrade to a release that contains the fix: 3.5.9, 3.6.2, or 3.7.0 for the allowedToCommit bug; 3.8.4, 3.9.2, or 3.10.0 for the DNS binding bug (ZOOKEEPER-5026, a follow-up issue, remains open). As a workaround for the DNS binding bug in Kubernetes, Strimzi documented using 0.0.0.0 for the local server address in server.X configuration, or delaying ZooKeeper startup until the pod hostname resolves (strimzi/strimzi-kafka-operator#8675).

Leader isolated from clients

If both rates are flat and clients are timing out, the leader may be unreachable from clients even though it has internal quorum. Check client-side error rates, load balancer or DNS health, and network ACLs between client subnets and the leader’s client port (default 2181).

Prevention

• Alert on the rate gap, not the absolute counters. Compute rate(zk_proposal_count[1m]) - rate(zk_commit_count[1m]) and alert on any sustained non-zero value during active write load.
• Alert on zk_synced_followers at the quorum threshold. For an N-node ensemble, page when zk_synced_followers == floor(N/2), because one more failure stops writes.
• Track zk_quorum_ack_latency as a leading indicator. A creeping baseline predicts the next divergence incident.
• Keep ZK patched. The allowedToCommit and DNS binding bugs are silent until triggered by ensemble expansion or a pod restart; stay on a fixed release.
• Run mixed-version checks before expanding the ensemble. Adding a node running a version affected by ZOOKEEPER-3830 is a known trigger.
• Confirm dataLogDir is on dedicated storage on every node. The most common slow-ACK cause is shared disk on a follower.
• Exercise follower restarts in a controlled window. SNAP sync behavior under load is something you want to have observed before an incident, not during one.

How Netdata helps

• Per-second collection of zk_proposal_count and zk_commit_count surfaces the rate gap within seconds of onset, rather than at a 30- or 60-second scrape interval where short divergences are invisible.
• Leader-aware collection surfaces zk_followers, zk_synced_followers, zk_pending_syncs, and zk_quorum_ack_latency from whichever node currently holds the role, so you do not need to know the leader ahead of time.
• ML anomaly detection on the rate gap and on zk_quorum_ack_latency flags slow-burn divergences before they become sustained stalls.
• Correlation across nodes lets you put the leader’s quorum signals next to each follower’s zk_fsynctime and zk_jvm_pause_time_ms in one view, which is usually the fastest path to identifying the slow follower.
• Per-host dashboards make the zxid comparison straightforward, since each node’s zk_zxid is collected alongside its role.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper read latency high: memory reads that should never be slow
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum