ZooKeeper outstanding requests growing: the request pipeline is backing up

zk_outstanding_requests counts requests queued in the server’s request processor pipeline that have not yet completed. In steady state it sits at or near zero. When it climbs, something downstream in the pipeline has stopped draining faster than clients are submitting.

The queue is a leading indicator. Requests pile up before zk_avg_latency reacts, and well before the server starts dropping client traffic. If you wait for latency alerts, you have already lost the lead time needed to keep dependent services like Kafka, HBase, and Solr from feeling the stall.

This is a field guide for triaging a rising outstanding-requests counter on an Apache ZooKeeper ensemble. The fastest path to root cause depends on which nodes are affected and what the rest of the pipeline is doing. We will work through the three patterns that cover nearly every real incident: a leader-only stall (disk or quorum bottleneck), an all-nodes stall (usually GC), and a follower-only stall (a slow commit stream from the leader).

What this means

When zk_outstanding_requests is non-zero, the server is admitting requests faster than it can complete them. The queue lives inside the request processor pipeline between the connection handler and the final response. Reads queue there, writes queue there, and on followers, forwarded writes also queue there while they wait for the leader to commit.

Three structural facts drive triage:

1. The queue is shared across request types. Reads (local memory lookups) and writes (quorum plus fsync) share the same counter. A read-biased node can show a spike during a GC pause even if the write path is fine.
2. The queue is per-server, not per-ensemble. You have to look at every node individually. Leader and followers behave differently because only the leader assigns zxids, writes its txnlog as the source of truth, and broadcasts proposals.
3. The queue has a hard ceiling. When it reaches globalOutstandingLimit (default 1000), the server stops reading from client sockets. TCP backpressure propagates to clients, which experience timeouts and, if the stall outlives the session timeout, session expirations. zk_throttled_ops is a real mntr counter on 3.7+; on 3.6 the equivalent counter is zk_request_throttle_wait_count.

A globalOutstandingLimit hit can happen in well under a second during a severe GC stop-the-world pause. By the time a human notices, the queue may already be draining, or sessions may already be dying. The artifact you want is per-second metric history, not a five-minute average.

The diagnostic flow narrows root cause from topology in one step.

flowchart TD
    A["outstanding_requests climbing"] --> B{"Which nodes?"}
    B -->|"Leader only"| C["Disk or quorum bottleneck"]
    B -->|"All nodes"| D["JVM GC pause"]
    B -->|"Followers only"| E["Slow leader"]
    C --> C1["zk_p99_fsynctime"]
    C --> C2["zk_quorum_ack_latency"]
    D --> D1["zk_jvm_pause_time_ms p99"]
    E --> E1["leader outstanding_requests"]
    E --> E2["proposal vs commit rate"]

Common causes

Quick checks

Run these read-only checks across every ensemble member. The pattern of which nodes show the symptom is the diagnosis.

# Check outstanding requests and server role, per node
echo mntr | nc localhost 2181 | grep -E 'zk_outstanding_requests|zk_server_state'

# Check whether we are already throttling - non-zero means the limit was hit
echo mntr | nc localhost 2181 | grep zk_throttled_ops

# Check write path health on the leader
echo mntr | nc localhost 2181 | grep -E 'zk_.*fsynctime|zk_.*updatelatency'

# Check quorum ACK latency (leader-only)
echo mntr | nc localhost 2181 | grep zk_.*quorum_ack_latency

# Check follower sync queue depth (leader-only)
echo mntr | nc localhost 2181 | grep -E 'zk_(learners|synced_followers|pending_syncs)'

# Check JVM GC pause time, all percentiles
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# Check proposal vs commit throughput - divergence means a quorum problem
echo mntr | nc localhost 2181 | grep -E 'zk_(proposal|commit)_count'

# Check read latency - reads stay fast in a disk stall, slow under GC
echo mntr | nc localhost 2181 | grep zk_.*readlatency

# Check OS-level disk saturation on the txnlog device
iostat -x 1 5

# Live GC view of the ZK JVM (run as the ZK user or root)
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 5

If mntr returns nothing on ZooKeeper 3.5.3+, the four-letter command is not whitelisted. Add mntr (plus ruok, isro, srvr) to 4lw.commands.whitelist and restart, or query the AdminServer HTTP endpoint on port 8080 instead. The default whitelist in 3.5.3+ enables only srvr, so mntr requires an explicit change.

How to diagnose it

Diagnosis is driven by the topology of the symptom, not the absolute value of the counter. Any sustained non-zero value with active traffic is abnormal, but the fix depends on which nodes are affected.

1. Pull outstanding_requests from every node at the same instant. Compare leader and follower values. The ensemble topology is the entire diagnosis.

   • Leader non-zero, followers zero: leader-local bottleneck, almost always disk or quorum.
   • All nodes non-zero: systemic cause, almost always GC.
   • Followers non-zero, leader zero: slow commit stream from the leader.

2. If only the leader is affected, chase the write path. Read zk_p99_fsynctime and zk_p99_quorum_ack_latency on the leader. If fsync is elevated, the leader’s txnlog disk is the bottleneck. If quorum ACK latency is elevated but fsync is fine, followers are slow to ACK (their own disk, GC, or network). Reads on followers should stay fast in either case, which confirms the problem is in the write path.

3. If all nodes are affected, look for a JVM-level freeze. Pull zk_p99_jvm_pause_time_ms from each node. A stop-the-world pause stops request processing while the network layer keeps accepting traffic, so the queue fills on every node at the same instant. If the pauses are rhythmic (matching GC cycle frequency) and disk I/O looks normal during the event, GC is the cause. Confirm with GC logs.

4. If only followers are affected, look at the commit stream. Followers forward writes to the leader and wait for commit messages back. If the leader’s own outstanding_requests is zero but followers are climbing, the leader is producing commits slower than followers are submitting writes. Check zk_proposal_count vs zk_commit_count on the leader: divergence means proposals are not being committed (quorum ACK problem). Check zk_pending_syncs on the leader for replication lag.

5. Confirm severity against the limit. Compute zk_outstanding_requests / globalOutstandingLimit. Above roughly 0.7, you are one burst away from throttling. Once zk_throttled_ops increments, clients are already experiencing TCP backpressure and will start timing out.

6. Correlate with downstream signals. If zk_stale_sessions_expired or zk_connection_drop_count are also moving, clients have already started losing state. Kafka broker sessions and HBase RegionServer ephemeral nodes are at risk.

7. Cross-check against known noise. Brief outstanding-request spikes during snapshot creation, rolling restart elections, and the reconnection wave after a leader election are expected. Gate non-critical alerts on zk_uptime > 300 seconds to suppress cold-start artifacts.

Metrics and signals to monitor

Fixes

Triage by cause. All of these assume you have already identified which pattern you are in via the steps above.

Leader txnlog disk stall

This is the most common cause and the one most likely to take down dependent services. The leader blocks on fsync for every write, so a slow disk stalls the entire write path.

• Confirm with iostat -x 1 5 on the dataLogDir device. await and %util are the smoking gun.
• If dataLogDir is not configured, or shares a device with dataDir, move it to a dedicated low-latency device. Snapshots are bulk sequential writes and contend directly with fsync. This is the highest-impact single-line config change for ZooKeeper write performance.
• If the device is shared with another workload (colocated application, backup job, monitoring agent writing local metrics), move the workload or move the txnlog.
• If this is cloud storage (AWS EBS gp2/gp3, GCP persistent disk), check burst credit balance and provisioned IOPS. Burst credit exhaustion produces a sudden latency cliff after a period of normal performance.
• Verify disk health. Increasing sector reallocation rates in SMART data indicate failing hardware.
• Watch for the snapshot-vs-txnlog pattern. Spikes that align with snapshot rotation (every snapCount transactions, default 100,000) almost always mean shared disk.

See ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls and ZooKeeper write latency high: read zk_updatelatency, not just avg_latency for deeper write-path diagnosis.

JVM GC pause

A stop-the-world pause freezes every ZooKeeper thread simultaneously. The queue fills because the server cannot process requests while the JVM is collecting, even though the network layer keeps accepting them.

• Confirm by pulling zk_p99_jvm_pause_time_ms and cross-referencing the GC log (-Xlog:gc*:file=... on Java 9+, or -XX:+PrintGCDetails on Java 8).

• Check heap size against the data tree. If zk_znode_count and zk_approximate_data_size have crept up over time, the live data set may now dominate the heap. Increase heap or clean up the tree.

• Check the GC algorithm. CMS is deprecated and fragments badly on large heaps. G1GC (the default on Java 9+) is better. ZGC (Java 15+ production-ready) gives sub-millisecond pauses for large heaps. A rolling restart is required to change GC settings.

• Check Transparent Huge Pages: cat /sys/kernel/mm/transparent_hugepage/enabled. THP can multiply GC pause durations by 2-10x. To disable for ZK hosts (system-wide change, requires root, not persistent across reboot until you write a systemd-tuned or rc.local entry):

  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  

• Verify the JVM is not swapping. Swapping makes GC pauses catastrophic because the collector must page in objects to scan them. Set vm.swappiness to 0 or 1 (persistent via /etc/sysctl.d/) and ensure RSS stays within resident memory.

Slow leader (follower-only outstanding)

Followers queue forwarded writes while they wait for commit messages. A zero on the leader and a rising value on every follower means the commit stream cannot keep up with the proposal stream.

• On the leader, compare zk_proposal_count rate to zk_commit_count rate. If proposals are growing but commits are stalled, the quorum ACK path is the bottleneck.
• Pull zk_quorum_ack_latency (leader-only). Elevated ACK latency means followers are slow to ACK, which means their own fsync, GC, or network path is the problem.
• Pull zk_pending_syncs (leader-only). Sustained non-zero values mean followers cannot keep up with the write rate. This is ZooKeeper’s replication-lag signal.
• Compare zxids across the ensemble. They should be identical within a few transactions. A persistent delta on one follower points to that follower’s local problem (disk, GC, network to leader).
• Check whether a follower is in SNAP sync (full snapshot transfer from leader). SNAP sync consumes leader resources and reduces its capacity during the transfer.

Saturation from client traffic

If none of the above causes apply, you may simply be exceeding the leader’s write capacity. Writes serialize through the leader, so adding followers does not help write throughput.

• Measure sustained write rate against the leader’s maximum (from load tests or historical peaks). Above roughly 80% of capacity, queuing begins.
• If you have observers configured, verify client read traffic is actually routing to them. A misconfigured load balancer sending all traffic to the leader defeats the purpose.
• Consider whether the workload belongs in ZooKeeper at all. Large znode data (near the jute.maxbuffer default of 1MB), heavy writes, or frequent large multi-operations are anti-patterns.

Throttling already active

If zk_throttled_ops is non-zero, clients are already being subjected to backpressure.

• Do not raise globalOutstandingLimit without understanding the cause. The limit exists to prevent the server from running out of memory by queuing unbounded requests. Raising it trades latency for memory risk.
• The proper fix is to address the underlying bottleneck so the queue drains naturally.
• Temporarily increasing the limit to absorb a reconnection storm is a legitimate tactic, but it must be reversed once the storm passes.

Prevention

• Monitor outstanding_requests per node, not aggregated. Aggregating leader and followers hides the topology that is the diagnosis. Alert on any node’s sustained non-zero value.
• Treat the queue as a leading indicator. Alert on sustained non-zero outstanding requests with active traffic, not on zk_avg_latency thresholds. By the time latency moves, the queue has already filled.
• Monitor fsync and JVM pause percentiles. zk_p99_fsynctime and zk_p99_jvm_pause_time_ms are the two underlying causes that produce nearly every outstanding-request incident. Without them, root cause takes hours.
• Put dataLogDir on a dedicated device. This eliminates the snapshot-vs-txnlog contention pattern and is the single most impactful write-path config change.
• Track zk_znode_count and zk_approximate_data_size. Data-tree bloat leads to heap pressure and GC death spirals over months.
• Enable GC logging on every ZK JVM. -Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m on Java 9+ gives you the artifact you need when GC is suspected.
• Configure autopurge. Set autopurge.purgeInterval and autopurge.snapRetainCount to prevent txnlog accumulation from filling the partition and crashing the server.
• Set ulimits explicitly. ulimit -n 65536 or higher in the ZK startup script or container spec. Container defaults are often far too low.
• Test failover deliberately. Kill a ZK server in a controlled environment and confirm the ensemble re-elects, clients reconnect, and monitoring catches the election. This is the gap that turns a small stall into a major outage.

How Netdata helps

Netdata’s per-second collection turns a queue depth metric that can spike and resolve in under a second into something you can actually see after the fact.

• Per-second zk_outstanding_requests per node, with the leader labeled, lets you read the diagnosis directly off the chart: leader-only, all-nodes, or follower-only.
• Correlate queue depth with zk_p99_fsynctime, zk_p99_jvm_pause_time_ms, zk_throttled_ops, and zk_stale_sessions_expired on the same timeline to identify the underlying cause in seconds rather than minutes.
• ML anomaly detection flags the leading-indicator rise before a latency threshold would fire, which preserves the lead time that makes this metric valuable.
• Leader-only metrics (zk_followers, zk_synced_followers, zk_pending_syncs, zk_quorum_ack_latency) are filtered correctly so dashboards show them where they exist instead of hiding them on follower nodes.
• Host-level disk and JVM signals (per-device %util and await, JVM GC pause times, RSS vs heap) appear alongside ZooKeeper metrics, removing the need to pivot between tools during triage.
• Composite dashboards for the GC cascade, disk-sync deadlock, and session-expiration storm patterns let you recognize the failure archetype at a glance.

For more on the broader mental model, see How ZooKeeper actually works in production: a mental model for operators.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum
• ZooKeeper split-brain: two nodes both reporting leader
• ZooKeeper unexpected leader election: finding why the leader dropped
• ZooKeeper write latency high: read zk_updatelatency, not just avg_latency