ZooKeeper request throttling: globalOutstandingLimit and TCP backpressure

zk_throttled_ops (ZooKeeper 3.7.0+) counts only the opt-in RequestThrottler dropping or stalling requests (zookeeper.request_throttle_max_requests, default 0 = disabled). It is not incremented by globalOutstandingLimit backpressure. The saturation alarm for the default path is zk_outstanding_requests pinned near globalOutstandingLimit (default 1000): when the queue reaches the limit, the server stops reading from client sockets, TCP backpressure propagates to every connected client, and any client that cannot absorb the added latency is on its way to a session expiration.

The throttling mechanism prevents unbounded request queuing from OOM-killing the JVM. It cannot fix whatever made the queue fill. The diagnostic question is never “should I raise the limit?” but “what is keeping requests in the queue longer than they should be?”

What this means

ZooKeeper’s request pipeline has a single submission queue per server. Every incoming request, read or write, enters this queue before being processed. The hard cap on outstanding requests is globalOutstandingLimit, configured via the zookeeper.globalOutstandingLimit Java system property and defaulting to 1000.

When the queue fills to that limit, the connection layer stops selecting client sockets for read. The server’s receive buffer fills, the kernel advertises a zero window, and the client’s send stalls. The server does not close the connection or return an error. It just stops reading.

There is no dedicated counter for the global-limit backpressure; the observable is zk_outstanding_requests sitting at or near the limit while clients see stalls. If the opt-in RequestThrottler is enabled, zk_throttled_ops increments on each drop or stall, and any non-zero rate on it is a binary saturation signal. Sessions whose heartbeats cannot get through during the stall eventually expire, cascading to ephemeral node deletion, watch fan-out, and downstream reconnection storms in Kafka, HBase, Solr, or anything else using ZooKeeper for coordination.

The upstream condition that made the queue fill is the incident: a slow fsync, a JVM GC pause, slow quorum ACKs, or a genuine write capacity ceiling.

flowchart TD
    A[Client requests arrive] --> B[Submission queue]
    B --> C{outstanding < 1000?}
    C -- yes --> D[Process request]
    C -- no --> E[Stop reading client sockets]
    E --> F[TCP zero-window to clients]
    E --> G[zk_outstanding_requests stays at limit]
    F --> H[Client latency, timeouts, session risk]
    D --> I{Pipeline keeping up?}
    I -- slow disk, GC, quorum --> B

Common causes

Quick checks

These commands are safe and read-only. They assume four-letter commands (mntr, srvr) are whitelisted, which is required from ZooKeeper 3.5.3 onward.

With the default metrics provider, mntr emits all of these keys: fsynctime/readlatency/updatelatency/quorum_ack_latency since 3.6.0, jvm_pause_time_ms since 3.6.2, and throttled_ops since 3.7.0. No metrics-provider configuration is required for mntr.

# Confirm throttling is active and identify the leader
echo mntr | nc localhost 2181 | grep -E 'zk_throttled_ops|zk_outstanding_requests|zk_server_state'

# Write-path health (separate read and write latency in 3.6+)
echo mntr | nc localhost 2181 | grep -E 'zk_(avg|p99)_(updatelatency|readlatency|fsynctime)'

# GC pressure
echo mntr | nc localhost 2181 | grep -E 'zk_(avg|p99)_jvm_pause_time_ms'

# Quorum ACK health (leader-only metrics)
echo mntr | nc localhost 2181 | grep -E 'zk_(avg|p99)_quorum_ack_latency|zk_pending_syncs|zk_synced_followers'

# Confirm the configured limit (path varies by distribution; may also be a JVM property in the start script)
grep -i globalOutstandingLimit /etc/zookeeper/zoo.cfg /opt/zookeeper/conf/zoo.cfg 2>/dev/null

# OS-level disk latency on the transaction log device
iostat -x 1 5

If mntr does not return metrics, verify that 4lw.commands.whitelist includes mntr. Whitelist at least mntr, ruok, and isro for monitoring.

How to diagnose it

1. Confirm saturation is real and current. With the opt-in RequestThrottler enabled, zk_throttled_ops is a counter; alert on rate, not absolute value. For the default globalOutstandingLimit path, zk_outstanding_requests pinned at or near 1000 with client TCP stalls is the equivalent signal.

2. Identify where the queue is filling. Query every ensemble member. If zk_outstanding_requests grows on the leader but stays at zero on followers, the leader’s write path is the bottleneck, typically a slow SyncRequestProcessor pipeline from disk fsync. If outstanding requests grow on all nodes simultaneously, suspect JVM GC, which freezes the entire process.

3. Check the leader’s fsync latency. This is the single most common root cause. The leader broadcasts proposals to followers concurrently with writing them to its own transaction log, but the SyncRequestProcessor processes writes sequentially, so slow fsync limits throughput. Followers must fsync each proposal before sending an ACK (forceSync defaults to true), putting follower disk latency directly on the quorum ACK path. zk_p99_fsynctime should be under 2 ms on dedicated SSD. Sustained values above 10 ms point at storage contention, cloud IOPS exhaustion, snapshot/txnlog sharing a disk, or hardware degradation.

4. Check JVM pause time. GC pauses freeze the process: no heartbeats, no request processing, no quorum ACKs. The queue drains on resume but refills immediately if the pause is rhythmic. Correlate zk_p99_jvm_pause_time_ms against latency spikes. If pauses approach a meaningful fraction of minSessionTimeout (default 2 x tickTime = 4000 ms), sessions are at risk.

5. Check quorum ACK latency. If the leader is healthy but followers are slow to ACK, the leader’s commit pipeline stalls waiting for quorum. zk_p99_quorum_ack_latency should stay well below syncLimit x tickTime (default 5 x 2000 ms = 10 seconds). Sustained elevation points at slow follower disks, follower GC, or network degradation between leader and followers.

6. Confirm the write rate. Compare zk_proposal_count rate (or zk_commit_count rate) against any benchmark you have for this hardware. If you are at or above benchmark with healthy disk and GC, you are genuinely capacity-bound and the architecture needs to change, not the limit.

zk_proposal_count and zk_commit_count are valid leader-side mntr counters (proposal_count, commit_count in ServerMetrics, 3.6+; leader only).

7. Rule out the thundering herd. A reconnection storm after a network blip or a deployment can temporarily saturate the pipeline. The signature is a V-shape in zk_num_alive_connections followed by a reconnection spike. These usually self-resolve but can tip the ensemble into a feedback loop if globalOutstandingLimit is too low for the burst.

Metrics and signals to monitor

Fixes

Disk-bound stalls (the most common case)

Put the transaction log on a dedicated, low-latency device. This is the single most impactful configuration change for ZooKeeper write performance. dataLogDir must point at a separate disk from dataDir. If they share a disk, snapshot writes compete with txnlog fsync and produce periodic latency spikes. In cloud environments, watch for gp2/gp3 burst credit exhaustion and provisioned IOPS limits; the cliff is abrupt.

GC-bound stalls

Size the heap correctly (not too large; full GC pause scales with heap). Switch to G1GC if you are still on CMS or Parallel. Consider ZGC on Java 15+. Disable Transparent Huge Pages and set vm.swappiness to 0 or 1 for ZooKeeper hosts. Enable GC logging so future stalls are diagnosable rather than mysterious.

Quorum-bound stalls

A single slow follower can stall the leader’s commit pipeline because the leader waits for quorum ACKs. Investigate the slow follower’s disk and GC separately. If network latency between datacenters is the cause, either tighten the topology or accept the higher baseline. zk_p99_quorum_ack_latency is leader-only, so monitoring must identify the leader and query it specifically. Leader-only metrics are invisible on followers, which is one of the most common monitoring gaps.

Genuine capacity overload

If disk, GC, and quorum are all healthy and throttling persists, you are at the leader’s write throughput ceiling. Options: move read traffic to observers (offloads reads but does not help writes), shard the workload across multiple ZooKeeper ensembles, or upgrade the leader’s storage. Adding nodes to a single ensemble is rarely the answer for write throughput, because every write still goes through the leader and still requires quorum ACK from more peers.

The temptation to raise globalOutstandingLimit

Raising globalOutstandingLimit buys queue headroom. It does not increase throughput. If the pipeline is saturating because fsync takes 50 ms, a deeper queue just means more requests waiting 50 ms longer. You will mask the symptom until the queue fills again, at which point throttling resumes with worse client-side latency and more heap pressure.

A deeper queue also makes recovery from a stall longer. When the underlying cause resolves (GC ends, disk I/O settles), the server has to drain a larger backlog before clients see improvement. There are narrow cases where raising the limit is correct, for example a known transient burst such as a deployment reconnection wave where you can absorb the queue depth and revert afterward. Treat any change here as a stopgap, document it, and set a reminder to revert.

The RequestThrottler (ZooKeeper 3.6+, zookeeper.request_throttle_max_requests, default 0 = disabled; drop-stall behavior via zookeeper.request_throttle_stall_time and zookeeper.request_throttle_drop_stale) is a separate, opt-in mechanism that sheds requests instead of backpressuring, and it is what increments zk_throttled_ops (3.7.0+). Raising globalOutstandingLimit does not change RequestThrottler behavior, and enabling RequestThrottler does not change the stop-reading backpressure path.

Prevention

• Alert on zk_throttled_ops rate as a binary saturation signal. Any non-zero sustained rate is a ticket, not a warning to ignore.
• Alert on zk_outstanding_requests as a leading indicator. Catch the queue growth before it hits the limit.
• Track the p99 latency family. zk_p99_fsynctime, zk_p99_jvm_pause_time_ms, and zk_p99_quorum_ack_latency are the three signals whose elevation predicts throttling.
• Keep dataLogDir on dedicated storage with monitored IOPS and burst credits. This is one line of config and prevents most write stalls.
• Capacity-test your write rate against your hardware and alert at roughly 60% of measured maximum.
• Treat every unplanned leader election as an incident. Elections are downstream of stalls that often begin as throttling.

How Netdata helps

• Per-second collection of zk_throttled_ops, zk_outstanding_requests, and the p99 latency family (updatelatency, readlatency, fsynctime, quorum_ack_latency, jvm_pause_time_ms) shows the queue filling before throttling starts.
• Anomaly detection on latency distributions flags the fsync or GC stall before zk_throttled_ops increments.
• Host-level metrics (disk await and %util, CPU iowait, network retransmits) on the same timeline as the ZooKeeper metrics make the disk-versus-GC-versus-quorum distinction immediate.
• Leader identification is automatic, so leader-only metrics like zk_pending_syncs and zk_quorum_ack_latency are always visible without manual filtering.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert