$ guides / zookeeper / zookeeper-stale-requests-dropped ▌

Operations Guides

ZooKeeper stale requests dropped: requests aging out of the pipeline

zk_stale_requests_dropped incrementing is a late signal in a ZooKeeper saturation cascade. By the time a request ages out of the pipeline and is dropped, the server has already exhausted queue headroom, engaged throttling, and held the request long enough that the client gave up or the connection died. Treat any non-zero rate as an incident, and treat it as proof that an earlier signal was missed.

The metric is a counter pair. zk_stale_requests counts requests the server marked as stale; zk_stale_requests_dropped counts the subset that were discarded instead of processed. Both should be zero in steady state. When they move, the write pipeline stalled long enough to outlive a request’s useful lifetime. The two dominant causes are transaction-log fsync saturation and long JVM GC pauses.

This article walks the diagnostic path from the stale-drop counter back to root cause, using the leading indicators (zk_outstanding_requests, zk_throttled_ops, zk_fsynctime, zk_jvm_pause_time_ms) that should have caught the problem first.

What this means

ZooKeeper’s request pipeline, in its throttled form introduced in 3.6.0 via the RequestThrottler, classifies a request as stale when one of two conditions holds: the client connection that submitted it has closed, or the request’s time in the pipeline has exceeded its session timeout. The checks are controlled by two boolean options, both 3.6.0 additions: requestStaleConnectionCheck (default true) and requestStaleLatencyCheck (default false). When requestThrottleDropStale is enabled (default true), the throttler discards stale requests instead of handing them to the processor chain.

The stale-drop counter is therefore not a primary signal. It is the back end of a cascade:

flowchart TD
    A["Write request arrives"] --> B["Request enters pipeline"]
    B --> C{"fsync or GC stall?"}
    C -->|yes, sustained| D["zk_outstanding_requests grows"]
    D --> E{"Queue nears globalOutstandingLimit?"}
    E -->|yes| F["zk_throttled_ops increments"]
    F --> G{"Stall outlives request timeout?"}
    G -->|yes| H["zk_stale_requests increments"]
    H --> I["zk_stale_requests_dropped increments"]
    C -->|no| J["Normal processing, counters flat"]

Because zk_stale_requests_dropped only moves after the queue has already filled and throttling has engaged, a non-zero value almost always means your alerting on earlier signals is missing or thresholded too loosely. The diagnostic job is to identify which upstream stall (disk, GC, or both) held the pipeline long enough to age requests out.

Common causes

Cause	What it looks like	First thing to check
Transaction-log fsync saturation	`zk_fsynctime` p99 elevated (often >10ms, sometimes seconds), write latency tracks fsync, reads on followers stay fast	`iostat -x` on the `dataLogDir` device; leader `zk_fsynctime`
Long JVM GC pause	`zk_jvm_pause_time_ms` p99 spikes, both read and write latency spike together, pause frequency matches staleness bursts	GC log; `zk_jvm_pause_time_ms`
Compound disk + GC pressure	fsync and GC both degraded, `zk_outstanding_requests` never drains, elections may follow	Both `zk_fsynctime` and `zk_jvm_pause_time_ms` elevated simultaneously
Shared or co-located txnlog disk	fsync spikes during snapshot creation or co-located workload I/O	Whether `dataLogDir` is set and on a dedicated device

Quick checks

All read-only and safe to run during an incident.

# Stale request counters (should be flat / zero-delta)
echo mntr | nc localhost 2181 | grep zk_stale_request

# Leading indicators that should have fired first
echo mntr | nc localhost 2181 | grep -E 'zk_outstanding_requests|zk_throttled_ops'

# Functional state - is the node even serving writes?
echo isro | nc localhost 2181

# OS-level disk saturation on the txnlog device
iostat -x 1 5

# GC log for Stop-the-World pauses (path varies by deployment)
grep -E "Pause (Full|Young)" /var/log/zookeeper/gc.log | tail -30

mntr includes the percentile breakdowns on 3.6+ (for example zk_p99_fsynctime and zk_p99_jvm_pause_time_ms); the AdminServer /commands/monitor and the Prometheus endpoint expose the same metrics.

If mntr returns nothing, confirm 4lw.commands.whitelist includes mntr (required since 3.5.3). On 3.6+ the AdminServer exposes the same metrics over HTTP when four-letter commands are locked down.

How to diagnose it

Confirm the counter is actually moving. Stale-drop counters are cumulative. Compute a delta over a short window rather than alerting on the absolute value. A single historical burst keeps the counter elevated forever.
Locate the node. Run the quick checks on every ensemble member. Stale drops on the leader point at the write path (fsync or proposal pipeline). Stale drops on a follower point at local processing (usually GC) or at the follower being unable to forward writes to a stalled leader.
Separate disk from GC. This is the most important fork.
- If zk_fsynctime p99 is elevated and zk_jvm_pause_time_ms is flat, the disk is the bottleneck. Reads on followers will typically stay fast because reads do not fsync.
- If zk_jvm_pause_time_ms p99 is elevated and fsync is flat, GC is freezing the pipeline. Both read and write latency spike together during each pause.
- If both are elevated, you have a compound stall. Either alone might be tolerable; together they age requests out.
Check the queue and throttle relationship. zk_outstanding_requests should be near zero in steady state. If it sits at or near globalOutstandingLimit (default 1000) and zk_throttled_ops is incrementing, the pipeline is saturated. Stale drops are the downstream consequence.
Confirm whether the stall has escalated. Cross-check whether the stall triggered an election (leader state transitions, zk_looking_count), whether clients lost sessions (zk_stale_sessions_expired), and connection drops. If any of these moved in the same window, saturation has already cascaded beyond stale requests.
Verify throttle configuration. If you have explicitly enabled requestStaleLatencyCheck (defaults to false) or set zookeeper.request_throttle_max_requests (default 0, which disables throttling), confirm the values are intentional. An aggressively low throttle limit with latency-based staleness checking can produce stale drops under load that the default config would absorb by queueing.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`zk_stale_requests_dropped`	Late-stage saturation signal; proves the pipeline aged a request out	Any non-zero delta
`zk_outstanding_requests`	Leading indicator; queue fills before latency spikes	Sustained non-zero, especially approaching 1000
`zk_throttled_ops`	Backpressure has engaged; server stopped reading client sockets	Any non-zero rate
`zk_fsynctime` (p99)	Root cause metric for write stalls	Trend upward; p99 >10ms on dedicated SSD
`zk_jvm_pause_time_ms` (p99)	Root cause metric for GC freezes	p99 approaching a fraction of `tickTime` (2000ms)
`zk_updatelatency` (p99)	Confirms the write path is degraded, separated from reads	p99 climbing, tracking fsync or quorum ack
`zk_looking_count`	Tells you whether the stall triggered an election	Increment outside maintenance
`zk_stale_sessions_expired`	Tells you whether clients lost state	Any non-zero rate

Fixes

fsync saturation (the most common cause)

Start here. Shared or slow transaction-log storage is the number-one cause of write stalls, and stale drops are the downstream symptom.

Put dataLogDir on a dedicated device. If dataLogDir is unset, the txnlog shares dataDir with snapshots, and snapshot I/O competes with fsync. This is the highest-impact fix available.
Check cloud storage throttling. EBS gp2/gp3 burst credit exhaustion produces an abrupt fsync latency cliff. Confirm provisioned IOPS against actual write rate.
Check co-located I/O. Anything else writing to the txnlog device (backups, monitoring agents, other services) will inflate fsync p99.
Confirm forceSync is not disabled. Setting forceSync=no removes the fsync and is dangerous for durability.

These require a rolling restart to take effect. Do not restart as a first reflex during an active stall. Diagnose first, schedule the change, and roll one node at a time.

Long GC pauses

If GC is the cause, the fix is heap and collector tuning, not disk work.

Check heap sizing against the data tree. Use zk_znode_count and zk_approximate_data_size to estimate live data. A heap that is too small for the tree produces frequent Full GCs; a heap that is too large produces very long Full GCs when they do occur.
Review collector choice. G1GC has been the JVM default since JDK 9 and ZooKeeper’s launcher scripts do not override it. ZGC (production-ready in JDK 15+) dramatically reduces pause times for large heaps.
Enable GC logging if it is not already on. Recommended flag set: -Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m. Without GC logs you are guessing.
Check Transparent Huge Pages. THP enabled can multiply GC pause duration. Verify cat /sys/kernel/mm/transparent_hugepage/enabled and set to never for ZooKeeper hosts.

Throttle and staleness configuration

If root cause is neither disk nor GC and stale drops persist, review whether your throttle settings are too aggressive for the workload.

zookeeper.request_throttle_max_requests defaults to 0 (disabled). If you have set it low, the throttler stalls requests for requestThrottleStallTime (default 100ms) once the limit is hit. Under sustained load this can push request age past the staleness threshold.
requestStaleLatencyCheck defaults to false. If you enabled it, the server marks requests stale based on session-timeout latency in addition to closed connections. That is a stricter policy and will produce stale drops under load that the default config would have absorbed.

Changing these is a tradeoff between absorbing load via queueing versus failing fast. Neither is universally correct; document the choice.

Prevention

Stale drops are a trailing signal. The prevention strategy is to alert on leading indicators so you never see the trailing one move.

Alert on zk_outstanding_requests sustained non-zero before it approaches globalOutstandingLimit.
Alert on any zk_throttled_ops increment rate. Throttling is the gate right before stale drops.
Track zk_fsynctime p99 as a trend, not just a threshold. Any sustained upward drift on dedicated SSD is abnormal.
Track zk_jvm_pause_time_ms p99 against tickTime (2000ms) and syncLimit * tickTime (default 10s). Pauses approaching those values threaten sessions and quorum.
Keep dataLogDir on dedicated storage as a deployment standard. Verify it on every node, every upgrade.
Gate non-critical alerts on zk_uptime > 300s to avoid noise from cold-start recovery, where transient saturation is expected.

How Netdata helps

Netdata collects zk_stale_requests, zk_stale_requests_dropped, zk_outstanding_requests, zk_throttled_ops, zk_fsynctime, and zk_jvm_pause_time_ms at per-second resolution. The cascade from queue growth to stale drops can complete in seconds, so per-second granularity matters.
Correlating stale drops against zk_outstanding_requests and zk_throttled_ops on a single timeline shows whether the stall built up gradually (queue first, throttle next, drops last) or arrived instantly (a GC spike).
Viewing zk_fsynctime alongside zk_jvm_pause_time_ms resolves the disk-versus-GC fork without switching tools.
Per-node dashboards let you compare leader and follower behavior side by side, which is necessary because stale drops on a follower mean something different than stale drops on the leader.
Anomaly advisors on the leading indicators (rising fsync p99, rising outstanding requests) can surface the problem before the stale-drop counter moves, which is where alerting should ideally fire.