$ guides / zookeeper / zookeeper-avg-latency-hiding-write-stalls ▌

Operations Guides

ZooKeeper avg_latency hides write stalls: why the headline number lies

The dashboard says zk_avg_latency is 1.2 ms. Clients are timing out on writes. Both can be true. On a read-heavy ZooKeeper ensemble, the headline latency number can look healthy while the write path is stalled.

Two properties cause this. First, zk_avg_latency, zk_min_latency, and zk_max_latency aggregate reads and writes into one number. Reads are served from local memory and complete in microseconds. Writes require a quorum round-trip plus a transaction log fsync before acknowledgment. When reads dominate the request mix, a severe write stall is diluted by thousands of cheap reads and disappears into the average.

Second, all three metrics are server-cumulative since the last srst reset or process start. They are not sliding windows. A three-hour-old GC pause keeps zk_max_latency pinned at that duration indefinitely, and the average drifts toward the workload average over the entire uptime. Dashboards built on these values are unreliable as recent-behavior signals.

What this means

The aggregation problem is arithmetic. Suppose an ensemble handles 950 reads per interval at 0.5 ms each and 50 writes. If write latency climbs to 500 ms during a disk stall, the weighted average across all requests is (950 * 0.5 + 50 * 500) / 1000, or about 25 ms. A 25 ms average looks like mild congestion. It is not. Every write is taking half a second, and clients with short session timeouts or tight lock-acquire budgets are failing. The same arithmetic that makes the average look benign also breaks threshold alerts: a “page when avg_latency > 100 ms” rule will not fire even when every write is broken.

The cumulative trap compounds this. zk_max_latency only goes up until you issue srst or restart the process. A 15-second GC pause from last Tuesday pins the max at 15,000 ms indefinitely. Teams learn to ignore it, and when a real stall arrives the alert is already suppressed as noise.

Together these produce a classic failure pattern: a write-heavy downstream system (Kafka controller election, HBase region assignment, a distributed lock service) starts failing, the on-call engineer checks the ZooKeeper dashboard, sees a flat average and a max that has “always been high,” and concludes ZooKeeper is fine. It is not.

Common causes

Cause	What it looks like	First thing to check
Transaction log disk stall	Fsync p99 elevated, write latency p99 tracking it, `zk_outstanding_requests` climbing on the leader	Fsync percentiles via the metrics endpoint, plus `iostat -x` on the txnlog device
Shared or co-located storage	Periodic fsync spikes, often aligned with snapshot creation when `dataLogDir` equals `dataDir`	Whether `dataLogDir` is set in `zoo.cfg` and points to a dedicated device
Cloud storage throttling	Sudden fsync cliff after a period of normal latency, correlates with burst credit exhaustion	Cloud provider IOPS and burst metrics for the txnlog volume
JVM GC pauses	JVM pause p99 elevated, both read and write latency spike together, `zk_outstanding_requests` builds then drains	JVM pause percentiles via the metrics endpoint, plus GC logs
Quorum ACK delay	Quorum ACK p99 elevated on the leader, writes slow even when leader disk is fine	Quorum ACK latency on the leader, follower fsync times
Cumulative-metric dashboard illusion	`zk_avg_latency` looks flat and low, `zk_max_latency` pinned high for days, clients report write timeouts	Whether your monitoring ever issues `srst`, and whether you collect per-type write latency at all

Quick checks

The mntr four-letter-word command exposes the aggregated latency metrics (zk_avg_latency, zk_min_latency, zk_max_latency), basic pipeline counters (zk_outstanding_requests, zk_server_state), and, since 3.6, per-type latency percentiles, fsync times, quorum ACK latency, JVM pause metrics, and throttled operations. These 3.6+ metrics are also available in Prometheus format from the Prometheus metrics provider’s HTTP endpoint (default port 7000).

# Confirm the version - per-type latency metrics require 3.6+
echo srvr | nc localhost 2181 | grep -E "Zookeeper version|Mode"

# Aggregated latency (cumulative since last srst) - from mntr
echo mntr | nc localhost 2181 | grep -E "zk_(avg|min|max)_latency"

# Per-type read/write latencies (3.6+) - from mntr
echo mntr | nc localhost 2181 | grep -E "zk_.*(update|read)latency"

# Fsync percentiles (3.6+) - the usual root cause of write stalls
echo mntr | nc localhost 2181 | grep -E "zk_.*fsynctime"

# Request pipeline backlog - from mntr
echo mntr | nc localhost 2181 | grep "zk_outstanding_requests"

# Quorum ACK latency (leader) and JVM pause time (3.6+) - from mntr
echo mntr | nc localhost 2181 | grep -E "zk_.*(quorum_ack|jvm_pause)"

# Throttled operations (3.6+) - means clients are being dropped at the global limit
echo mntr | nc localhost 2181 | grep "zk_throttled_ops"

# Verify the four-letter-word whitelist includes mntr (3.5.3+)
echo mntr | nc localhost 2181 | head -1
# An empty response means mntr is not whitelisted and monitoring is silently broken

How to diagnose it

Confirm the ZooKeeper version exposes per-type latency metrics. Anything before 3.6.0 only provides the aggregated zk_avg/min/max_latency, so you cannot separate reads from writes at all. If you are on an older version, the only options are computing deltas between scrapes and periodically issuing srst.
Pull per-type write and read latency percentiles from mntr. If write latency is several orders of magnitude above read latency, the write path is the problem and the aggregated average is hiding it.
Pull fsync percentiles. On a healthy dedicated SSD, p99 fsync should be under 2 ms. If it is in the tens or hundreds of milliseconds, disk I/O is the root cause.
Check zk_outstanding_requests on the leader via mntr. It should be zero in steady state. A sustained non-zero value with active traffic means the pipeline cannot keep up. If it is approaching globalOutstandingLimit (default 1000), throttling will kick in and clients will see timeouts. The zk_throttled_ops counter tracks operations already dropped at this limit.
Check quorum ACK latency percentiles on the leader. If this is elevated while leader fsync is fine, the bottleneck is follower-side: follower GC, follower disk, or network between leader and followers.
Check JVM pause time percentiles. If GC pauses are the cause, both read and write latency will spike together rather than writes alone.
If you are stuck on pre-3.6 metrics and cannot upgrade, you can issue srst at a fixed interval and compute deltas. Warning: srst resets all server statistics simultaneously. Every monitoring system scraping that node will see a counter reset at the same moment. If multiple tools depend on the cumulative counters, coordinate the reset cadence or only one of them will see meaningful deltas.

flowchart TD
    A["zk_avg_latency looks fine"] --> B{"Collecting per-type write latency?"}
    B -- "No (pre-3.6 or not scraping)" --> C["Hidden: cannot tell reads from writes"]
    B -- "Yes" --> D{"Write latency p99 high?"}
    D -- "No" --> E["Read path or client-side issue"]
    D -- "Yes" --> F{"Fsync p99 high?"}
    F -- "Yes" --> G["Disk stall: shared storage, throttling, or hardware"]
    F -- "No" --> H{"Quorum ACK p99 high?"}
    H -- "Yes" --> I["Follower GC, follower disk, or network"]
    H -- "No" --> J["Check JVM pauses and pipeline queue"]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Write latency percentiles	Write-path latency separated from reads. The metric that actually reflects write stalls.	p99 climbing while read latency stays flat
Read latency percentiles	Read-path latency. Reads are memory lookups, so elevation usually means GC or deep hierarchies.	p99 above single-digit milliseconds
Fsync percentiles	Time to fsync the transaction log. The single most common write-stall root cause.	p99 above 10 ms on SSD, or any sustained upward trend
Quorum ACK latency percentiles	Time from PROPOSE to quorum ACK on the leader. Captures follower and network delays.	p99 above 50 ms, or approaching `syncLimit * tickTime`
`zk_outstanding_requests`	Request pipeline backlog. Leading indicator: it fills before latency spikes.	Sustained non-zero, or approaching `globalOutstandingLimit`
`zk_throttled_ops`	Counter of operations throttled at the global limit. Means clients are already being dropped.	Any non-zero rate
JVM pause time percentiles	GC pause impact. When elevated, both reads and writes spike together.	p99 approaching a meaningful fraction of `minSessionTimeout`
`zk_max_latency` (handled with care)	Cumulative extreme since last `srst`. Useful only as a delta.	Pinned at a high value for days means nobody is resetting or computing deltas

Fixes

Separate reads from writes in monitoring

The first fix is instrumentation, not infrastructure. If you are on 3.6 or later, collect write latency and read latency percentiles separately via mntr and alert on them independently. Alert on write latency p99 crossing a baseline-relative threshold, not on zk_avg_latency.

If you are on a version before 3.6, you have two options. The first is to issue srst at the end of each scrape and compute deltas, which gives you per-interval min, avg, and max. The second is to upgrade. Given that 3.6 has been out for years and per-type percentile metrics are the only reliable way to see write stalls, upgrading is usually the right call.

Tradeoff: srst resets the cumulative counters that other tools may depend on. If you have multiple monitoring systems scraping the same node, coordinate the reset cadence or only one of them will see meaningful deltas.

Put the transaction log on dedicated storage

If fsync p99 is the smoking gun, the most impactful single change is setting dataLogDir to a dedicated low-latency device that holds nothing else. The default is for the transaction log and snapshots to share dataDir, which means snapshot writes (large, bulk, sequential) compete with fsync (small, latency-critical). This shows up as periodic fsync spikes aligned with snapshot creation.

On cloud instances, also check whether the txnlog volume is a burstable type. Burst credit exhaustion produces a sudden fsync cliff that looks like hardware failure but is really throttling. Move to provisioned IOPS or a higher tier.

Tradeoff: dedicated storage costs money and adds an operational variable. It is almost always worth it for ZooKeeper, because fsync latency dominates write latency and ensemble stability.

Investigate quorum ACK latency

If leader fsync is fine but quorum ACK p99 is elevated, the bottleneck is on the followers or the network. Pull fsync percentiles from each follower. A single slow follower (failing disk, long GC) can inflate quorum ACK latency because the leader must wait for a majority.

Check whether zk_synced_followers on the leader is below ensemble_size - 1. A follower that is connected but lagging will drag quorum ACK time without failing health checks.

Tradeoff: removing a slow follower from the ensemble to protect quorum ACK latency reduces fault tolerance. In a three-node ensemble, removing one follower leaves no redundancy.

Address JVM GC pauses

If JVM pause p99 is elevated and both read and write latency spike together, GC is the cause. Common fixes: size the heap to the data tree (monitor zk_znode_count and zk_approximate_data_size), switch to G1GC if you are still on CMS or Parallel, and consider ZGC on JDK 15+ for low-pause collection.

Tradeoff: a larger heap means longer full GC pauses when they do occur. The goal is enough headroom that full GC is rare, not so large that a full GC is catastrophic.

Stop trusting `zk_max_latency` as a live signal

If your dashboards show zk_max_latency pinned at a high value for days, either reset it periodically with srst or compute deltas in your monitoring system. Treat the raw cumulative value as a “worst ever since start” marker, not a current-condition signal.

Prevention

Collect per-type latency from day one. On 3.6+, scrape write latency and read latency with percentiles. The aggregated average is structurally unable to surface write stalls on read-heavy clusters.
Alert on write latency p99, not zk_avg_latency. Use a baseline-relative threshold (for example, 3x the rolling p99) rather than an absolute number, since write latency baselines vary by workload.
Track fsync p99 as a first-class metric. It is the leading indicator for the most common write-stall root cause and is almost never collected by default.
Set dataLogDir to a dedicated device. Every production ensemble should isolate the transaction log from snapshot I/O to avoid periodic fsync spikes during snapshot creation.
Reset or delta zk_max_latency. Decide on a reset cadence (per scrape, per minute, per hour) and stick to it, or have your monitoring system compute deltas. A pinned zk_max_latency is a monitoring smell.
Gate cold starts. Suppress non-critical latency alerts when zk_uptime is below 300 seconds, because latency metrics are noisy immediately after restart.

How Netdata helps

Netdata collects the full mntr output per second, including zk_avg_latency, zk_min_latency, zk_max_latency, zk_outstanding_requests, and leader-only metrics like zk_synced_followers and zk_pending_syncs.
Per-second collection means a write stall that lasts only a few seconds still shows up as a distinct spike, rather than being averaged away in a longer scrape window.
The anomaly advisor correlates write latency with fsync time, outstanding requests, quorum ACK latency, and JVM pause time, so when latency deviates from baseline the related root-cause metrics are highlighted in the same view.
Because Netdata derives per-interval rates from cumulative counters, the zk_max_latency cumulative trap is handled by computing deltas rather than charting the raw monotonically-increasing value.
Leader-only metrics are collected from every node and filtered by reported zk_server_state, so replication health is visible without manually identifying the leader first.