ZooKeeper GC pause cascade: how a Stop-the-World freeze expires sessions and re-elects the leader

A ZooKeeper ensemble loses connections in bursts. Sessions expire en masse. The leader changes without a network cause. Request latency spikes on a rhythm that does not match disk I/O. The cause is usually a JVM Stop-the-World pause, and the fix is on the JVM, not the network.

The mechanism: a JVM STW pause freezes the entire ZooKeeper process. No heartbeats go out. No requests are processed. No quorum ACKs flow. The TCP listener still accepts sockets, so the server looks alive from the outside, but the process does nothing with them. Clients miss their heartbeat window and their sessions expire. Followers miss the leader’s heartbeat and, if the pause runs past syncLimit * tickTime (10 seconds with the shipped sample settings tickTime=2000 ms and syncLimit=5; when tickTime is left unset the server default is 3000 ms, and syncLimit must be set explicitly in replicated mode), they declare the leader dead and start a new election. When the JVM resumes, the queued backlog drains as a latency spike and disconnected clients reconnect in a herd.

The metric that proves this pattern is zk_jvm_pause_time_ms, and it is one of the least-collected ZooKeeper metrics in production.

What it means

A GC pause cascade is a self-inflicted availability event caused by JVM behavior. The damage has three layers.

Client sessions expire. During the pause, the server stops responding to client heartbeats. Sessions whose timeout elapses during the pause are marked expired. Their ephemeral nodes vanish and their watches fire. Downstream systems that depend on those ephemeral nodes (Kafka broker registration, HBase region assignment, distributed locks) react to the deletions.

Connections drop in bursts. Expired sessions are closed. zk_connection_drop_count and zk_stale_sessions_expired increment together. zk_num_alive_connections drops sharply, then rebounds as clients reconnect.

The leader may be re-elected. If the paused node is the leader and the pause exceeds syncLimit * tickTime, followers trigger FastLeaderElection. zk_looking_count increments across the ensemble, zk_sum_leader_unavailable_time grows, and writes are impossible for the duration of the election. With the sample settings the threshold is 10 seconds. A Full GC that long is rare on a well-tuned JVM but common on an undersized or wrongly-collected one.

The post-pause drain is its own event. When the JVM resumes, the request queue that built up during the freeze drains as a latency spike. Clients reconnect in a thundering herd, watch notifications fire in bulk, and zk_outstanding_requests may briefly exceed globalOutstandingLimit (default 1000), triggering throttling.

The signature that distinguishes this from a disk stall is rhythm. The spikes repeat at the GC frequency, and zk_jvm_pause_time_ms rises in lockstep with zk_avg_latency. Disk I/O metrics stay flat throughout.

flowchart TD
    A[JVM Full GC pause begins] --> B[Process frozen: no heartbeats, no ACKs, no request processing]
    B --> C{Pause longer than syncLimit x tickTime?}
    B --> D{Pause longer than client session timeout?}
    C -->|Yes| E[Followers trigger FastLeaderElection]
    D -->|Yes| F[Sessions expire, ephemeral nodes vanish]
    B --> G[Outstanding request queue builds]
    A --> H[JVM resumes]
    H --> I[Queue drains as latency spike]
    F --> J[Clients reconnect simultaneously]

Common causes

Quick checks

Safe, read-only.

# JVM pause time (3.6+; populates only when jvm.pause.monitor=true is enabled) - the smoking gun
echo mntr | nc localhost 2181 | grep -E 'zk_.*jvm_pause'

# Outstanding requests: should be 0 in steady state
echo mntr | nc localhost 2181 | grep zk_outstanding_requests

# Recent elections and write unavailability
echo mntr | nc localhost 2181 | grep -E 'zk_(looking_count|.*leader_unavailable_time)'

# Session expiry and connection drop counters
echo mntr | nc localhost 2181 | grep -E 'zk_(stale_sessions_expired|connection_drop_count)'

# Live data tree pressure (usual root cause of growing heap)
echo mntr | nc localhost 2181 | grep -E 'zk_(znode_count|watch_count|approximate_data_size)'

# Real-time GC: FGC count and FGCT cumulative seconds
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 5

# Verify THP is disabled (should say [never] or [madvise])
cat /sys/kernel/mm/transparent_hugepage/enabled

# Verify swappiness is 0 or 1
cat /proc/sys/vm/swappiness

# Confirm the JVM flags actually applied (look for UseG1GC or UseZGC)
ps -o args= -p $(pgrep -f QuorumPeerMain) | tr ' ' '\n' | grep -E 'Use|Xmx|Xms|MaxGCPause'

If four-letter commands are blocked by 4lw.commands.whitelist (default in ZooKeeper 3.5.3+), use the AdminServer instead: curl -s http://localhost:8080/commands/monitor.

How to diagnose it

The GC pause cascade is the only common ZK failure mode where the JVM pause metric rises in lockstep with latency spikes. Use that.

1. Confirm the JVM is pausing. Plot the JVM pause metric against zk_avg_latency. If the spikes align in time, GC is the cause. If zk_avg_latency spikes without a corresponding JVM pause, check zk_max_fsynctime instead. That is a different failure pattern.

2. Confirm the downstream damage. Check whether zk_stale_sessions_expired, zk_connection_drop_count, and zk_looking_count incremented during the same window. Each increment that lines up with a JVM pause is a GC-driven event, not a network blip.

3. Rule out disk. Compare zk_max_fsynctime during the event. If fsync is normal (sub-2ms on dedicated SSD, sub-10ms on cloud storage), the write path is not the cause. If fsync is also elevated, you have a compounding problem and need to address both.

4. Identify the GC algorithm and heap size. From the process command line, confirm whether the JVM is running G1GC, ZGC, Parallel, or CMS. Confirm -Xmx and -Xms. If -Xmx is more than a few GB and the algorithm is Parallel or CMS, the algorithm is the problem. If -Xmx is small relative to the live data tree, heap sizing is the problem.

5. Look at the heap trough, not the peak. Run jstat -gcutil for several minutes during steady state. The number that matters is the Old generation percentage after a Full GC. If the trough stays above 70-75%, the live set is too large for the heap. Grow the heap or shrink the data tree.

6. Check the kernel settings. Verify THP is disabled and swappiness is 0 or 1. Both silently amplify GC pause duration and are easy to miss because they are not ZooKeeper metrics.

7. Find the cause of live-set growth. Pull zk_znode_count, zk_watch_count, and zk_approximate_data_size over a few weeks. Monotonic growth is usually a leak in an application (frameworks creating per-task znodes without cleanup) or in client libraries. The fix is upstream, but the heap is the safety net.

zk_jvm_pause_time_ms is an ADVANCED metrics-framework summary: mntr exposes zk_avg_jvm_pause_time_ms, zk_p50_jvm_pause_time_ms, zk_p95_jvm_pause_time_ms, zk_p99_jvm_pause_time_ms, zk_p999_jvm_pause_time_ms, zk_cnt_jvm_pause_time_ms, and zk_sum_jvm_pause_time_ms directly (no collector-side derivation), whenever the JVM pause monitor is enabled.

Metrics and signals to monitor

Fixes

Heap too small, GC algorithm wrong

The most common combination. Target at least 30% free heap in steady state, with a post-GC trough below 50% of max. Bigger is not better for pauses: a larger heap means longer Full GC pauses when they do occur. The goal is the smallest heap that keeps the live set under 70% Old-gen utilization after collection.

Algorithm choice:

• G1GC has been the JVM default since JDK 9 (JEP 248), and ZooKeeper ships no GC flags, so on JDK 9+ it is ZooKeeper’s default too. Good baseline for most production ensembles. Tune -XX:MaxGCPauseMillis to a value well below the smaller of minSessionTimeout / 3 and syncLimit * tickTime / 3. The flag is a target, not a guarantee, but it biases G1 toward shorter pauses.
• ZGC has been production-ready since JDK 15 and provides sub-millisecond pauses regardless of heap size. Worth the migration if you have ever paged on a ZK leader election caused by GC. It does require more total memory than G1 for the same workload.
• Parallel GC and CMS are the wrong choice for a coordination service. CMS was removed in JDK 14. Parallel GC stop-the-world events scale with heap size.

For a dedicated ZK host, production deployments typically run 1-4 GB heap depending on data tree size, with headroom for the OS and any co-located processes. The data tree, not the heap size, should drive the decision.

A rolling restart is required for any JVM flag change. Expect a brief election during each restart. Suppress election alerts during the maintenance window.

Live set growth (data tree bloat)

If zk_znode_count or zk_watch_count is growing monotonically, growing the heap only delays the problem. Find the subtree that is leaking:

echo dump | nc localhost 2181 | sort | head -50

The fix is usually in the application that is creating znodes without cleanup.

If you must clean up znodes in production, deleting nodes triggers watch events. Batch and rate-limit cleanup to avoid creating your own watch storm.

Transparent Huge Pages enabled

THP defragmentation can extend GC pauses by 2-10x. This is a host-level setting that affects every JVM on the box.

# Verify current state
cat /sys/kernel/mm/transparent_hugepage/enabled

# Disable at runtime (effective immediately, not persistent across reboot)
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Persist via kernel boot parameter
# transparent_hugepage=never

Coordinate with the host owner. On a shared host, other workloads may depend on THP.

Swap activity

If the JVM is swapping, GC pauses become unpredictable because the collector must page in objects to scan them. Do not swap a ZooKeeper JVM.

cat /proc/sys/vm/swappiness   # should be 0 or 1

Set swappiness persistently in /etc/sysctl.d/. Verify the host has enough physical RAM for the JVM heap plus the OS and co-located processes. If a host is oversubscribed, the JVM will swap regardless of swappiness.

Enable GC logging

GC logs are the most important log file in a ZooKeeper deployment. Without them, post-incident root-cause analysis is guesswork.

# Java 9+
-Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m

# Java 8 (legacy)
-verbose:gc -XX:+PrintGCDetails -Xloggc:/var/log/zookeeper/gc.log

Prevention

• Alert on JVM pause time directly. Threshold based on your tickTime and syncLimit. With the sample tickTime=2000 ms, alert when p99 exceeds ~1333ms (a third of minSessionTimeout) and page when it approaches syncLimit * tickTime (10s).
• Track heap trough, not peak. Dashboard post-GC Old-gen utilization and alert when it stays above 75%.
• Track live-set growth. Dashboard zk_znode_count, zk_watch_count, and zk_approximate_data_size as long-window trends. Catch leaks months before they cascade.
• Standardize the JVM flags. G1GC or ZGC, sized heap, GC logging on. Treat the JVM configuration as production infrastructure.
• Enforce host settings. THP disabled and swappiness 0 or 1 on every ZK host, enforced by configuration management.
• Test failover regularly. A controlled leader kill in staging validates that the ensemble re-elects cleanly and that monitoring catches the event. An untested ensemble will surprise you during a real GC cascade.

How Netdata helps

• Per-second JVM pause metrics. The GC cascade signature is rhythmic spikes in zk_jvm_pause_time_ms aligned with latency and connection drops. Per-second resolution makes the rhythm visible. Minute-level averages can hide it entirely.
• Correlated session and election counters. zk_stale_sessions_expired, zk_connection_drop_count, zk_looking_count, and zk_sum_leader_unavailable_time on the same timeline as the JVM pause metric turn a multi-cause investigation into a one-glance confirmation.
• ML anomaly detection on JVM pause and latency series. The first sign of a developing GC problem is often a subtle upward trend in pause time, not a single spike. Anomaly detection surfaces that trend before it crosses a fixed threshold.
• Host-level context alongside ZK metrics. THP state, swappiness, swap usage, and per-process RSS on the same dashboard as zk_jvm_pause_time_ms lets you rule out kernel and host causes without switching tools.
• Composite pattern detection. The GC cascade is a multi-signal pattern (pause, queue buildup, session expiry, possible election). Correlating these signals in one place shortens diagnosis from “what is happening” to “this is the documented GC cascade, here is the runbook.”

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up