ZooKeeper "Detected pause in JVM or host machine (eg GC)": the pause-monitor warning

The log line looks like this:

Detected pause in JVM or host machine (eg GC): pause of approximately 5234 ms, total pause: info level: 0, warn level: 0

ZooKeeper’s JvmPauseMonitor emits that line when the process froze longer than its configured threshold. The monitor thread sleeps for a fixed interval, wakes, and measures how long the sleep actually took. Anything beyond the expected sleep plus the warn threshold gets logged. If the JVM was not in a visible GC at that moment, the line ends with “No GCs detected”, which is the operator’s cue that something else on the host stole CPU.

This warning is the direct fingerprint of the pause cascade ZooKeeper operators fear most. The process freezes, the leader cannot send or receive ZAB heartbeats, request processing stops, and when the JVM resumes the outstanding queue drains in a burst. A pause above tickTime (default 2000ms) risks an unnecessary leader election. A pause approaching the negotiated session timeout starts expiring client sessions, which cascades to ephemeral node deletion, watch firing, and reconnection storms in Kafka, HBase, Solr, and anything else using those sessions for coordination.

The warning is rarely about Java alone. Stop-the-World GC is the common cause, but the same line fires for host swap, Transparent Huge Page (THP) compaction, CPU steal from noisy neighbors, and kernel livelock. The first decision is “GC or host?”, because the fix is completely different.

What this means

The JvmPauseMonitor runs a watcher thread that loops:

1. Sleep for jvm.pause.sleep.time.ms (default 500ms).
2. Measure how long the sleep actually took.
3. If the extra time exceeds the warning threshold (jvm.pause.warn-threshold.ms, default 10000ms), log a WARN entry with the approximate pause duration and GC pool statistics.
4. If it exceeds the info threshold (jvm.pause.info-threshold.ms, default 1000ms), log at INFO.

The threshold-based logic means the warning fires only when the JVM, the OS scheduler, or both failed to give the monitor thread its expected CPU slot. The monitor cannot distinguish causes itself. It can only report that the wall clock advanced much further than its sleep requested. The GC pool data it appends is what lets you start the root-cause decision.

Three things about this warning catch operators off guard.

It is disabled by default. The flag jvm.pause.monitor=false ships as the default in QuorumPeerConfig. If you have never set jvm.pause.monitor=true in zoo.cfg, the warning never appears and you are relying on GC logs alone. The feature was introduced in ZooKeeper 3.6.0 via ZOOKEEPER-3037 and backported to 3.5.10 via ZOOKEEPER-4202, so older deployments do not have it.

“No GCs detected” is not a clean bill of health. It is the opposite. When the line reads “No GCs detected”, the JVM was not stopped for GC. That means the freeze came from outside the JVM: host swap, THP compaction, CPU starvation, NUMA remote-page faults, or kernel work. This branch is harder to debug because there is no GC log entry to match against.

The pause duration is approximate. Reports of 12 seconds, 31 seconds, and even 129 seconds exist in the wild, often with “No GCs detected”. Treat any pause approaching tickTime (2000ms) as a near-miss for a leader election. Treat anything approaching half the session timeout as a near-miss for a session-expiry cascade.

flowchart TD
    A[PauseMonitor sleeps 500ms] --> B{Sleep ran long?}
    B -- no --> A
    B -- "yes, > 10000ms extra" --> D[WARN log line]
    B -- "yes, 1000-10000ms extra" --> C[INFO log line]
    C --> E{GC pools advanced?}
    D --> E
    E -- "yes, GC visible" --> F[Stop-the-World GC cause]
    E -- "No GCs detected" --> G[Host-level cause]
    G --> H[Swap, THP, CPU steal, NUMA]
    F --> I[Heap pressure, CMS, humongous alloc]

Common causes

Quick checks

These commands are all read-only. Run them on the ZooKeeper host where the warning fired. Paths vary by distribution; the locations shown are the common ones.

# Confirm the JvmPauseMonitor is actually enabled (otherwise the warning could not have fired)
grep -E 'jvm\.pause\.monitor|jvm\.pause\.' /etc/zookeeper/conf/zoo.cfg 2>/dev/null \
  || grep -E 'jvm\.pause\.monitor|jvm\.pause\.' "$ZK_HOME/conf/zoo.cfg" 2>/dev/null

# Get the JVM PID for the ZK process
ZK_PID=$(pgrep -f QuorumPeerMain)

# Heap snapshot - one line per second for 5 samples
# Columns of interest: O (old gen), FGC (full GC count), FGCT (full GC time, cumulative seconds)
jstat -gcutil "$ZK_PID" 1000 5

# Find the most recent pause warnings and whether GC was blamed
grep "Detected pause in JVM" /var/log/zookeeper/zookeeper.log | tail -20

# Check the GC log for Stop-the-World events near the warning timestamp
# (path depends on your JVM flags; common locations shown)
ls -la /var/log/zookeeper/gc*.log 2>/dev/null
grep -E "Pause (Full|Young|Old)" /var/log/zookeeper/gc.log.0.current 2>/dev/null | tail -20

# THP setting - "[never]" is the safe value for ZooKeeper hosts
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

# Swap configuration and current swap usage of the ZK process
sysctl vm.swappiness
grep VmSwap /proc/"$ZK_PID"/status
free -m

# CPU steal time on the host (look at the 'st' column over 5 samples)
vmstat 1 5

# mntr signals that move during a pause (collect twice to compute deltas)
echo mntr | nc localhost 2181 | grep -E 'zk_(avg|p99)_jvm_pause_time_ms|zk_outstanding_requests|zk_looking_count|zk_stale_sessions_expired|zk_connection_drop_count|zk_uptime'

How to diagnose it

1. Confirm the warning is real and timestamped. Pull the log line, capture the exact timestamp, and note the approximate pause duration. Everything that follows keys off that timestamp.
2. Look at the GC log first. If GC logging is enabled (-Xlog:gc*:file=... on JDK 9+, or -XX:+PrintGCDetails on JDK 8), find the GC entries in the same 10-second window. A matching Full GC, G1 mixed pause, or humongous allocation pause confirms the JVM-side cause.
3. Match the JVM pause metric. zk_avg_jvm_pause_time_ms and zk_p99_jvm_pause_time_ms from mntr (3.7+, requires jvm.pause.monitor=true) show the same event in metric form. If zk_p99_jvm_pause_time_ms was well below the warning threshold in steady state and then spikes to the warned value, the correlation is tight.
4. Decide: GC or host. If the GC log shows no corresponding Stop-the-World event, treat the warning as a host-level freeze and pivot to THP, swap, CPU steal, and NUMA. The “No GCs detected” tail of the log line is the hint.
5. Correlate downstream damage. Check zk_looking_count for a leader election around the same time. Check zk_stale_sessions_expired and zk_connection_drop_count for the session-expiry cascade. Check zk_outstanding_requests for the post-pause queue burst. Each of these tells you how bad the blast radius was.
6. Check data tree pressure. If GC is the cause, the next question is why. zk_znode_count, zk_watch_count, and zk_approximate_data_size give you the long-running pressure trend. A rising heap trough means the live set is growing.
7. Verify the host is not the actual culprit. Even if GC shows up in the log, THP compaction and swap can multiply GC pause duration by 2x to 10x. Check both before concluding “Java tuning only.”

Metrics and signals to monitor

Fixes

If it is Stop-the-World GC

The fix depends on which collector is running and why it is pausing.

Heap too small for the live set. This is the most common cause. The data tree grows until old generation fills, Full GC fires, and pause duration grows with heap size. Identify the growth with zk_znode_count and zk_approximate_data_size, then either clean up the leaking subtree (often a framework leaving per-task nodes behind) or grow the heap. Larger heaps reduce frequency but increase pause duration per Full GC, so this is a stopgap, not a permanent fix.

Collector is wrong. CMS is deprecated since JDK 9 and removed in JDK 14. If you are still on CMS, the long-term fix is G1GC or ZGC. G1GC is the default from JDK 9 onward and handles multi-GB heaps more gracefully than ParallelGC. ZGC, production-ready from JDK 15+ (experimental in JDK 11-14), drops pause times to sub-millisecond for most workloads and is the recommended choice for new deployments on supported JDK versions.

Humongous allocation in G1. A single object larger than half the G1 region size triggers humongous allocation handling, which can stall the JVM. Check the GC log for “Humongous” entries. The fix is usually application-side: large znode payloads (an anti-pattern) or oversized serialization buffers.

If it is THP compaction

Transparent Huge Pages are a known cause of multi-second pauses with no GC activity. The kernel compaction that backs THP can stall the JVM for seconds at a time, and the JvmPauseMonitor reports these with “No GCs detected”. The standard recommendation for ZooKeeper hosts is to disable THP entirely.

# Read current state (look for the bracketed value)
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

The following commands modify the running kernel configuration. Apply them during a maintenance window or after confirming no other workload on the host depends on THP.

# WARNING: changes kernel behaviour for all processes on this host
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Persist this across reboots with a systemd unit, a tuned profile, or your config management of choice. This change does not require restarting the ZooKeeper JVM.

If it is host swap

When the JVM heap is paged out, every GC scan becomes a page-fault storm. The result is pauses that look like GC in the log but are 10x to 100x longer than they should be. Two fixes apply.

# WARNING: changes live kernel parameter
sysctl -w vm.swappiness=1
# Persist in /etc/sysctl.d/

The other half is memory headroom. Ensure the JVM heap plus JVM overhead plus the OS page cache fit in resident memory. A cgroup memory limit lower than RSS plus working set will push the JVM into swap regardless of vm.swappiness.

If it is CPU steal or noisy neighbor

Cloud instances with noisy neighbors report this as steal time in top, vmstat, or mpstat. The JvmPauseMonitor sees the freeze as “No GCs detected” with no THP and no swap. The fix is at the cloud layer: migrate the instance to a dedicated host or a less-contended physical machine, or move ZooKeeper onto bare metal where CPU isolation is enforceable. ZooKeeper’s write path goes through a single request processor thread, so it is particularly sensitive to CPU scheduling jitter.

Prevention

• Enable the JvmPauseMonitor on every production ZK host. Set jvm.pause.monitor=true in zoo.cfg. It is opt-in by default, so without this line the warning never fires and you lose the first signal of a freeze.
• Keep GC logging enabled. Without -Xlog:gc*:file=... (JDK 9+) or -XX:+PrintGCDetails (JDK 8), every pause warning becomes a “No GCs detected” mystery. The GC log is the only way to confirm GC as the cause.
• Disable THP on ZK hosts. Treat this as a baseline host configuration, not an incident response.
• Set vm.swappiness to 0 or 1. ZooKeeper should never be swapped. Size host memory accordingly.
• Watch the heap trough, not the peak. A rising post-GC minimum means the live set is growing. Catch this months before it becomes an OOM.
• Standardize on G1GC or ZGC. Remove CMS from any ZK deployment still running it.
• Monitor zk_p99_jvm_pause_time_ms (3.7+). This is the metric equivalent of the log warning and lets you alert on pause trends before they cross the WARN threshold.

How Netdata helps

• Per-second JVM pause metrics from the ZooKeeper collector let you see the exact second a pause started and how long it lasted, rather than relying on the periodic mntr scrape interval.
• Correlate zk_jvm_pause_time_ms with zk_looking_count, zk_stale_sessions_expired, and zk_connection_drop_count on a single timeline. The pause, the leader election, and the session cascade line up, so you can read the blast radius at a glance.
• Distinguish GC cause from disk cause. Overlaying zk_p99_jvm_pause_time_ms against zk_p99_fsynctime tells you whether the freeze was the JVM or the disk. They have different fixes.
• Host-level signals alongside the JVM. CPU steal, swap usage, and memory pressure metrics sit in the same dashboard as ZooKeeper, so “No GCs detected” pauses become diagnosable without SSH’ing to the box.
• Anomaly detection on zk_outstanding_requests and zk_avg_latency catches the post-pause queue drain and the latency burst even when no one is watching the pause metric directly.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper pending syncs growing: followers can’t keep up with the write rate