ZooKeeper heap usage climbing: catching the GC death spiral before it starts

ZooKeeper keeps its entire data tree on the JVM heap: every znode, its data bytes, ACLs, children lists, watch registrations, and session state. When heap climbs, it is the leading indicator for the OOM that eventually kills the ensemble. The signal that matters is not the sawtooth peaks from young-generation GC, but the rising post-GC trough that means the live set itself is growing.

Sustained heap above roughly 75% of max drives GC pressure non-linearly. Above 90%, Full GC pauses stretch long enough to drop client sessions and trigger leader elections. The death spiral proper starts when those pauses themselves prevent requests from completing, which prevents objects from being freed, which forces even more GC. By the time you see the OOMKill, you are already inside the cascade.

The goal here is narrow: catch heap pressure while it is still a capacity trend, before it becomes a pause cascade that takes down sessions and quorum.

What this means

Read heap pressure through the post-GC trough, not the instantaneous usage number. A sawtooth that ramps to 70%, drops to 30%, and ramps again is healthy: young-generation GC collecting short-lived request objects. A sawtooth where the bottom keeps rising (30%, then 40%, then 50%) is the live set growing. That is the trend that ends in OOM.

Working thresholds:

• Heap sustained above 75% of max: elevated GC pressure.
• Heap sustained above 90% of max: imminent Full GC and OOM risk.
• Post-GC trough above 50% of max: the live set is too large for the configured heap and capacity work is overdue.
• Maintain at least 30% free heap as standing headroom.

A single Full GC pause above roughly 1 second is the leading indicator for both client session expiry and leader election. The danger windows are minSessionTimeout (default 2 x tickTime = 4000ms) for client sessions, and syncLimit x tickTime (default 5 x 2000ms = 10 seconds) for quorum stability. Once pause times start approaching those windows, every GC event is a near-miss.

One trap to name up front: heap pressure does not show up cleanly in mntr. The four-letter-word output does not expose JVM heap metrics in any current ZooKeeper release (still an open request, ZOOKEEPER-4818). You need JMX, jstat or jcmd, the AdminServer /commands/environment endpoint (ZooKeeper 3.6+), or an external Prometheus JVM exporter. Treat mntr as the source for tree and pipeline metrics, not for heap.

flowchart TD
  A[Live set grows: znodes watches sessions] --> B[Old generation fills]
  B --> C[GC frequency rises, Full GC pauses stretch]
  C --> D[Heartbeats and quorum ACKs missed]
  D --> E[Sessions expire, leader election]
  E --> F[Reconnect storm and election allocations]
  F --> B

Common causes

Quick checks

Read-only and safe on a production server.

# Current heap from JMX via jcmd (needs JDK tools and the ZK PID)
jcmd $(pgrep -f QuorumPeerMain) GC.heap_info

# Live GC summary, 5 samples 1 second apart
# Columns include O% (old gen used), FGC (full GC count), FGCT (full GC time)
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 5

# Data tree composition from mntr (does NOT include heap)
echo mntr | nc localhost 2181 | grep -E 'zk_(znode_count|approximate_data_size|watch_count|ephemerals_count|num_alive_connections)'

# JVM pause time percentiles (ZK 3.6+)
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# AdminServer environment (3.6+) shows configured max heap
curl -s http://localhost:8080/commands/environment

# Process RSS vs configured heap (swap detection)
ps -o pid,rss,vsz,cmd -p $(pgrep -f QuorumPeerMain)

# Transparent Huge Pages status (THP multiplies GC pause times)
cat /sys/kernel/mm/transparent_hugepage/enabled

If mntr returns nothing, confirm 4lw.commands.whitelist includes mntr (required since ZooKeeper 3.5.3). A misconfigured whitelist silently returns empty output, which monitoring systems often interpret as zeros.

How to diagnose it

1. Confirm the trend is live-set growth, not allocation spikes. Plot the post-GC trough over hours or days, not the instantaneous usage. If the trough is flat but peaks are spiking, the problem is allocation rate (request load, per-request allocation, or watch notification fan-out), not tree size.
2. Pull the tree composition. zk_znode_count and zk_approximate_data_size should move together if nodes are accumulating. If data size grows faster than node count, individual znodes are getting larger, which usually means an application is using ZooKeeper as a data store against the jute.maxbuffer 1MB default.
3. Pull watch and ephemeral counts. Watch registrations are tracked on heap. A watch leak shows up as rising zk_watch_count without proportional growth in zk_num_alive_connections. The ratio of watches per connection should be stable.
4. Confirm GC is actually driving latency symptoms. Cross-reference zk_jvm_pause_time_ms p99 with zk_avg_latency and zk_outstanding_requests. If pause spikes and queue depth spikes coincide, GC is the cause. If only write latency moves and read latency stays flat, the disk is more likely the cause.
5. Estimate runway. Track zk_znode_count growth rate, then estimate bytes per znode as approximate_data_size / znode_count. Rule of thumb: roughly 300 bytes of per-znode overhead, with JVM object overhead typically 2-3x the raw data size once headers, references, and HashMap internals are included.
6. Check the GC algorithm and Java version. CMS is deprecated since JDK 9 and removed in JDK 14. G1GC is the JVM default since JDK 9. ZGC is production-ready from JDK 15 (experimental since JDK 11). The -XX:+UseConcMarkSweepGC flag on JDK 14+ is silently ignored or errors at startup.
7. Confirm the JVM is not swapping. Swelling RSS past the configured heap means the JVM is touching native or off-heap memory, and any paging makes GC pauses 10-100x worse because GC must page objects in to scan them. Confirm vm.swappiness is 0 or 1 on ZooKeeper hosts.

Metrics and signals to monitor

Fixes

Data tree bloat

The most common root cause. Persistent znodes accumulate forever; only ephemeral nodes die with their sessions. Identify which subtree is leaking.

# Top ephemeral holders by session (expensive on large trees; use sparingly)
echo dump | nc localhost 2181 | sort | head -50

Once the leaking application is identified, delete the unnecessary znodes. Warning: deleting heavily-watched znodes in production triggers watch-notification storms, which can themselves spike heap and outstanding requests. Coordinate with the owning application, do cleanup in batches, and prefer off-peak windows. For new subtrees that should auto-clean, use container nodes (ZooKeeper 3.5+) or TTL nodes (3.5+) so the leak does not recur.

Heap undersized

If the tree is appropriately sized but heap is still climbing, the heap is too small. Update ZOO_HEAP_SIZE or JVMFLAGS in zkEnv.sh (or the equivalent in your distribution’s service file). A rolling restart is required.

Tradeoff: bigger heaps mean longer Full GC pauses with the wrong collector. With G1GC or ZGC, larger heaps are safe and pauses stay bounded. With Parallel GC or CMS, larger heaps make things worse. The Apache admin guide recommends sizing conservatively relative to physical memory: “if you have 4G of RAM, do not set the Java max heap size to 6G or even 4G. For example, it is more likely you would use a 3G heap for a 4G machine, as the operating system and the cache also need memory.” A common operator pattern is to start around a few GB and grow as the tree demands.

Set Xms = Xmx to avoid heap resizing overhead during operation, and consider -XX:+AlwaysPreTouch so memory pages are faulted in at startup rather than during request processing.

Wrong GC algorithm

• G1GC is the safe default. The default MaxGCPauseMillis is 200ms.
• ZGC (JDK 15+) gives sub-millisecond pauses at the cost of more total memory.
• CMS is deprecated since JDK 9 and removed in JDK 14. Any deployment still on CMS should treat migrating off it as capacity work, not tuning.

If GC logging is not enabled, turn it on now. Operating a ZooKeeper server without GC logs means operating blind on the single metric most likely to take the ensemble down.

# Java 9+ GC logging flag
-Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m

# Java 8 equivalent
-verbose:gc -XX:+PrintGCDetails -Xloggc:/var/log/zookeeper/gc.log

Watch leak

If zk_watch_count is growing out of proportion to connections, the fix is on the client side, not the server. ZooKeeper 3.6+ added persistent and recursive watches that do not need re-registration, which can drive sustained higher watch counts than one-shot watches. Identify the clients registering watches without consuming them.

Per-request allocation pressure

A known serialization path allocated unnecessary heap per request before being fixed in 3.9.0 (ZOOKEEPER-4718). If you are on 3.8.x and seeing GC pressure under high request rate, upgrade to 3.9.x as part of the fix.

JVM swap

If RSS is well above configured heap, investigate off-heap and native memory (NIO buffers, Netty buffers, JVM internals). Set vm.swappiness=0 or 1 and ensure the host has enough physical memory for heap plus off-heap overhead plus the rest of the system.

Prevention

• Track the post-GC trough as a capacity metric. Instantaneous usage hides the live-set trend that predicts OOM. Keep the trough below 50% of max heap.
• Alert on tree growth trends, not just absolutes. Page on zk_znode_count and zk_approximate_data_size growth rate, not only on absolute thresholds. Re-baseline after every major application onboarding.
• Size heap for tree growth 6-12 months out. Re-baseline after every major application onboarding.
• Prefer G1GC or ZGC. Do not deploy new ensembles on CMS, and migrate existing ones off it.
• Set Xms = Xmx and consider -XX:+AlwaysPreTouch. Avoid runtime heap resize and page-fault stalls.
• Disable Transparent Huge Pages on ZooKeeper hosts. THP can multiply GC pause times by 2-10x.
• Pin vm.swappiness to 0 or 1. Swapping makes GC pauses 10-100x worse.
• Run ZooKeeper 3.9.x or later when feasible. Newer releases reduce per-request heap allocation pressure.
• Use container nodes or TTL nodes for subtrees that should auto-clean. Prevents the slow persistent-znode accumulation that causes most long-running heap incidents.

How Netdata helps

• Per-second JVM heap metrics (used, committed, max) surface both the sawtooth and the rising post-GC trough that slower scrapes blur into a flat line.
• ML anomaly detection flags the non-linear climb from roughly 75% to 90% as a single behavioral change, instead of waiting on a static threshold that fires after the spiral has started.
• zk_znode_count, zk_approximate_data_size, and zk_watch_count sit alongside JVM heap in one view, so the driver of live-set growth is visible without manual cross-referencing.
• zk_jvm_pause_time_ms p99 paired with zk_outstanding_requests, zk_stale_sessions_expired, and zk_looking_count tells you immediately whether a heap trend has started cascading into sessions and elections.
• JVM collector and pause-time percentiles tracked against the heap baseline make a GC algorithm regression obvious against historical behavior.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius
• ZooKeeper follower sync time climbing: a follower approaching ejection
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert