ZooKeeper connection drops spiking: sessions dying in bursts

A burst in zk_connection_drop_count means connections to a ZooKeeper server are closing in a tight window, not one at a time. When the burst pushes zk_stale_sessions_expired up simultaneously, you are looking at a session expiration storm in progress or one about to land on dependent services.

Occasional single drops across a large fleet are background noise. A sustained drop rate above roughly 0.1% of total connections per minute is where the signal stops being normal churn. Bursts that fire on a rhythm (every few minutes, hourly, at the same minute past the hour) almost always point to a JVM garbage collection cycle or a scheduled job that briefly saturates the leader.

The worst version cascades. Sessions expire, ephemeral nodes vanish, watches fire, clients reconnect simultaneously. That reconnection wave can push zk_outstanding_requests toward globalOutstandingLimit, which throttles new reads from client sockets and produces another round of timeouts. This article covers how to read the burst, find the trigger, and stop the cascade.

What this means

zk_connection_drop_count is a server-side counter of connections that closed. A connection can close for four reasons: the client session expired and the server dropped it, the client process crashed or cleanly closed its handle, the network dropped the TCP session, or the server closed it (throttle rejection, large request rejection, auth failure). The counter alone does not tell you which.

Correlate the burst with adjacent signals:

• If zk_stale_sessions_expired jumps with the same shape, sessions are timing out. The drop is the consequence; the expiry is the cause.
• If zk_jvm_pause_time_ms p99 spikes immediately before the burst, the trigger is GC. The pause stopped the heartbeat thread long enough to miss a renewal window.
• If zk_connection_rejected spikes instead, the server is refusing new connections at maxClientCnxns per source IP, not killing existing sessions.
• If zk_num_alive_connections drops sharply but zk_stale_sessions_expired is flat, clients are disconnecting on their own (client-side GC, load balancer change, fleet restart, DNS issue).

flowchart TD
  A[GC pause or network event] --> B[Heartbeats missed]
  B --> C{Pause exceeds session timeout?}
  C -- yes --> D[Sessions expire en masse]
  C -- no --> H[Connections drop, then recover]
  D --> E[Ephemeral nodes vanish]
  E --> F[Watch notifications fire]
  F --> G[Reconnection thundering herd]
  G --> I[Outstanding requests spike]
  I --> J[Latency spike, more drops]

Common causes

Quick checks

All read-only. Run on the node that owns the burst.

# Confirm the drop and adjacent signals in one pull
echo mntr | nc localhost 2181 | grep -E 'zk_(connection_drop_count|connection_rejected|stale_sessions_expired|num_alive_connections)'

# Confirm node is functional, not read-only (expect "rw")
echo isro | nc localhost 2181

# JVM pause signature on the same node
echo mntr | nc localhost 2181 | grep -E 'zk_.*jvm_pause'

# Leader and election history
echo mntr | nc localhost 2181 | grep -E 'zk_(server_state|looking_count|uptime|sum_leader_unavailable_time)'

# Rule out disk-induced election (fsync warning pattern in ZooKeeper logs)
grep "fsync-ing the write ahead log" /var/log/zookeeper/zookeeper.log | tail -20

# Server logs: "Expiring session 0x..., timeout of Nms exceeded"; client logs:
# "Client session timed out, have not heard from server in Nms for session id 0x..."
# Recent session expiry events
grep -iE "expir" /var/log/zookeeper/zookeeper.log | tail -50

# Connection distribution by source IP (expensive on large ensembles, use sparingly)
echo cons | nc localhost 2181 | grep -oE '^/[0-9.]+' | sort | uniq -c | sort -rn | head

# OS listen backlog and SYN queue drops
ss -ltn | grep 2181
nstat -az TcpExtListenOverflows TcpExtListenDrops | tail -5

How to diagnose it

1. Confirm the burst is on the metric you think. Pull zk_connection_drop_count over the burst window. Compare against zk_stale_sessions_expired and zk_connection_rejected. If only zk_connection_rejected is moving, this is not your incident. Skip to the maxClientCnxns fix below.

2. Time-align the burst with JVM pause and leader metrics. The single most common cause is a JVM Stop-the-World GC pause that exceeds the heartbeat cadence. If zk_jvm_pause_time_ms p99 spikes within the same minute as the burst, GC is your trigger.

3. Check for an election in the same window. Pull zk_looking_count, zk_server_state, and zk_sum_leader_unavailable_time for all ensemble members. An election produces a burst of session expirations as clients reconnect during convergence.

4. Quantify client impact. Pull zk_num_alive_connections and zk_ephemerals_count. If both drop by the same fraction, sessions expired and ephemeral state was lost. Downstream systems that read those ephemerals (Kafka broker registration, HBase RegionServer registration) are now reacting.

5. Look for the network signature. OS-level TCP retransmits, NIC error counters, and listen-queue overflows all produce burst disconnects without server-side health issues. Compare burst timestamps across ensemble members: correlated bursts across all members point to a network or client-fleet event. A burst on one member points to that member’s GC or disk.

6. Check client library versions for known regressions. ZOOKEEPER-4921 documents that the Java client shipped in ZooKeeper 3.9.3 fails to reconnect after network failures and expires the session instead of retrying; it is fixed in 3.9.4 and later. If your clients are pinned to 3.9.3, the fix is a client library upgrade, not a server-side change.

Metrics and signals to monitor

Fixes

JVM GC pauses

If zk_jvm_pause_time_ms p99 aligns with the bursts, the trigger is GC. The fix is on the JVM, not on ZooKeeper itself.

• Confirm the GC algorithm. ZooKeeper’s launcher scripts do not set a collector, so on JDK 9+ the JVM default G1GC applies. ZGC (JDK 15+) reduces pause times for heaps above a few GB.
• Check heap sizing against data tree size. Pull zk_znode_count, zk_approximate_data_size, and zk_watch_count. If all three are growing, you are looking at the leading indicators of a GC death spiral. The fix is to cap the data tree (clean up unused znodes) or grow the heap.
• Inspect the GC log for Full GC events longer than tickTime (default 2000ms). A Full GC that long risks leader election on top of session expiry.
• Disable Transparent Huge Pages on the ZooKeeper host. THP can extend GC pauses significantly.

Do not restart ZooKeeper as the first action. A restart forces leader election, which produces another burst.

Network event

If the burst has no GC signature and no election, look at the path between clients and the ensemble.

• Pull OS retransmit counters and NIC error counters on the ZooKeeper hosts.
• Pull listen-queue overflows (TcpExtListenOverflows). If non-zero, raise clientPortListenBacklog in zoo.cfg instead of relying on the default. The documented default of -1 applies a socket backlog of 50 on Linux, and the OS somaxconn caps the effective value; both are too low for production reconnect storms.
• Check switch firmware and NIC offload settings. Bad firmware under load produces burst packet loss that looks identical to GC pauses from the metrics side.
• If clients sit behind a load balancer, check whether the balancer recently changed idle timeout or health check behavior.

maxClientCnxns rejection (different symptom)

If zk_connection_rejected is the moving counter rather than zk_connection_drop_count, the fix is to raise maxClientCnxns or fix the source IP concentration. The default of 60 per source IP is easily exceeded in containerized deployments where many pods share a host IP via NAT. The rejection is silent from the server side. Clients see connection refused or timeout.

Version-specific: 3.9.3 reconnect regression

ZOOKEEPER-4921 documents that the Java client shipped with ZooKeeper 3.9.3 fails to reconnect after network failures; it is fixed in 3.9.4 and later. Sessions die after a single reconnect failure instead of retrying. The fix is a client library upgrade.

Watch storms after the burst

Once sessions have expired and ephemeral nodes have vanished, watch notifications fire for every watcher of those nodes. If zk_packets_sent spikes while zk_packets_received stays flat, you are in the watch fan-out phase. The reconnection wave that follows can push zk_outstanding_requests to globalOutstandingLimit.

If the ensemble is destabilizing, you can temporarily block new connections at the firewall to let in-flight state settle, then gradually re-admit clients. Warning: this is disruptive. It drops all clients that are mid-reconnect and can extend the outage if done wrong. Use only as a last resort when the ensemble is already failing.

Prevention

• Monitor zk_jvm_pause_time_ms p99. GC is the number-one trigger. Alert on p99 above one-third of minSessionTimeout.
• Monitor zk_stale_sessions_expired directly. Alert on any non-zero rate outside maintenance. This catches a session storm before dependent services page.
• Tune clientPortListenBacklog. Set it explicitly in zoo.cfg to a value appropriate to your fleet instead of relying on the default.
• Use the 3.6+ percentile metrics. Avg/min/max latency are cumulative since server start and hide burst behavior. p99 and p999 catch the tail.
• Validate client session timeouts. Virtualized environments (especially burstable VMs) with sub-10s timeouts are too aggressive. The negotiated range is [2*tickTime, 20*tickTime] by default, with minSessionTimeout and maxSessionTimeout overriding.
• Track client library versions across the fleet. Regressions like the 3.9.3 reconnect bug survive silent rollout. Pinning and inventory prevent surprise session storms.
• Separate dataLogDir from dataDir. A shared disk produces fsync spikes that trigger elections, which in turn produce session storms.

How Netdata helps

• Per-second collection on zk_connection_drop_count, zk_stale_sessions_expired, and zk_num_alive_connections makes the shape of the burst visible. Minute-level scraping flattens a 20-second GC-driven burst into noise.
• ML anomaly detection on zk_jvm_pause_time_ms surfaces GC pauses before they cross a fixed threshold, which matters because the right threshold depends on each ensemble’s minSessionTimeout.
• Correlating zk_looking_count with zk_connection_drop_count on one timeline distinguishes election-caused storms from GC-caused storms without manual log hopping.
• Composite alerts that gate zk_connection_drop_count on zk_uptime suppress false positives from rolling restarts and cold-start reconnection waves.
• Side-by-side per-node dashboards let you distinguish a one-node GC problem from a fleet-wide network event at a glance.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper read latency high: memory reads that should never be slow