ZooKeeper session expiration storm: the ephemeral-node thundering herd

A session expiration storm is the worst-case thundering herd in a coordination service. Many clients lose contact long enough for the ensemble to declare their sessions dead. The cluster then deletes every ephemeral node owned by those sessions and fires every watch attached to those nodes. Every disconnected client reconnects at the same time, recreates its ephemeral nodes, and re-registers watches, hammering an ensemble that is already stressed.

This is not a server crash. The ZooKeeper nodes are typically still up and answering ruok. The damage is on the client side: Kafka brokers see their registration disappear, HBase RegionServers are marked dead, Solr shards lose leadership.

The defining signature is a V-shape in zk_num_alive_connections: a sharp drop, a plateau roughly the width of the negotiated session timeout, then a reconnect spike. zk_ephemerals_count mirrors the drop. zk_packets_sent surges as watch notifications fan out. zk_stale_sessions_expired and zk_connection_drop_count jump in lockstep.

What this means

When a client misses heartbeats for longer than its negotiated session timeout, the ensemble expires the session. ZooKeeper treats this as authoritative: it deletes every ephemeral znode owned by that session and clears all watches attached to those znodes. Other clients watching those znodes receive a watch event notification. For service discovery built on ephemeral nodes (Kafka broker IDs, HBase RegionServer ephemerals, Curator leader locks), this looks identical to the owning service dying.

The recovery path is worse than the loss. Every expired client reconnects, re-creates its ephemeral nodes, and re-registers watches. The ensemble absorbs a burst of new connections, create operations, and watch registrations while it is still delivering deletion notifications from the previous wave. If the trigger is still active (a saturated network path, an overloaded leader, a GC death spiral on the client fleet), the second wave of sessions expires too.

flowchart TD
    A[Trigger: net event, client GC, leader failover] --> B[Clients miss heartbeats]
    B --> C[Session timeout elapses]
    C --> D[zk_stale_sessions_expired jumps]
    C --> E[Ephemeral nodes deleted]
    E --> F[Watches fire: zk_packets_sent spikes]
    E --> G[Dependent services react]
    C --> H[zk_num_alive_connections drops]
    H --> I[zk_ephemerals_count drops]
    C --> J[Expired clients reconnect]
    J --> K[Reconnect spike: creates, watch re-reg]
    K --> L{Ensemble absorbs burst?}
    L -->|Yes| M[Recovery]
    L -->|No, trigger persists| C

Common causes

Quick checks

All read-only and safe to run during the incident. Note: the four-letter-word commands (ruok, mntr, dump, cons) require explicit whitelisting via 4lw.commands.whitelist in zoo.cfg. In ZooKeeper 3.5.3+, only srvr is enabled by default (isro is added only when read-only mode is on). If these return empty, check the whitelist before assuming ZK is down.

# Confirm the ZK process is alive and not in read-only mode
echo ruok | nc localhost 2181
echo isro | nc localhost 2181

# Check ensemble state, recent elections, uptime
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_uptime|zk_looking_count'

# Watch the V-shape forming
echo mntr | nc localhost 2181 | grep -E 'zk_num_alive_connections|zk_ephemerals_count|zk_packets_sent|zk_packets_received'

# Confirm session expirations are the source of the connection drop
echo mntr | nc localhost 2181 | grep -E 'zk_stale_sessions_expired|zk_connection_drop_count'

# Check for ZK-side GC pause correlation
echo mntr | nc localhost 2181 | grep jvm_pause

# Check for leader unavailability window
echo mntr | nc localhost 2181 | grep leader_unavailable_time

# Check for fsync stalls that could have caused leader to miss heartbeats
echo mntr | nc localhost 2181 | grep fsynctime

# Check for maxClientCnxns saturation
echo mntr | nc localhost 2181 | grep -E 'zk_connection_rejected|zk_connection_request_count'

# Check for write pipeline saturation that would block session recreation
echo mntr | nc localhost 2181 | grep -E 'zk_outstanding_requests|zk_throttled_ops'

How to diagnose it

1. Confirm it is a session storm, not a server crash. ruok returns imok and isro returns rw. The server is up; the clients are not.

2. Confirm the V-shape. Sample zk_num_alive_connections every few seconds. You should see a sharp drop, a plateau roughly the width of the session timeout, then a reconnect spike. If you only catch the spike, you missed the drop. Check zk_stale_sessions_expired for the delta.

3. Confirm ephemerals dropped. zk_ephemerals_count should mirror the connection drop within a few seconds. If connections dropped but ephemerals did not, suspect a session-tracking bug rather than a clean storm (see the stale-ephemeral conditions below: ZOOKEEPER-4837 is still open on current releases; ZOOKEEPER-2774 and ZOOKEEPER-2355 were fixed in 3.4.11 / 3.4.11-3.5.4-3.6.0 respectively).

4. Confirm the watch fan-out. zk_packets_sent should spike relative to zk_packets_received. That gap is watch notifications firing.

5. Time the trigger. The drop in zk_num_alive_connections is when clients lost contact. Session expirations lag by the session timeout. The reconnect spike follows. Find what happened at the drop time.

6. Rule out ZK-side causes. Check zk_looking_count (leader failover?), zk_jvm_pause_time_ms p99 (ZK paused?), zk_fsynctime p99 (write stall?), zk_sum_leader_unavailable_time (leader down?). If any of these moved before the drop, the trigger is server-side.

7. If ZK-side signals are clean, the trigger is between clients and ZK. Look at network path telemetry (retransmits, LB events, DNS), client-side GC logs, and client deploy or restart events. Cross-check downstream service logs: Kafka brokers log SessionExpiredException from SessionExpireListener, HBase logs RegionServer session loss, Solr logs shard leadership loss.

8. Estimate downstream blast radius. Identify which ephemeral znodes vanished by inspecting the affected subtrees. For Kafka pre-KRaft, check /brokers/ids/. For HBase, check /hbase/rs/. For Solr, check collection state znodes. The set of vanished ephemerals is the set of services that reacted.

Metrics and signals to monitor

Fixes

If the trigger is still active

Stop the bleeding first. If the network event, GC death spiral, or leader instability is still firing, the storm renews itself.

• Network partition: fix the path. Until the path is healthy, clients cannot reconnect and the next wave of sessions also expires.
• Client-side GC storm: client owners must reduce heap pressure or switch collectors. You cannot fix this from the ZK side.
• Leader failover loop: address the underlying cause (disk, GC, network between ensemble members). See the leader election storm guide.

If the ensemble is overwhelmed by the reconnect wave

The reconnect spike can saturate the ZK request pipeline even after the original trigger is gone.

• Watch zk_outstanding_requests and zk_throttled_ops. If zk_outstanding_requests is climbing toward globalOutstandingLimit (default 1000), the leader is saturated.
• Watch zk_connection_rejected. If this is incrementing, maxClientCnxns (default 60 per source IP) is the bottleneck and reconnecting clients are being refused silently. In containerized deployments where many pods share a host IP, raising maxClientCnxns may be necessary. There is no server-side log for the rejection by default.
• Watch the accept queue. clientPortListenBacklog defaults to -1, which makes the JVM fall back to its own backlog of 50 pending connections (OpenJDK ServerSocketChannelImpl calls Net.listen(fd, backlog < 1 ? 50 : backlog)), easy to overflow during a reconnect herd. If you cannot tune it now, rate-limit reconnects at the network layer (a temporary firewall rule that admits new connections at a controlled rate). This is disruptive and must be coordinated with service owners.

Stale ephemeral nodes after the storm

In some versions, ephemeral nodes can survive their owning session under specific failure conditions: a network issue during the PROPOSAL request (ZOOKEEPER-4837, still open), a follower failing while reading the proposal packet (ZOOKEEPER-2355, fixed in 3.4.11/3.5.4/3.6.0 but reported as not fully resolved by ZOOKEEPER-4837), or an unexpected system clock change to an earlier time (ZOOKEEPER-2774, fixed in 3.4.11).

If zk_ephemerals_count does not return to its pre-storm baseline while zk_num_alive_connections does, suspect stale nodes. Stale ephemeral nodes break dependent services: a Kafka broker ID that is “alive” with no broker behind it, a Curator leader lock held by a dead process. Identify owning session IDs with dump, confirm the sessions are gone, then remove the nodes manually with the ZK CLI.

Warning: deleting ephemeral znodes by hand is destructive. A service whose node you delete will lose its registration and may be declared dead by its peers. Confirm the owning session is truly gone and coordinate with the owning service team before touching production znodes.

Dependent service recovery

ZK recovering does not equal downstream recovering.

• Kafka (pre-KRaft): broker deregistration triggers partition leadership changes. Watch controller failover and under-replicated partition counts. Brokers must re-register and re-create their ephemeral before they count as live.
• HBase: RegionServer session loss triggers region reassignment. Monitor the Master for assignment backlog.
• Solr: shard leader elections run through ZK. Watch for repeated elections after the storm clears.

Prevention

• Separate dataLogDir from dataDir on dedicated low-latency storage. fsync stalls are a top cause of leader failover, which is a top cause of session storms.

• Monitor zk_jvm_pause_time_ms p99 on the ZK nodes. GC pauses on ZK cause heartbeats to be missed server-side. p99 should stay well below minSessionTimeout (default 2 x tickTime = 4000ms).

• Push client owners to monitor their own GC. Client-side GC pauses are the most common cause of fleet-wide session expiration. Clients should log ConnectionLoss and SessionExpired events and track reconnection rate.

• Review client session timeouts against network reality. The negotiated timeout is bounded by the server: minimum is 2 x tickTime, maximum is 20 x tickTime. With tickTime=2000 the server-enforced range is 4s to 40s. Clients requesting shorter timeouts than the server minimum silently get the minimum, which mismatches application assumptions.

• Review load-balancer idle timeouts against the negotiated session timeout. An LB that idle-times out a quiet ZK connection before the client sends its next heartbeat will cause session loss.

• Tune clientPortListenBacklog for fleets with thousands of clients. The default of -1 resolves to the JVM’s 50-connection fallback, which is too low for reconnect storms; set it explicitly to absorb the worst reconnect burst (hundreds is a common starting point).

• Consider local sessions (3.5.0+) if your workload allows. localSessionsEnabled avoids quorum confirmation for each session, reducing the cost of mass reconnection. Local sessions cannot create ephemeral nodes unless localSessionsUpgradingEnabled is also enabled; that behavior is unchanged in 3.9.x.

• Consider WatchManagerOptimized (3.6.0+), selected via the watchManagerName system property. It includes lazy dead-watcher cleanup with configurable threads (watcherCleanThreadsNum, default 2) and helps when many sessions expire simultaneously and generate many dead watchers.

• Run periodic chaos exercises. Deliberately kill a ZK leader in a controlled window and observe how the client fleet and dependent services react. Most session-storm incidents expose client-side bugs that only surface under failover conditions.

How Netdata helps

Per-second collection and anomaly detection help catch the V-shape signature before downstream services page.

• Per-second zk_num_alive_connections, zk_ephemerals_count, zk_packets_sent, and zk_stale_sessions_expired make the V-shape and its session-timeout lag visible at the resolution needed to distinguish a storm from a single client disconnect.
• Correlating zk_jvm_pause_time_ms p99 with the connection drop tells you in one view whether the trigger was ZK-side GC or external.
• Correlating zk_looking_count and zk_sum_leader_unavailable_time with the drop tells you whether a leader gap preceded the storm.
• Anomaly detection on the zk_packets_sent to zk_packets_received ratio surfaces watch fan-out before the connection drop is large enough to threshold.
• The same signals on leader and followers, side by side, make it obvious whether the trigger is ensemble-wide or specific to one node.
• Composite alerts (connection drop plus zk_stale_sessions_expired delta plus ephemerals drop within a session-timeout window) match the actual failure pattern rather than threshold-gaming individual metrics.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper read latency high: memory reads that should never be slow