ZooKeeper leader election storm: an ensemble that keeps re-electing

zk_looking_count is incrementing on multiple nodes. The ZooKeeper log fills with repeated LEADING and FOLLOWING transitions. Every few seconds or minutes, the ensemble elects a new leader, and that leader quickly loses quorum. Writes are intermittent, and downstream systems that depend on ZooKeeper for coordination (Kafka controllers, HBase region assignment, distributed locks) experience cascading failures.

This is a leader election storm. Unlike a single failover where remaining nodes elect a stable replacement, here every potential leader hits the same wall. The root cause is shared across ensemble members: fsync stalls on overloaded storage, GC pauses exceeding the quorum timeout, or intermittent network failures between server pairs. Because every candidate experiences the same problem, no leader holds the role long enough for the cluster to recover.

More than one unplanned election per hour is instability. Each election is an availability event. During the LOOKING phase, no writes are processed, client sessions approach expiry, and downstream coordination systems react to the gap.

What this means

In a healthy ensemble, elections happen during planned maintenance or single-node failure. The remaining nodes elect a new leader, the cluster converges, and writes resume within seconds, inside the initLimit x tickTime window.

An election storm is a different pattern. The ensemble cycles through elections without stabilizing. The mechanism is a feedback loop: a shared resource problem causes the current leader to become unreachable. Followers exceed the syncLimit x tickTime window (default: 5 x 2000ms = 10 seconds), declare the leader dead, and start a new election. The new leader is elected, but it runs on the same hardware with the same overloaded disk or the same JVM heap pressure. Within seconds, it too misses heartbeats. The cycle repeats.

flowchart TD
    A["Shared root cause"] --> B["Leader misses heartbeats"]
    B --> C["Followers declare leader dead"]
    C --> D["Followers enter LOOKING"]
    D --> E["New leader elected"]
    E --> F["Same problem recurs"]
    F --> B

Key diagnostic insight: because the root cause is shared, every node is affected. If only one node were the problem, the ensemble would elect a different leader and stabilize. The storm pattern means you need to look for a common factor across all members: the same storage tier, the same heap configuration, the same network path.

Common causes

Note: the percentile metrics shown above require ZooKeeper 3.6+ (jvm_pause_time and leader_unavailable_time only from 3.7+), and mntr must be listed in 4lw.commands.whitelist (default since 3.5.3: only srvr).

Quick checks

Run these on each ensemble member. They are read-only and safe for production.

# Current role - any node reporting other than leader/follower is in trouble
echo mntr | nc localhost 2181 | grep zk_server_state

# Election count since process start
echo mntr | nc localhost 2181 | grep zk_looking_count

# Cumulative leader unavailability (sum = total ms, cnt = episode count)
echo mntr | nc localhost 2181 | grep zk_.*leader_unavailable_time

# Fsync latency percentiles - smoking gun for disk problems
echo mntr | nc localhost 2181 | grep zk_.*fsynctime

# JVM pause time percentiles - smoking gun for GC problems
echo mntr | nc localhost 2181 | grep zk_.*jvm_pause

# Quorum ACK latency (leader only) - network and follower processing
echo mntr | nc localhost 2181 | grep zk_.*quorum_ack_latency

# Synced followers and pending syncs (leader only)
echo mntr | nc localhost 2181 | grep -E "synced_followers|pending_syncs"

# Recent fsync warnings from the log
grep "fsync-ing the write ahead log" /var/log/zookeeper/zookeeper.log | tail -20

# Election transitions in the log
grep -E "LEADING|FOLLOWING|LOOKING" /var/log/zookeeper/zookeeper.log | tail -30

# OS-level disk latency on the transaction log device
iostat -x 1 5

How to diagnose it

1. Confirm it is a storm, not a single election. Check zk_looking_count on all members. If only one node incremented, it may have restarted. If multiple nodes show increments within the same window, the ensemble is cycling.

2. Correlate election timestamps with fsync latency. Pull zk_max_fsynctime (3.6+; mntr exposes fsynctime only as avg/min/max/cnt/sum) alongside election events from the log. If fsync spikes immediately precede each LOOKING transition, the disk is the trigger. The fsync warning log line fires when fsync exceeds fsync.warningthresholdms (default 1000ms). Any appearance of that message during a storm is a direct signal.

3. Correlate election timestamps with JVM pause time. Pull zk_p99_jvm_pause_time_ms (3.7+, requires jvm.pause.monitor=true) alongside election events. GC pauses approaching syncLimit x tickTime (10 seconds with defaults) will cause followers to declare the leader dead. Check the GC log for Full GC events aligned with each election.

4. Check for network-specific patterns. If neither fsync nor GC correlates, examine zk_p99_quorum_ack_latency for spikes on specific follower connections. ZooKeeper uses two inter-server ports: one for follower-to-leader communication and one for leader election. If only one is blocked, the ensemble may elect a leader but cannot re-elect after a failure. Check for asymmetric connectivity between specific pairs.

5. Verify ZooKeeper version. If running 3.5.7 or earlier on a 3-node ensemble, ZOOKEEPER-2164 is a known cause. When the leader shuts down, the remaining two nodes both vote, but the winner does not receive votes within 5 seconds because the connection attempt to the dead node does not time out in time. By the time votes arrive, the follower has given up. This is fixed in 3.5.8, 3.6.1, and 3.7.0 (ZOOKEEPER-2164 fixVersions per JIRA).

6. Check for sidecar proxy interference. If running ZooKeeper behind a service mesh (for example, Istio sidecar proxies), connection timeouts introduced by the sidecar can trigger repeated elections. This is tracked as ZOOKEEPER-3923.

Metrics and signals to monitor

Fixes

Fsync stalls

Confirm with zk_max_fsynctime and iostat -x on the transaction log device. If dataLogDir is not configured, ZooKeeper writes transaction logs to dataDir, sharing the disk with snapshots. Snapshot I/O competes with fsync, causing latency spikes during snapshot creation. Separating dataLogDir onto a dedicated device is the single most impactful configuration change for ZooKeeper write path stability.

If dataLogDir is already on a separate disk, check whether that disk is shared with other I/O-heavy workloads. In cloud environments, EBS gp2/gp3 burst credit exhaustion causes sudden fsync latency cliffs. The transition from burst to baseline is abrupt. Check provisioned IOPS and burst credit balance.

If all members share the same storage tier and that tier is the bottleneck, the fix must address the storage layer across all members. Moving one member to better storage does not help because any member can become leader.

GC pauses

Confirm with zk_p99_jvm_pause_time_ms (3.7+) and the GC log. If Full GC pauses exceed 1 second, the JVM is under heap pressure. Check heap usage against data tree size: zk_znode_count and zk_approximate_data_size tell you how much live data the heap carries. If znode count is growing without bound, application frameworks are leaking nodes.

Short-term mitigation: increase heap size. But if the data tree is growing, this only delays the next crisis. Long-term fix: identify and clean up the leaking subtree, then size the heap with at least 30% headroom above the live data set.

GC algorithm choice matters. G1GC has been the default JVM collector since JDK 9 (ZooKeeper’s launch scripts do not set a collector). Older versions may use CMS or Parallel GC, which produce longer stop-the-world pauses on large heaps. ZGC (available from JDK 15+) dramatically reduces pause times and is worth evaluating for ensembles with large data trees.

Also check Transparent Huge Pages on the host. THP can cause GC pauses to be significantly longer because GC must page in objects to scan them. Verify with cat /sys/kernel/mm/transparent_hugepage/enabled and disable if set to always.

Network issues between pairs

If fsync and GC both look clean, the problem is likely network. ZooKeeper requires both inter-server ports to be reachable for healthy operation: the follower-to-leader port and the leader election port. A firewall blocking only the election port will not be noticed until the next election, at which point the ensemble cannot converge.

Check for asymmetric connectivity: can every node reach every other node on both ports? In cloud environments, security group changes are a common cause. If the ensemble spans datacenters, network latency between sites increases quorum_ack_latency baseline. Cross-datacenter deployments may need maxTimeToWaitForEpoch tuning.

If running with a service mesh sidecar, verify that the sidecar’s connection timeout does not interfere with ZooKeeper’s quorum communication timeout (quorumCnxnTimeoutMs, default -1, which means it uses syncLimit x tickTime).

maxTimeToWaitForEpoch tuning (3.6+)

Introduced in 3.6.0, maxTimeToWaitForEpoch controls how long a leader waits for epoch packets from a majority after receiving a LOOKING notification from a voter. If the leader does not receive epoch packets within this window, it goes back to LOOKING and triggers a new election. If this value is too low for your network latency profile, the leader may prematurely abandon leadership on stray notifications.

The property defaults to -1, which disables the wait entirely. If elections correlate with stray LOOKING notifications rather than heartbeat misses, check this setting. The admin guide notes this can be tuned to reduce quorum unavailability in cross-datacenter environments.

Version-specific bugs

If running 3.5.7 or earlier, upgrade. ZOOKEEPER-2164 causes 3-node ensembles to repeatedly re-elect after a leader shutdown because the connection attempt to the dead node does not time out fast enough. This is fixed in 3.5.8, 3.6.1, and 3.7.0 (ZOOKEEPER-2164 fixVersions per JIRA). Versions 3.5.7 and earlier are reported as unreliable for 3-node leader election.

If running 3.9.x, check for ZOOKEEPER-4925 (diff sync can introduce a hole in a stale follower’s committed log, fixed in 3.9.4 and 3.10.0) and ensure you are on the latest patch release.

Prevention

• Separate transaction log storage. Put dataLogDir on a dedicated low-latency device with no competing I/O. This is the highest-leverage configuration change for write path stability.
• Enable GC logging. Without GC logs, you cannot correlate GC pauses with election events. Enable with -Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m (JDK 9+ unified logging).
• Monitor znode growth. All ensemble members carry the same data tree. They will OOM at the same time. Track zk_znode_count and zk_approximate_data_size against heap capacity.
• Size heap with headroom. Maintain at least 30% free heap above the live data set. The degradation curve is cliff-edge above 85% sustained usage.
• Disable Transparent Huge Pages. THP amplifies GC pause duration on ZooKeeper hosts.
• Configure autopurge. Set autopurge.purgeInterval and autopurge.snapRetainCount to prevent unbounded transaction log and snapshot accumulation.
• Test failover regularly. Controlled leader kills validate that the ensemble can re-elect within the initLimit x tickTime window and that monitoring detects the event.
• Verify both inter-server ports. Network issues that block only the election port are invisible until the next election.

How Netdata helps

• Per-second collection of zk_looking_count across all ensemble members surfaces election events the moment they happen, rather than after a scrape interval delay.
• Correlating zk_p99_fsynctime and zk_p99_jvm_pause_time_ms against election timestamps distinguishes disk-caused storms from GC-caused storms in seconds.
• ML anomaly detection on zk_server_state transitions catches rapid role cycling that a static threshold might miss during the first minutes of a storm.
• Leader-only metrics (zk_synced_followers, zk_pending_syncs, zk_p99_quorum_ack_latency) are collected from whichever node currently holds the leader role, so replication health remains visible even during transitions.
• zk_sum_leader_unavailable_time deltas quantify the actual write availability impact, converting an election count into a severity signal that reflects business impact.

Related guides

• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing