ZooKeeper unexpected leader election: finding why the leader dropped

An unexpected ZooKeeper leader election is an availability event. While the ensemble is in LOOKING state, no writes are processed. Systems that depend on ZK for coordination queue or fail their mutations, and zk_sum_leader_unavailable_time climbs. Treat every unplanned election like a database failover: the cluster recovered, but you still need the root cause.

The cause is on the old leader or on the path to it. Followers call an election when they have not heard from the leader within syncLimit * tickTime (default 5 * 2000ms = 10 seconds). The question is always: what stopped the old leader from sending heartbeats, or what stopped a quorum of followers from receiving them.

The usual suspects are a JVM stop-the-world GC pause, an fsync stall on the transaction log, a network fault between the leader and a quorum of followers, or a process crash. The playbook: confirm an election happened, identify the old leader, inspect its GC and fsync signals in the seconds before the event.

Metric availability notes (verified against ZooKeeper source): zk_looking_count, zk_jvm_pause_time_ms (and zk_p99_jvm_pause_time_ms), and zk_uptime are in mntr since 3.6; zk_sum_leader_unavailable_time since 3.7. The last-processed zxid is not exposed by mntr in any version — read it from the srvr command (Zxid: 0x...) or from logs.

What an election means

zk_server_state reports “looking” during the election, then “leader” on the new leader and “follower” on the rest. The most reliable indicator of how many elections have occurred is the epoch, which is the upper 32 bits of zk_zxid. The epoch increments on each successful election. If you do not have log access, the zxid delta tells you whether this was a single election or a storm.

During a rolling restart, elections happen only when the current leader is restarted. The total depends on restart ordering: if leadership moves to a node that is later restarted, you get another election. Correlate any elections outside a maintenance window with GC, disk, and network signals.

Common causes

Quick checks

Run these on each ensemble node. They are all read-only.

# Confirm role and that an election is not still in progress
echo mntr | nc localhost 2181 | grep zk_server_state

# Confirm the ensemble has a leader and is read-write
echo isro | nc localhost 2181

# Count elections observed by this node since start
echo mntr | nc localhost 2181 | grep zk_looking_count

# Cumulative leader-unavailable time (ms); non-zero delta means writes were impossible
echo mntr | nc localhost 2181 | grep -E 'zk_.*leader_unavailable_time'

# Current zxid (upper 32 bits is the epoch); mntr does not expose zxid
echo srvr | nc localhost 2181 | grep '^Zxid'

# Detect a restart on this node (uptime near zero means crash or restart)
echo mntr | nc localhost 2181 | grep zk_uptime

# Election-related log lines from this node
grep -E "LEADING|FOLLOWING|LOOKING|New election" /var/log/zookeeper/zookeeper.log | tail -30

# Explicit fsync warnings (logged when fsync exceeds fsync.warningthresholdms, default 1000ms)
grep "fsync-ing the write ahead log" /var/log/zookeeper/zookeeper.log | tail -20

Whichever node most recently transitioned out of “leader” is the old leader. Focus the rest of the investigation there.

How to diagnose it

flowchart td
  A[zk_looking_count increments] --> B{zk_uptime reset on old leader?}
  B -- Yes --> C[Process crash or OOMKill]
  B -- No --> D{GC pause before event in GC log?}
  D -- Yes --> E[GC stall stopped heartbeats]
  D -- No --> F{fsync spike before event?}
  F -- Yes --> G[Transaction log disk stalled]
  F -- No --> H{Network errors around event?}
  H -- Yes --> I[Partition or blocked port]
  H -- No --> J[Check zxid overflow, rolling restart]

1. Confirm an election actually occurred. Check zk_looking_count and zk_server_state across all nodes. One node briefly in LOOKING is a transient blip; multiple nodes in LOOKING with leader-unavailable time incrementing is a real election.
2. Identify the old leader and timestamp the event. Match the transition in the ZK log: grep -E "LEADING|FOLLOWING|New election" /var/log/zookeeper/zookeeper.log. The node that went from LEADING to FOLLOWING (or LOOKING) is the one that dropped.
3. Rule out a process crash first. Check zk_uptime on the old leader. A reset to near-zero means the process restarted. Check the process manager (systemd, supervisor, container runtime) and dmesg for OOMKill. A crash is a different root cause from unresponsiveness.
4. Inspect GC on the old leader in the seconds before the event. A Full GC pause longer than tickTime (default 2000ms) is enough to miss heartbeats. Look in the GC log around the election timestamp. With scraped metrics, look for zk_jvm_pause_time_ms p99 spikes immediately before the election.
5. Inspect fsync on the old leader in the seconds before the event. If GC looks clean, the next likely cause is the transaction log disk. Look for “fsync-ing the write ahead log” warnings. Check iostat -x on the dataLogDir device for the period leading up to the election.
6. Check for a network cause. If the old leader stayed responsive and its disk was fine, the issue was between the leader and a quorum of followers. Confirm both inter-server ports (the follower-to-leader port and the leader election port) are reachable between all pairs. A firewall change that blocks only the election port is invisible until the next election, then catastrophic.
7. Rule out zxid overflow and rolling restarts. If write rate is extreme, grep the log for “zxid lower 32 bits have rolled over” - this is an intentional election, not a bug. Confirm the timestamp did not line up with a deployment.

Metrics and signals to monitor

Fixes

GC pauses on the leader

GC pauses are the most common cause of avoidable leader elections. Confirm with the GC log first, then act.

• Increase heap if zk_znode_count and zk_approximate_data_size have grown into the heap budget.
• Review the collector. Java 9+ defaults to G1GC; on Java 15+, ZGC substantially reduces pause time. The ZooKeeper startup scripts (zkEnv.sh/zkServer.sh) do not set a collector, so the JVM default applies.
• Find the leak. If heap trough is rising over days, look for unbounded znode creation (frameworks that create per-task nodes without cleanup) and fix the producer.

Tradeoff: a bigger heap makes individual Full GC pauses longer. Prefer fixing the data tree size or the collector before raising heap.

fsync stalls

The single most impactful configuration change in ZooKeeper is putting the transaction log on a dedicated device. Confirm dataLogDir is set and points to different storage than dataDir.

Other steps:

• On cloud, check whether the volume is being throttled. Burst credit exhaustion on EBS gp2/gp3 is a classic cause.
• Check for colocated I/O on the same disk and move it.
• Check iostat -x for %util near 100 on the txnlog device.

Network faults

• Verify both quorum ports are reachable between every pair of ensemble members, not just from the leader out.
• Check security groups, firewalls, and any TCP proxy in the path. Sidecars such as Envoy or Istio can introduce election-time races.
• DNS: if the ensemble is configured with hostnames, a DNS hiccup can look like a network partition.

Process crashes and OOMKill

If zk_uptime reset, check the process manager logs and dmesg. An OOMKill points back to heap sizing (see GC fixes above). A JVM-level crash (segfault in native code) is rarer and usually version-specific; collect the hs_err file and check the ZK version against known issues.

Rolling restarts and alert suppression

During a rolling restart, elections happen only when the current leader is restarted. Suppress election alerts for the maintenance window and use zk_sum_leader_unavailable_time as the backstop: if unavailability exceeds the expected per-election duration, something else is wrong.

Prevention

• Treat fsync latency and JVM pause time as first-class signals. Most teams only discover these during an election postmortem. zk_jvm_pause_time_ms is in mntr since 3.6; fsync stats are also exposed there (zk_fsynctime_*).
• Separate dataLogDir from dataDir on dedicated storage. Prevents the most common fsync-driven elections.
• Track zk_looking_count and zk_sum_leader_unavailable_time as rates, not absolute values. Absolute counters are not useful; the rate of change is.
• Test failover deliberately. In a controlled maintenance window, stop the leader process and confirm the ensemble re-elects within expected time. Warning: this causes a real election and briefly stops all writes. Only do this outside peak traffic.
• Watch the data tree. zk_znode_count and zk_approximate_data_size growing over time will eventually push the leader into longer GC pauses.

How Netdata helps

• Per-second granularity on zk_server_state, zk_looking_count, and zk_sum_leader_unavailable_time catches short elections that minute-scraped systems miss.
• Correlating zk_jvm_pause_time_ms p99 with zk_looking_count increments pinpoints GC as the cause without manually trawling GC logs.
• Correlating zk_fsynctime with election events isolates disk stalls from JVM stalls, which is the hardest call to make during an incident.
• Anomaly detection on zk_zxid epoch increments surfaces unplanned elections even when no static alert threshold was tuned.
• Cold-start awareness via zk_uptime suppresses noise from rolling restarts while still paging on real leader loss.

Related guides

• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing