ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum

A single ZooKeeper node sits in LOOKING long after the ensemble has settled, or flaps between LOOKING and FOLLOWING. The rest of the ensemble holds a stable leader with healthy write throughput. Restart the orphaned node and it drops back into LOOKING. Restart the leader and it briefly rejoins, then falls out again.

This is not quorum loss. Quorum loss is every node entering LOOKING at once because no majority can form. That case is covered in ZooKeeper quorum loss: no leader elected and every write is failing. This article covers the narrower symptom: one permanently-orphaned node while the rest of the ensemble serves traffic.

The causes cluster into three themes. The node cannot reach its peers on the election port (3888) or quorum port (2888) in the direction that matters. The node’s local state is incompatible with the leader (zxid ahead of the leader’s epoch, or missing epoch files). Or the node’s local disk and JVM cannot persist a vote fast enough to survive an election round. Distinguishing among these three is the whole job, and the rest of this article walks through how to do it without making the situation worse.

What this means

A ZooKeeper ensemble member participates in two distinct TCP planes. The quorum port (2888 by default, the first port in server.X=host:2888:3888) carries follower-leader traffic: PROPOSE, ACK, COMMIT. The leader election port (3888, the second port) carries FastLeaderElection notifications during leader election. Both must be reachable, in both directions, between every pair of ensemble members. A firewall that blocks 3888 in one direction is invisible until the next election. At that point the affected node cannot send or receive votes and gets stuck in LOOKING indefinitely.

When a node enters LOOKING, it opens TCP connections to every peer on 3888 and exchanges vote notifications. To leave LOOKING it must receive a valid notification from a quorum of peers establishing a leader. If 3888 is reachable from peers but not from this node, or vice versa, the vote never completes. The same applies to the subsequent sync phase on 2888. A node that elected a leader but cannot reach it on 2888 falls back into LOOKING and the cycle repeats.

This produces the characteristic “flapping” signature. zk_looking_count on the affected node increments steadily while zk_looking_count on every other node is flat. The rest of the ensemble holds a stable leader in BROADCAST. From the leader’s point of view, zk_synced_followers is one short of ensemble_size - 1.

flowchart TD
    A[One node stuck in LOOKING] --> B{Other nodes have
stable leader?}
    B -- No --> C[Quorum loss:
different playbook]
    B -- Yes --> D{Can this node reach
peers on 3888?}
    D -- No --> E[Election port blocked
or DNS to loopback]
    D -- Yes --> F{zxid comparable
to leader?}
    F -- Far behind --> G[Long catch-up
or SNAP sync]
    F -- Ahead of leader --> H[Unrecoverable:
rebuild from leader]
    F -- Comparable --> I{fsync or GC
healthy on this node?}
    I -- No --> J[Disk or JVM stalls
preventing vote persistence]
    I -- Yes --> K[Known bug pattern
ZOOKEEPER-2938]

Common causes

Quick checks

Run these read-only commands before changing anything.

# This node's state. Should report follower (or leader). LOOKING means election in progress.
echo srvr | nc localhost 2181 | grep Mode

# Functional state. rw means serving writes; ro means read-only mode (quorum lost).
echo isro | nc localhost 2181

# Election and state counters on this node.
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_looking_count|zk_uptime|zk_zxid'

# Same metrics from the leader's perspective.
echo mntr | nc <leader-host> 2181 | grep -E 'zk_server_state|zk_followers|zk_synced_followers|zk_pending_syncs'

# Compare zxid across every ensemble member. They should match within a few transactions.
for h in zk1 zk2 zk3; do
  printf '%s ' "$h"
  echo mntr | nc "$h" 2181 | grep zk_zxid
done

# TCP reachability from this node to every peer on both inter-server ports.
for h in zk1 zk2 zk3; do
  for p in 2888 3888; do
    timeout 2 bash -c "exec 3<>/dev/tcp/$h/$p" 2>/dev/null && echo "$h:$p open" || echo "$h:$p blocked"
  done
done

# What the affected node is logging right now.
tail -n 200 /var/log/zookeeper/zookeeper.log | grep -E 'LOOKING|FOLLOWING|LEADING|smaller server|Got zxid|Not following'

# Cold-start suppression check. If uptime is small, give the node time to settle before diagnosing.
echo mntr | nc localhost 2181 | grep zk_uptime

Replace zk1 zk2 zk3 and /var/log/zookeeper/zookeeper.log with your actual hostnames and log path. The four-letter commands require 4lw.commands.whitelist to include srvr, mntr, and isro (ZooKeeper 3.5.3+). On 3.6+, the AdminServer on port 8080 exposes the same data over HTTP, for example /commands/server_stats and /commands/leader.

How to diagnose it

1. Confirm only one node is affected. Check zk_server_state on every ensemble member. If two or more nodes are in LOOKING, you have a different problem; see the quorum-loss guide. This article applies only when exactly one node is stuck and the rest have a stable leader.

2. Check whether this is a cold start. If zk_uptime on the affected node is below 300 seconds, suppress investigation for a few minutes. Large data trees take time to load from snapshot plus transaction log replay. The node can look stuck while it is actually recovering.

3. Verify bidirectional election-port reachability. Run the TCP reachability loop in Quick checks from this node to every peer on 3888, then run the same check from one peer back to this node. Asymmetric blocking is the most common cause of a single stuck node. A firewall rule that allows outbound 3888 but blocks inbound 3888 (or vice versa) is invisible until election.

4. Check whether DNS resolves this node’s hostname to a loopback address. In containerized deployments (Kubernetes, Docker, Strimzi), getent hosts <this-node-name> may return 127.0.0.1, which causes the node to bind 3888 to loopback and become unreachable from peers. The server.X= entry in zoo.cfg for the local node should use a routable IP, or 0.0.0.0:2888:3888 to bind to all interfaces. The 0.0.0.0 local entry only affects the local bind and still works on 3.8.x/3.9.x; it does not break quorum TLS as long as the peer entries (the addresses other members use to reach this node) remain hostnames, because ssl.quorum.hostnameVerification (new in 3.5.5, on by default) validates the connecting side’s hostname against the certificate.

5. Compare zxid across the ensemble. A node with a zxid significantly behind the leader is in long catch-up. If the leader must send a full snapshot (SNAP sync), this can take minutes for a large data tree. A node with a zxid ahead of the leader’s epoch is in the unrecoverable “Got zxid X expected Y” state and must be rebuilt from the leader.

6. Check local fsync and GC. Look at zk_fsynctime p99 and zk_jvm_pause_time_ms p99 on the affected node. Election rounds have timeouts governed by tickTime (default 2000ms). If fsync or GC stalls push vote persistence past that boundary, the node cannot complete an election round and re-enters LOOKING.

7. Inspect the log for known failure signatures. The string “Have smaller server identifier, so dropping the connection” indicates the ZOOKEEPER-2938 pattern where the leader drops a reconnecting follower with a lower server ID. “Got zxid 0x… expected 0x…” with ClosedChannelException indicates the unrecoverable zxid-ahead state. “currentEpoch not found!” indicates the epoch file is missing from the data directory.

Metrics and signals to monitor

zk_followers, zk_synced_followers, and zk_pending_syncs are leader-only metrics. A monitoring setup that scrapes only followers will never see them. Query the leader, or query all nodes and filter for the one reporting leader state.

Fixes

Election port (3888) blocked from this node

Verify firewall, security group, and iptables rules allow 3888 bidirectionally between this node and every peer. Test in both directions, not just outbound from the affected node. Do not restart ZooKeeper until reachability is verified; the restart will not help and will lose diagnostic state in the logs. Once connectivity is fixed, the next election round (initiated by the node itself or by a planned leader restart) should let it rejoin.

DNS resolving to loopback in containers

In Kubernetes, Docker, or any environment where DNS may resolve a pod’s own hostname to 127.0.0.1, set the local server entry to bind all interfaces:

server.1=0.0.0.0:2888:3888
server.2=zk2.example.com:2888:3888
server.3=zk3.example.com:2888:3888

Alternatively, use static IPs in every server.X= line. The 0.0.0.0 form binds to all interfaces on that node only; peers continue to use their routable addresses. Verify the bind with ss -lntp | grep 3888 after restart.

zxid ahead of leader (unrecoverable)

This state is unrecoverable without intervention. The node’s local transaction log has a zxid incompatible with the leader’s epoch, and ZAB will not allow it to rejoin.

Destructive operation. This procedure discards the local copy of the data tree. It is safe only because the node is already not serving traffic.

1. Stop ZooKeeper on the affected node only.
2. Back up the data directory (the contents of dataDir, typically version-2/).
3. Remove the contents of the data directory on the affected node, including currentEpoch, acceptedEpoch, and all log.* and snapshot.* files.
4. Preserve the myid file (dataDir/myid). It identifies this server’s position in the ensemble and must not be deleted.
5. Restart ZooKeeper. The node performs a SNAP sync from the leader, receiving a full snapshot.
6. Verify zk_server_state returns follower and zk_zxid matches the leader.

Long catch-up after extended downtime

If zk_zxid on the affected node is advancing, just slowly, this is a SNAP sync in progress. Do not interrupt it. Watch zk_follower_sync_time on the leader; once it returns to baseline, the node should report follower. If the leader is overloaded by the snapshot transfer and client traffic is affected, reduce client load on the leader temporarily.

Local fsync or GC stalls

See the playbook’s Disk Sync Deadlock and GC Death Spiral composite patterns. The short version: move the transaction log to its own dedicated disk (dataLogDir on a separate volume, not shared with snapshots or other workloads), enable GC logging (-Xlog:gc*:file=/var/log/zookeeper/gc.log:time,uptime,level,tags:filecount=5,filesize=100m), and size the heap so that the post-GC trough stays below roughly 50% of max. A node that cannot persist a vote within tickTime cannot complete an election round.

ZOOKEEPER-2938 (“Have smaller server identifier”)

This is a known bug pattern where the leader drops the reconnecting follower’s connection because the follower has a lower server ID. ZOOKEEPER-2938 remains open without an assigned fix version as of the last JIRA update, including on the 3.9.x line. The temporary workaround is to restart the leader so a fresh election round runs cleanly. The longer-term mitigation is to upgrade to the newest practical release and watch the JIRA for a fix.

Prevention

• Alert on zk_server_state per node, not per ensemble. Aggregating state across nodes hides single-node LOOKING. Each node must be alertable independently.
• Alert on zk_looking_count rate per node. Any sustained increment outside a planned maintenance window is a ticket. The threshold is rate, not absolute value.
• Verify symmetric firewall rules for 2888 and 3888 between every pair of ensemble members, in both directions. Add this to provisioning checks. Cloud security group changes are a recurring cause of asymmetric blocking.
• In containers, use static IPs or 0.0.0.0 binding for the local server.X= entry. Do not rely on DNS resolving pod hostnames to routable addresses.
• Put the transaction log on a dedicated disk. Set dataLogDir to a separate volume from dataDir. This is the single highest-impact configuration change for write-path stability and the most common cause of fsync-induced LOOKING flapping.
• Compare zk_zxid across the ensemble as a scheduled check. Divergence is the earliest signal of a node about to fall out.
• Suppress non-critical alerts when zk_uptime < 300 seconds. Cold-start LOOKING is normal during snapshot and log replay and should not page.

How Netdata helps

• Per-second zk_server_state per node surfaces single-node LOOKING quickly, before it becomes a long-running incident.
• Per-node correlation of zk_looking_count increments with zk_fsynctime p99 and zk_jvm_pause_time_ms p99 distinguishes network causes from local disk and JVM causes without manual log scraping.
• Side-by-side zk_zxid across ensemble members shows divergence early, including the slow drift of a node in long catch-up versus the abrupt ahead-of-leader signature.
• Leader-only metrics (zk_followers, zk_synced_followers, zk_pending_syncs) are collected automatically and shown in the leader context, so a missing follower is visible in one view.
• ML anomaly detection on zk_looking_count and zk_fsynctime p99 catches the slow trend that precedes a stuck election, even when absolute values still look normal.

Related guides

• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing