ZooKeeper "Cannot open channel to N at election address": the blocked election port

The log line is Cannot open channel to <id> at election address /host:3888. It is emitted by QuorumCnxManager.initiateConnection() when Socket.connect() to a peer’s leader election port fails with ConnectException (refused) or SocketTimeoutException (timed out). The error is harmless during steady state and fatal during an election.

ZooKeeper ensembles use two inter-server TCP ports. Port 2888 (the quorum port) carries the ZAB proposal/ACK/commit stream between followers and the active leader. Port 3888 (the leader election port) is touched only when FastLeaderElection needs pairwise TCP channels to every voting peer. If 2888 is reachable but 3888 is not, the ensemble runs fine until the leader is lost, at which point no new leader can be elected.

The classic trap is a firewall or security-group change that opens or closes only one of the two ports. Replication keeps working, dashboards stay green, and the defect stays invisible until the next election, when writes fail for every dependent service. Treat this error as a connectivity defect first and a ZooKeeper defect last.

What this means

This server tried to open a TCP connection to another ensemble member’s election port and the connect syscall failed. The peer host and port come from zoo.cfg:

server.<id>=<host>:<quorum_port>:<election_port>
# e.g. server.2=zk-2:2888:3888

The connection is initiated by FastLeaderElection, the only algorithm available since ZooKeeper 3.6.0 (UDP-based electionAlg 1 and 2 were deprecated in 3.4.0 and removed in 3.6.0). Every voting pair must be able to open a TCP socket on the election port in both directions, because the protocol uses bidirectional connections with a deterministic tie-break on server id to choose which side acts as client and which as server.

The default cnxTimeout (zookeeper.cnxTimeout) for opening an election connection is 5 seconds. During a stuck election expect roughly one Cannot open channel line per peer per 5 second window, with retries until initLimit * tickTime elapses.

Because the election path is only exercised during leader transition, the failure can hide for weeks. The two practical shapes are:

• Symmetric block. Both sides cannot reach the other’s 3888. The cluster cannot elect from cold start. Seen on first deploy or after a full restart.
• Asymmetric block. One direction is open, the other is not. The cluster may elect a leader at boot, then fail to re-elect after that leader dies. This is the insidious case: the first election completes using the working direction, and the broken direction only matters when a specific node needs to initiate a connection to a specific peer.

flowchart TD
  A[Leader election triggered] --> B{Can this node open TCP to peer 3888?}
  B -- yes --> C[Vote exchanged, election proceeds]
  B -- no, timeout --> D["Cannot open channel to N at election address"]
  D --> E[Retry up to initLimit x tickTime]
  E --> F{Quorum reached via other peers?}
  F -- yes --> G[Leader elected, ensemble recovers]
  F -- no --> H[No leader, writes failing]
  H --> I[ensemble stuck in LOOKING]

Common causes

Quick checks

Run these read-only. None change ZooKeeper state.

# Confirm whether this node can reach each peer's election port.
# Run from each ensemble member, against every other member.
nc -vz <peer-host> 3888
nc -vz <peer-host> 2888

# Show what is actually listening. Both ports should be bound.
ss -ltnp | grep -E ':2888|:3888'

# Confirm the ensemble config is identical on every member
grep '^server\.' /etc/zookeeper/conf/zoo.cfg

# Confirm myid matches the local server line in zoo.cfg
cat /var/lib/zookeeper/myid

# Check role and whether we are stuck in LOOKING
echo srvr | nc localhost 2181 | grep -E 'Mode|Zxid'
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_quorum_size'

# isro distinguishes "rw" (functioning) from "ro" (quorum lost, stale reads only)
echo isro | nc localhost 2181

# Look for the error itself and its companion election noise
journalctl -u zookeeper -n 500 | grep -E 'Cannot open channel|LOOKING|FOLLOWING|LEADING'
# or, if logging to file:
grep -E 'Cannot open channel|LOOKING|FOLLOWING|LEADING' /var/log/zookeeper/zookeeper.log | tail -50

# Confirm DNS resolution is not loopback for any peer
for h in zk-1 zk-2 zk-3; do printf '%s ' "$h"; getent hosts "$h"; done

If 4lw.commands.whitelist is set (3.5.3+) and a command returns empty, whitelist srvr, mntr, isro, stat at minimum. The silent-empty-response behavior of the whitelist is itself a monitoring trap.

How to diagnose it

1. Identify whether this is an active outage or a latent problem. Run echo srvr | nc localhost 2181 | grep Mode on every member. If all report leader or follower and writes work (isro returns rw), you are looking at a latent connectivity defect. If any reports standalone or is cycling in LOOKING, you are in an active election stall.

2. Map the full reachability matrix. ZooKeeper needs bidirectional connectivity on both 2888 and 3888 between every pair of voting members. Do not check only leader to follower. From each member, test both ports to each peer:

   for peer in zk-1 zk-2 zk-3; do
     for port in 2888 3888; do
       printf '%s:%s ' "$peer" "$port"
       nc -z -w 3 "$peer" "$port" && echo OK || echo FAIL
     done
   done
   

   Any FAIL cell is a defect, even if the cluster is currently serving traffic. Asymmetric results across the matrix are the smoking gun for the trap described in the opening.

3. Confirm the listener is actually bound. On the unreachable peer:

   ss -ltnp | grep -E ':2888|:3888'
   

   If 3888 is not listening at all, the peer failed to bind. On 3.5.6+, check whether zookeeper.electionPortBindRetry (default 3) is exhausting during transient DNS in containers and then giving up.

4. Resolve DNS carefully. getent hosts <peer-host> must return the intended interface IP, not 127.0.0.1 or a container loopback. This is the second most common cause after firewalls. In Kubernetes StatefulSets using a headless service, verify the FQDN in zoo.cfg resolves to the pod IP, not the service ClusterIP. For the local server entry, the canonical workaround is to bind 0.0.0.0:

   server.1=0.0.0.0:2888:3888
   server.2=zk-2:2888:3888
   server.3=zk-3:2888:3888
   

   On node 1, the local entry uses 0.0.0.0 so the listener binds all interfaces, while the same node is referenced by real hostname on the other members. Before 3.5.6 the election listener could stop listening while a peer hostname was temporarily unresolvable (ZOOKEEPER-3320); upgrade to 3.5.6 or newer.

5. Check for service mesh or CNI interference. Istio versions before 1.10 redirect inbound traffic to loopback via Envoy, which breaks ZooKeeper’s election listener binding. Operators such as Strimzi have hit related startup races. The fix is to upgrade the mesh, exclude 2888/3888 from redirection, or both.

6. Look at the cloud control plane. Security groups, network policies, and NACLs are usually the actual root cause. Verify the rules allow both ports, both directions, between every pair of members. “Allowed to the leader” is not sufficient. A common mistake is opening 2888 from clients to the leader and 3888 from the leader to followers but missing the reverse paths.

7. Check cnxTimeout and initLimit. If the matrix shows intermittent connectivity, election connects can race the 5 second cnxTimeout or the initLimit * tickTime election deadline. Tune only after fixing the underlying connectivity.

Metrics and signals to monitor

ruok is a poor health check for this failure mode. A server in LOOKING state still returns imok because the process is alive. Always combine ruok with isro and mntr.

Fixes

Fix the firewall or security group

Open both 2888 and 3888 between every pair of voting members, in both directions, for TCP. Verify with the reachability matrix in step 2 above. If you use Observers, they also need election port connectivity during their own startup. Do not scope rules to “the current leader.” The current leader will change, and the rules must already be correct when it does.

If you must operate with degraded connectivity for a short window, prefer taking the affected node out of the voting set via reconfiguration rather than running a partial mesh and waiting for the next election to bite.

Fix DNS to loopback

If getent hosts <peer-host> returns 127.0.0.1 or a container-local address for any peer, fix the record. The local server entry should bind 0.0.0.0, but peer entries must resolve to routable addresses. In Kubernetes, use the headless service FQDN (<pod-name>.<headless-svc>.<namespace>.svc.cluster.local), not the ClusterIP Service, so the address resolves to the specific pod.

Fix the 0.0.0.0 binding bug on 3.5.x

If you are on 3.5.5 or earlier and hitting the listener-stop problem, the supported fix is to upgrade to 3.5.6 or newer. CVE-2023-44981 (SASL quorum peer authorization bypass, fixed in 3.7.2 and 3.8.3) is an additional reason to be on a current 3.8.x or 3.9.x release.

Fix service mesh interference

For Istio, upgrade to 1.10 or later, which fixed the StatefulSet inbound redirection issue. If you cannot upgrade, exclude ZooKeeper ports from the mesh. ZooKeeper’s election protocol does not tolerate the extra hop and loopback translation. For other CNIs and sidecars the principle is the same: election traffic must reach the actual pod interface, not a proxy.

Fix transient DNS during container startup

On 3.5.6+, set zookeeper.electionPortBindRetry to 0 (infinite retry) or a higher number to ride out DNS convergence during pod startup. This is a startup resilience fix, not a substitute for steady-state connectivity. If you are hitting this on every restart, treat it as a DNS design issue.

Fix Docker Swarm VXLAN blocking

If ss shows 3888 LISTEN locally but cross-host nc times out, the overlay network’s data path may be blocked at the hypervisor. Changing the default VXLAN port via --data-path-port has been reported as a workaround.

Increase election resilience as a stopgap

zookeeper.tcpKeepAlive=true (3.5.4+) enables TCP keepalive on quorum election sockets, useful when NAT or firewall middleboxes drop idle election connections. Multi-address support (multiAddress.enabled=true, 3.6.0+) lets you specify multiple addresses per server using | as a separator, e.g. server.1=zoo1-net1:2888:3888|zoo1-net2:2889:3889, for ensembles reachable across multiple network interfaces. quorumCnxnTimeoutMs controls the read timeout for election connections.

Prevention

• Run a reachability matrix check as part of every firewall, security group, or network policy change. Automate it. The check should hit both 2888 and 3888 between every pair of voting members.
• Alert on zk_looking_count and zk_sum_leader_unavailable_time. Both are counters and both should be zero-delta outside maintenance. Any non-zero delta is a real incident, not noise.
• Test leader failover regularly. Deliberately kill the leader in a controlled maintenance window and confirm the ensemble re-elects within seconds. This is the only way to catch the asymmetric block before a real outage does. The election algorithm uses TCP on 3888; if it has not been exercised recently, you do not know it works.
• Pin DNS records to routable addresses and use the 0.0.0.0 local server binding. This removes the most common non-firewall cause.
• Treat every security-group change touching ZooKeeper as a change request requiring sign-off. The blast radius is the entire dependent stack.
• Keep ZooKeeper current. The 3.8.x and 3.9.x lines carry the relevant fixes for binding, retries, keepalive, and the recent CVEs.

How Netdata helps

• Per-second zk_server_state and zk_looking_count collection surfaces a stuck LOOKING state within seconds, before the first downstream service starts logging session expirations.
• Correlate election events with leader unavailable time. zk_sum_leader_unavailable_time delta aligned with zk_looking_count increments tells you both that an election happened and how long writes were impossible, which is the metric dependent teams actually need.
• Host-level TCP connection metrics on ports 2888 and 3888 expose connect failures, retransmits, and resets at the network layer, often before ZooKeeper logs anything.
• Anomaly detection on zk_quorum_size, zk_packets_received, and connection counters can flag gradual connectivity drift from an asymmetric change even when threshold-based alerts would miss it.
• Composite dashboards that put isro status, mntr counters, and host network metrics on one timeline shorten the diagnosis path from “ZooKeeper is slow” to “security group change at 14:03 broke 3888 between nodes 2 and 3.”

Related guides

• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing