ZooKeeper "Too many connections from /IP - max is 60": maxClientCnxns rejecting clients

WARN ... Error accepting new connection: Too many connections from /1.2.3.4 - max is 60 is ZooKeeper’s maxClientCnxns limiter refusing a new TCP connection from a specific source IP. By the time it appears in the server log, the client has already been denied.

The first trap: maxClientCnxns is enforced per source IP, not as a total. A single ZooKeeper node can hold thousands of healthy sessions while still refusing every new connection from one IP. zk_num_alive_connections, the metric most teams watch, is a total. It can look completely normal while clients behind a shared host IP are being silently turned away.

The second trap: silence. The server emits a WARN line, but the rejection produces no ZooKeeper error code the client can interpret. The client sees a TCP refused or a connect timeout and typically retries the next ensemble member, masking the problem until enough members reject the same source IP. The only reliable server-side signal is the zk_connection_rejected counter, which most teams do not collect.

What this means

maxClientCnxns defaults to 60. The limit counts concurrent socket-level connections from a single source IP to a single ensemble member. Existing sessions from that IP keep working. New connect attempts beyond the 60th are refused at the accept thread before any ZooKeeper protocol exchange happens.

Two things to internalize:

• Per source IP, not per client and not total. A physical app server running twenty JVMs, each with three ZK sessions, hits 60 from one box. A Kubernetes node running sixty pods behind NAT looks like one IP to ZooKeeper. None of those clients know about each other.
• Separate from maxCnxns. maxCnxns (the zookeeper.maxCnxns Java system property) caps total concurrent connections to a server and defaults to 0, meaning no limit. Operators frequently conflate the two and tune the wrong one.

There is also a separate server-side connection throttler (connectionMaxTokens, default 0 meaning disabled) introduced in ZooKeeper 3.6.0. That throttler is a token bucket applied to all incoming connections, not the per-IP limiter. If you have it enabled, see the ZOOKEEPER-4933 note below; the two mechanisms are easily confused.

Containerization makes this failure common. In Kubernetes, unless client pods use hostNetwork: true or reach ZooKeeper through a Service with externalTrafficPolicy: Local, all pods on a node share the node’s outbound IP. Sixty pods on one node, each opening one ZK session, exhaust the per-IP limit against every ensemble member even though the cluster’s total connection count is trivial.

flowchart LR
  P1[pod 1] --> NAT
  P2[pod 2] --> NAT
  P3[pod N] --> NAT
  NAT["node NAT\none host IP"] --> ZK["ZooKeeper\nmaxClientCnxns=60"]
  ZK -->|"sessions 1-60"| OK[accepted]
  ZK -->|"session 61+"| REJ["rejected\nzk_connection_rejected"]

Common causes

Quick checks

# Confirm the rejection counter is incrementing alongside total connections
echo mntr | nc localhost 2181 | grep -E 'zk_(connection_rejected|num_alive_connections)'

# Per-connection view grouped by source IP. Expensive on busy servers; sample sparingly.
echo cons | nc localhost 2181 | awk '{print $1}' | cut -d: -f1 | tr -d '/' | sort | uniq -c | sort -rn

# Rejects only happen on a writable server, so this should return 'rw'
echo isro | nc localhost 2181

# Server mode to confirm you are looking at a participating ensemble member
echo srvr | nc localhost 2181 | grep -E 'Mode|Zxid'

# On 3.5.3+ cons and mntr return nothing if not whitelisted. Check the whitelist.
grep -r 4lw.commands.whitelist /etc/zookeeper/ /opt/zookeeper/ 2>/dev/null

# Confirm the configured limit (no output means the default 60 is in effect)
grep -rE '^maxClientCnxns' /etc/zookeeper/ /opt/zookeeper/ 2>/dev/null

# OS-level view of established connections to ZK port, grouped by peer IP (IPv4)
ss -tan state established | awk '$4 ~ /:2181$/ {split($5,a,":"); print a[1]}' | sort | uniq -c | sort -rn | head

# Rule out FD exhaustion as the real cause
echo mntr | nc localhost 2181 | grep -E 'zk_.*file_descriptor'

How to diagnose it

1. Confirm the symptom is maxClientCnxns, not something else. A grep for the WARN line plus an incrementing zk_connection_rejected counter are the two confirming signals. If the counter is flat and the WARN lines are absent, the client’s connect failure has a different cause: firewall, DNS, or SASL auth failure. Check zk_auth_failed_count next.
2. Identify the offending source IP. The WARN line names it. Cross-reference with cons to see how many concurrent connections that IP currently holds. The gap between current count and 60 is how close to the limit you are.
3. Decide whether the IP is legitimate. If it is a Kubernetes node, a load balancer, or a NAT gateway, the per-IP limit is the wrong boundary and you need to raise it. If it is a single misbehaving client leaking connections, raise the limit only as a stopgap and fix the client.
4. Correlate with a scaling or deploy event. If rejections started at the same time as a rollout, the new replica count simply pushed the fleet past 60 sessions per host. This is a capacity miss, not a bug.
5. Verify the four-letter whitelist. On 3.5.3+, cons and mntr return nothing if not whitelisted. A monitoring system that suddenly reports all zeros may have lost whitelist access rather than watching a healthy cluster.

Metrics and signals to monitor

Fixes

Raise maxClientCnxns

The direct fix. Edit zoo.cfg:

maxClientCnxns=200

A value of 0 disables the per-IP limit entirely. Pick a value that fits your fleet, or disable it if you enforce connection limits elsewhere (client-side pooling, sidecar rate limiting). Apply with a rolling restart.

Tradeoffs: removing the limit shifts the burden to the OS FD limit and the JVM heap. If a single client leaks connections, removing the limit lets them exhaust FDs instead of getting a clean refusal. Prefer a high explicit value such as 200, 500, or 1000 depending on fleet size, over zero, unless you have other controls in place.

Push the limit onto the right boundary

If the offending IP is a NAT gateway or a Kubernetes node, the per-IP limit is structurally wrong for your topology. Options:

• Raise maxClientCnxns to pods_per_node * sessions_per_pod plus headroom.
• Use hostNetwork: true on client pods so each pod appears as its own source IP. This sidesteps NAT entirely but changes networking semantics and may not be acceptable in your environment.
• Use externalTrafficPolicy: Local on a client-facing Service so the source IP is preserved. Only relevant if ZooKeeper is reached through a Service, which is unusual.

Fix the leaking client

If cons shows one IP creeping toward 60 and never releasing, the client is not closing sessions cleanly. Common offenders include Kafka client connection churn (especially older versions), HBase RegionServer connection cycling, SolrCloud watchers without proper close, and application code that creates a new ZooKeeper client object per request.

Look for CLOSE_WAIT on the client host:

ss -tan state close-wait '( sport = :2181 or dport = :2181 )'

A growing count means the client is not closing its socket after ZooKeeper shuts down its side.

Check ZOOKEEPER-4933 if the throttler is enabled

If you have connectionMaxTokens set above 0 on versions before 3.8.5, 3.9.4, or 3.10.0, you may be hitting ZOOKEEPER-4933, where the token bucket overflows after a long idle period and the refill value goes negative, rejecting all connections regardless of source IP. The signature differs from maxClientCnxns: zk_connection_rejected increments from every source IP, not just one. Upgrade to 3.8.5+, 3.9.4+, or 3.10.0 to resolve.

Prevention

• Alert on zk_connection_rejected, not on zk_num_alive_connections. The total count cannot detect this failure mode. The rejected counter is the only signal that fires when the per-IP limit bites.
• Document the NAT topology in your ZooKeeper runbook. Make the per-IP semantics of maxClientCnxns explicit so the next operator does not assume it is a total.
• Capacity-plan against fleet size, not host count. For Kubernetes the relevant number is max(pods_per_node) * sessions_per_pod.
• Sample cons periodically (for example once per minute) and store the per-IP breakdown. It is too expensive for continuous collection but invaluable when this fires.
• Set maxClientCnxns explicitly in zoo.cfg, even if you want the default. An explicit value documents the decision; an absent value invites someone to assume it is unlimited.
• Whitelist cons and mntr in 4lw.commands.whitelist on 3.5.3+. Without them, both diagnosis and monitoring go dark.

How Netdata helps

• The ZooKeeper collector surfaces zk_connection_rejected and zk_num_alive_connections side by side, making the per-IP-versus-total distinction visible at a glance.
• Per-second collection catches short rejection bursts that minute-scraping misses, particularly during deploy-driven reconnect storms.
• ML anomaly detection flags the moment zk_connection_rejected starts incrementing, before the absolute count crosses any static threshold you might have picked.
• Correlating zk_connection_rejected with zk_connection_request_count and zk_connection_drop_count in a single view distinguishes “limit hit” from “sessions dying” without manual cross-referencing.
• Leader-aware scraping ensures connection metrics come from the node that matters, not a random follower.
• zk_open_file_descriptor_count plotted next to the connection metrics rules out FD exhaustion as the cause from the same dashboard.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper read latency high: memory reads that should never be slow