ZooKeeper "Client session timed out, have not heard from server": the heartbeat miss

A Client session timed out, have not heard from server in <ms>ms for session id 0x..., closing socket connection and attempting reconnect log line is the client’s SendThread reporting that no PING response arrived inside its heartbeat window. This is a client-side symptom of a heartbeat miss. It is not the server expiring the session, and it is not yet SessionExpired. The client closes its socket and tries another ensemble member.

The outcome depends on whether reconnect succeeds before the negotiated session timeout elapses on the server. If the client re-establishes the session inside the window, the session continues and ephemeral nodes survive. If the timeout elapses first, the leader’s session tracker expires the session, every ephemeral node owned by that session is deleted, every watch on those nodes fires, and the client receives SessionExpired once it reconnects. The same root cause (a JVM stop-the-world pause, a network drop, a leader election) produces either outcome depending on duration.

Operators default to blaming the network. In production ensembles the more common root cause is a JVM pause on either side of the connection. Server-side JVM pause metrics (zk_jvm_pause_time_ms with percentile variants like zk_p99_jvm_pause_time_ms, available via mntr in ZooKeeper 3.7+ when the JvmPauseMonitor is enabled with jvm.pause.monitor=true) and the client JVM’s own GC logs are usually more diagnostic than packet captures. This article walks the heartbeat mechanism, the causes that account for most of these log lines, and the order in which to check signals.

What this means

ZooKeeper sessions are heartbeated by the client. SendThread sends a PING after roughly one third of the negotiated session timeout has elapsed, and expects a response before another third passes. If no response arrives inside that window, the client logs the timeout message and enters reconnect.

The session timeout is negotiated at connect time. The client proposes a value, and the server clamps it to the range enforced by tickTime:

• minSessionTimeout = 2 x tickTime
• maxSessionTimeout = 20 x tickTime

With the default tickTime of 2000ms, the server enforces a 4s to 40s range. A client requesting 60s silently receives 40s; a client requesting 2s silently receives 4s. Both ends of the clamp produce surprising behaviour. The long-timeout case fails earlier than the application expects, and the short-timeout case is more sensitive to GC pauses than the operator expects.

The progression that matters operationally:

1. Heartbeat miss. Client logs the message in the title. Socket closes. Client enters CONNECTING and tries another host.
2. ConnectionLoss. While disconnected, any in-flight operation throws ConnectionLoss. Most client libraries queue writes and reissue them after reconnect.
3. Session recovery or SessionExpired. If the client reconnects to any ensemble member and the server still has the session in its tracker, the session resumes with the same id. If the negotiated timeout elapses first, the leader’s session tracker expires the session, ephemerals vanish, and the client must create a new session.

The log line in the title sits between step 1 and step 3. It is the early signal and is recoverable. The expensive failure is step 3; the goal of diagnosis is to prevent repeated misses from accumulating into expirations.

Common causes

The first three rows are the common ones. The last two are configuration problems that surface through the same log line.

Quick checks

These commands are read-only and safe to run during an incident.

# Check server is alive and read-write (not read-only after quorum loss)
echo isro | nc localhost 2181

# Confirm which role this node currently holds
echo mntr | nc localhost 2181 | grep zk_server_state

# JVM pause metrics on the server (3.7+, requires jvm.pause.monitor=true). Compare p99 to minSessionTimeout.
echo mntr | nc localhost 2181 | grep -i jvm

# Sessions expired for missed heartbeats
echo mntr | nc localhost 2181 | grep -iE 'stale|expired'

# Connections dropped by the server
echo mntr | nc localhost 2181 | grep -i drop

# Outstanding request queue (should be 0 in steady state)
echo mntr | nc localhost 2181 | grep zk_outstanding_requests

# RTT and loss between the affected client and each ensemble member
for h in zk1 zk2 zk3; do ping -c 20 "$h"; done

# Client JVM GC stats (run on the affected client host)
jstat -gcutil <client_pid> 1000 10

# Server JVM GC stats
jstat -gcutil $(pgrep -f QuorumPeerMain) 1000 10

If mntr returns nothing, check the four-letter word whitelist. Since ZooKeeper 3.5.3, only srvr is allowed by default. Add mntr, stat, ruok, isro, and cons explicitly via 4lw.commands.whitelist in zoo.cfg or the zookeeper.4lw.commands.whitelist system property. The AdminServer on port 8080 (3.5+) exposes the same data over HTTP and is the recommended path forward as four-letter words are being deprecated.

How to diagnose it

Work through scope before tuning anything. The shape of the blast radius is the fastest signal.

flowchart TD
    A["Client logs session timed out"] --> B{"How many clients affected?"}
    B -- "Many, across networks" --> C["Server-side GC pause"]
    B -- "Many, one network" --> D["Network partition or RTT"]
    B -- "Only this JVM" --> E["Client-side GC pause"]
    B -- "Brief, during election" --> F["Leader election window"]

1. Establish scope. Triage across the fleet of clients. A single client logging timeouts points at that client’s JVM or its network path. A burst across many clients points at the server or the shared network.
2. Check the server first. Even if the symptom appears client-local, confirm the server is clean before chasing the client. A server JVM pause freezes PING handling and produces timeouts on every connected client. Pull JVM pause percentiles, then read the GC log directly. A Full GC entry whose timestamp matches the client log to the second is the answer.
3. Check for elections. During leader election the ensemble does not process writes and may not respond to heartbeats reliably. zk_server_state reports LOOKING during the election window, and zk_looking_count is a monotonic counter that tracks each election event. Brief elections during rolling restarts are normal. Recurring elections outside maintenance indicate the leader is repeatedly becoming unresponsive, usually from GC or disk pressure.
4. Check the client JVM. If the server is clean and only one client (or one JVM) is affected, the client’s SendThread may itself be stuck in a stop-the-world pause. A client-side pause longer than the session timeout is particularly dangerous: the server expires the session because no PING arrives, but the client believes it is still connected and may continue to act on ephemeral state it no longer owns.
5. Check the network last. If both JVMs are clean, verify the path. Look for asymmetric reachability, since the quorum port and the election port are separate TCP ports; a firewall that blocks only one will pass isro checks but break elections on the next failure. Also check NAT or conntrack table exhaustion on hosts that originate many connections, and recent security group changes that often precede cloud incidents.

The session id in the log line is the key correlation handle. Note the 0x... value, then look for matching entries in the server’s connection listings (cons or stat four-letter output) when the client reconnects.

Metrics and signals to monitor

Two composite patterns are particularly relevant. The GC pause cascade shows rhythmic latency spikes in zk_avg_latency and zk_outstanding_requests that build during the pause and drain after, with connection drops incrementing once the pause ends. The session expiration storm shows a V-shaped zk_num_alive_connections curve (drop, timeout pause, reconnect spike), accompanied by an zk_ephemerals_count drop and a zk_packets_sent burst from watch fan-out.

Fixes

Server-side JVM pause

This is the most common root cause and the easiest to confirm.

1. Confirm via the server GC log; JVM pause metrics corroborate but the GC log is authoritative.
2. Check the data tree size. zk_znode_count and zk_approximate_data_size together drive heap pressure. Keep total in-memory tree size below roughly 50% of heap.
3. Size the heap to the data tree with headroom. The rising-trough pattern in heap usage (each GC reclaims less than the last) means live data is growing.
4. Choose a collector suited to the heap size. G1GC has been the default JVM collector since JDK 9; ZooKeeper’s own launch scripts do not set a collector. For JDK 15 or later, ZGC dramatically reduces pause times and is worth the migration if pauses are a recurring incident driver.
5. Disable Transparent Huge Pages on the ZooKeeper host if you see pauses longer than the GC log suggests. THP can multiply pause duration.

See ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls for the related disk-side write stall pattern, which compounds with GC pressure.

Client-side JVM pause

If only one client is logging timeouts and the server is clean, the client’s SendThread is itself pausing.

1. Enable GC logging on the client JVM and look for stop-the-world events that overlap the client log timestamps.
2. Tune the client JVM so pause durations stay well below the negotiated session timeout. A client GC pause that exceeds the session timeout produces a window in which the server has already expired the session (and may have released locks to other clients) while the original client still believes it owns them.
3. Review the requested session timeout against what the application actually needs. A longer timeout tolerates longer pauses but delays failover in legitimate network partitions.

Network loss or RTT spike

When the JVMs are clean and the symptom clusters by network segment:

1. Run mtr to each ensemble member from the affected client population.
2. Verify both the quorum port and the election port are reachable between every pair of members. The election port only matters during a leader election, so a partial firewall passes every check until the next failure, then produces a catastrophic election.
3. Check conntrack and NAT table sizes on hosts that originate many client connections. NAT exhaustion produces silent drops that look like packet loss at the application layer.
4. Review recent security group or firewall changes. These are a frequent precursor in cloud environments.

Session timeout too low

If timeouts recur under load that the JVMs and network can handle, the negotiated timeout may be sitting at or near minSessionTimeout.

1. Confirm tickTime in zoo.cfg and compute minSessionTimeout = 2 x tickTime.
2. Compare the application’s requested timeout against the clamped value the server actually enforces.
3. Adjust tickTime (which raises both bounds) or document the clamp explicitly so application teams understand what they are actually getting.

Note that raising tickTime also widens the leader’s heartbeat interval to followers and changes the syncLimit x tickTime window. Review the related guides below before changing it on a production ensemble.

Prevention

• Treat server JVM pause time as a first-class signal. Most teams collect zk_avg_latency and skip pause time. Pauses are the dominant cause of session expirations and unnecessary elections.
• Alert on session expiration rate (zk_stale_sessions_expired), not the absolute counter. The counter is monotonic, so alert on the delta.
• Tune both JVMs, not just the server. Client-side GC is an underrated cause and is invisible to server-side monitoring.
• Test failover regularly. Controlled server kills are the only way to confirm the client population reconnects cleanly and that monitoring catches the resulting elections and session churn.
• Match session timeouts to the deployment environment. Kafka’s default zookeeper.session.timeout.ms moved from 6s to 18s in Kafka 2.5.0 (KAFKA-9102) because cloud environments produce more spurious timeouts. Apply the same thinking to your own clients.

How Netdata helps

• Per-second collection of ZooKeeper metrics makes the wall-clock overlap between a server GC event and a client-side heartbeat miss immediately visible on a single chart.
• ML anomaly detection on zk_num_alive_connections, zk_ephemerals_count, and zk_packets_sent catches the V-shaped signature of a session expiration storm before it cascades into dependent services.
• Composite dashboards let you place the leader’s zk_outstanding_requests next to client-side connection state, so a heartbeat miss can be triaged as server, client, or network in seconds rather than minutes.
• Anomaly advisors flag the rising-trough pattern in heap usage and rhythmic latency spikes that precede a GC pause cascade, giving lead time before sessions start expiring.

Related guides

• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum
• ZooKeeper read latency high: memory reads that should never be slow
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port