How ZooKeeper actually works in production: a mental model for operators

ZooKeeper is a distributed coordination service built on a replicated state machine. It maintains a hierarchical namespace of data nodes (znodes) entirely in memory and replicates every mutation across an ensemble of servers using the ZAB (ZooKeeper Atomic Broadcast) protocol.

This is the mental model the rest of the ZooKeeper runbooks assume. It is the set of abstractions an on-call engineer needs to reason about why writes stall, sessions expire, a “healthy” node can return stale reads, and why the leader matters more than any other node in the cluster.

Two facts anchor the whole model. First, the entire znode tree lives in the JVM heap of every server. Second, every write is fsync’d to the transaction log before it is acknowledged. Reads touch local memory. Writes touch local memory plus quorum plus disk. That asymmetry explains almost every ZooKeeper production incident.

What it is and why it matters

From the outside, ZooKeeper is a small hierarchical key-value store with strong consistency guarantees on writes. From the inside it is a replicated state machine: every server applies the same ordered sequence of transactions and converges on the same tree state. The ordering primitive is the zxid, a monotonically increasing 64-bit transaction ID assigned by the leader.

This determines the cost of every operation:

• A read is a local memory lookup. No quorum, no disk, no network round-trip beyond the client connection. Reads are cheap, fast, and scale horizontally by adding followers or observers.
• A write is a quorum operation. It must be assigned a zxid by the leader, persisted to the transaction log on disk with an fsync, broadcast to followers, acknowledged by a quorum, then committed. Writes are serialized through the leader and gated by disk I/O.

When something breaks, the symptom points back to one of those two paths: slow writes to the transaction log disk, stale reads to a follower lagging behind the leader, and mass session expirations to a process freeze (GC pause) that interrupted heartbeat traffic.

How it works

The request pipeline is the same on every server, but the work it does depends on role.

flowchart LR
  Client[Client request] --> Split{Read or write}
  Split -->|read| Tree[Local znode tree]
  Tree --> ReadResp[Respond from memory]
  Split -->|write| Leader[Leader]
  Leader --> Zxid[Assign zxid, fsync txnlog]
  Zxid --> Quorum[Quorum of followers ACK]
  Quorum --> Commit[Commit and apply]
  Commit --> WriteResp[Respond to client]

The in-memory znode tree

Every server holds the full data tree in JVM heap: every znode, its data payload, its ACL reference, its children list, and its stat structure. Session state, watch tables, queued requests, and serialization buffers consume heap on top of the tree itself.

This is why zk_znode_count and zk_approximate_data_size are leading indicators of heap exhaustion. A tree that grows slowly over months looks fine until heap pressure crosses roughly 85-90 percent, at which point garbage collection frequency spikes, stop-the-world pauses grow from milliseconds to seconds, and sessions begin expiring during those pauses. All servers carry the same tree, so they all OOM at roughly the same time.

The write path

Writes are processed exclusively by the leader. Followers that receive a write from a client forward it. For every mutation, the leader does the following:

1. Assigns a monotonically increasing zxid.
2. Appends the transaction to its write-ahead transaction log (txnlog) and fsyncs it to disk.
3. Broadcasts a PROPOSE message containing the transaction to all followers via ZAB.
4. Waits for a quorum of followers to ACK. Each follower ACKs only after it has also fsynced the transaction to its own txnlog.
5. Commits the transaction once quorum is reached, applies it to its in-memory tree, and sends a COMMIT to followers.
6. Followers apply the commit to their own in-memory trees.

The leader’s fsync is on the critical path of every write. Each follower’s fsync is on the critical path of every ACK. If either is slow, the whole write pipeline stalls. Dedicated low-latency storage for the txnlog is the single most important deployment decision for ZooKeeper. Shared or slow storage is the most common cause of write stalls in production.

The read path

Reads are served from the local in-memory copy. They never touch disk at runtime, never contact the leader, and never require quorum agreement. This is what makes ZooKeeper fast for read-heavy workloads, and it is also what makes reads potentially stale.

A follower that has fallen slightly behind the leader will happily serve reads from its older in-memory state. ZooKeeper offers sequential consistency for reads, not strict linearizability. Clients that need read-after-write consistency must issue a sync() call before reading, which forces the follower to catch up to the leader’s latest committed state.

Sessions, ephemeral nodes, and watches

A session is the unit of client identity. Ephemeral nodes, watches, and ACL enforcement all bind to sessions. Sessions have a negotiated timeout enforced by the leader’s session tracker. If a client fails to send a heartbeat within the timeout window, the session expires.

Session expiry is one of the most impactful events in ZooKeeper because it cascades:

• Every ephemeral node created by the session is deleted.
• Every watch registered by the session is cleared.
• Every system that watches those ephemeral nodes (Kafka broker registration, HBase RegionServer registration, distributed locks) receives a notification.

A mass session expiry event is ZooKeeper’s thundering-herd failure mode: ephemerals vanish, watches fire, clients reconnect simultaneously, and the leader (new or old) inherits the load spike.

Watches are one-shot triggers in classic ZooKeeper. When a watched znode changes, the server queues a notification to every watcher and then the watch is removed. A heavily-watched znode (a service discovery path with thousands of consumers) produces a fan-out notification storm when it changes. Persistent and recursive watches introduced in later versions change the volume characteristics, but not the underlying fan-out risk.

Persistent and recursive watches were introduced in ZooKeeper 3.6.0 (ZOOKEEPER-1416).

Persistence and recovery

Every mutation is appended to the write-ahead txnlog and fsynced before acknowledgment. This is the durability primitive. Periodically the server serializes the entire in-memory data tree to a snapshot file. Snapshots are fuzzy: they are taken without pausing the request pipeline and may include concurrent updates. They are consistent when replayed against the txnlog from the snapshot’s starting zxid forward.

On startup, a server loads the latest snapshot and replays the txnlog forward. For a large data tree this can take minutes, during which the node appears unresponsive. Recovery time scales linearly with snapshot size plus the volume of txnlog to replay.

Key defaults: forceSync defaults to true; setting it to false skips the fsync and is documented as unsafe. fsync.warningthresholdms defaults to 1000ms and logs a warning when an fsync exceeds it. snapCount defaults to 100,000 transactions between snapshots, with randomization to avoid synchronized snapshot storms across the ensemble. autopurge.purgeInterval defaults to 0, which disables autopurge; many deployments forget to enable it and slowly accumulate txnlog and snapshot files until disk fills. jute.maxbuffer defaults to roughly 1MB and caps the size of any single znode payload.

Where it shows up in production

The mental model maps directly onto recurring operational patterns:

• The leader is a serialization point. All writes go through it. Its disk, its CPU, and its network are the bottleneck for write throughput. A leader under write load can saturate while followers sit idle.
• Heap is the entire state. Slow znode growth is a silent killer. Track zk_znode_count and zk_approximate_data_size against heap allocation, not just current heap utilization.
• Disk fsync is the write latency floor. A write cannot complete until both the leader and a quorum of followers have fsynced. Anything that interferes with that fsync (shared storage, cloud burst credit exhaustion, colocated workloads, snapshot I/O on the same disk) directly elevates write latency.
• GC pauses freeze everything. A stop-the-world GC pause stops heartbeats, request processing, and quorum ACKs simultaneously. On the leader, a long enough pause triggers re-election. On any node, a long enough pause expires client sessions.
• Reads from followers can be stale. Applications that need strict read-after-write consistency must use sync(), or accept sequential consistency.

The default per-IP connection limit is another recurring trap. maxClientCnxns defaults to 60 per source IP, not per server. In containerized environments where many pods share a host IP behind NAT, new client connections are silently rejected.

zk_connection_rejected is a real counter (metric connection_rejected) surfaced by mntr on ZooKeeper 3.6.0+ when the default metrics provider is used; it increments when maxClientCnxns rejects a connection. It does not exist on 3.5.x.

Tradeoffs and when it matters

ZooKeeper is a coordination service, not a database. Storing large data in znodes is an anti-pattern that bloats the tree, slows snapshots, and lengthens recovery. Use it for what it is designed for: small configuration, leader election, distributed locks, service discovery registrations, and small state machines that benefit from linearizable writes.

For deployment topology, an ensemble of three, five, or seven voting members is standard. Quorum is floor(N/2)+1. A three-node ensemble tolerates one failure; a five-node tolerates two. Observers are non-voting members that receive the commit stream for read scaling without participating in quorum, useful when read load is geographically distributed but write latency must stay low.

Signals to watch in production

Metric availability is version-dependent with the default metrics provider: on 3.5.x mntr exposes only the aggregate zk_avg_latency, zk_min_latency, zk_max_latency (plus the legacy counters); the per-path summaries zk_fsynctime, zk_readlatency, zk_updatelatency and the counters zk_digest_mismatches_count, zk_connection_rejected, zk_unrecoverable_error_count, zk_stale_sessions_expired require ZooKeeper 3.6.0+, and zk_jvm_pause_time_ms requires 3.6.2+. With the Prometheus metrics provider (3.6+), all of these are available over the /metrics endpoint instead.

zk_followers, zk_synced_followers, and zk_pending_syncs are leader-only metrics. Monitoring must identify and query the current leader, or query all nodes and filter for the leader-reported values. A scrape that only hits followers will never see replication health, which is one of the most common monitoring gaps in ZooKeeper deployments.

How Netdata helps

zk_throttled_ops (counter throttled_ops, 3.7.0+) and zk_unrecoverable_error_count (counter unrecoverable_error_count, 3.6.0+) are the exposed names for these signals.

Per-second collection and anomaly detection suit ZooKeeper’s failure modes: slow drift (znode growth, fsync latency creep) that becomes a cliff-edge incident (heap exhaustion, write stall, session cascade).

• Correlate zk_fsynctime with zk_updatelatency on the same chart to confirm disk as the root cause of write stalls.
• Overlay zk_jvm_pause_time_ms against zk_stale_sessions_expired and zk_looking_count to identify GC as the trigger for session cascades and elections.
• Track zk_znode_count and zk_approximate_data_size as long-running trends to surface silent heap exhaustion months before OOM.
• Watch zk_outstanding_requests and zk_throttled_ops together to catch pipeline saturation before clients see timeouts.
• Surface zk_digest_mismatches_count and zk_unrecoverable_error_count as immediate-page signals for data integrity violations.

Related guides

• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper unexpected leader election: finding why the leader dropped
• ZooKeeper split-brain: two nodes both reporting leader
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper write latency high: read zk_updatelatency, not just avg_latency
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals