ZooKeeper monitoring checklist: the signals every production ensemble needs

A reference checklist for engineers running production ZooKeeper ensembles. Signals are organized into four maturity levels: survival, operational, mature, and expert. Each level adds visibility for failure modes the previous level cannot see.

The levels are cumulative. Level 2 assumes Level 1 is covered. Skipping to Level 4 without Levels 1 through 3 leaves gaps in the signals that actually page you during incidents: disk stalls, GC cascades, and quorum loss.

Work through the levels in order. If your tooling cannot reliably collect a level’s signals, close that gap before adding more advanced checks.

Collection sources. Signals below come from the mntr four-letter word, the AdminServer HTTP endpoint (default port 8080, ZooKeeper 3.5+), or the Prometheus metrics endpoint (ZooKeeper 3.6+). Since ZooKeeper 3.5.3, four-letter words require whitelisting via 4lw.commands.whitelist in zoo.cfg. At minimum, whitelist mntr, ruok, isro, srvr, and stat. Disk-space and OS-level signals come from the host, not from ZooKeeper.

flowchart TD
    L4["Level 4 - Expert
quorum_ack_latency, digest_mismatches,
unrecoverable_errors, leader_unavailable_time"]
    L3["Level 3 - Mature
fsynctime, updatelatency, readlatency,
jvm_pause_time, watch_count, pending_syncs"]
    L2["Level 2 - Operational
avg_latency, outstanding_requests,
synced_followers, znode_count, connections"]
    L1["Level 1 - Survival
ruok, isro, server_state, uptime,
disk free on dataDir and dataLogDir"]
    L1 -->|add saturation and capacity| L2
    L2 -->|add write-path and GC signals| L3
    L3 -->|add integrity and consensus| L4

Level 1: survival - is ZooKeeper alive and serving?

Minimum to know the process is up, the client port is open, and the node is part of a functioning ensemble. With only Level 1 you catch complete crashes, quorum loss, and full disks. You do not catch latency degradation, GC pressure, replication lag, or capacity creep.

Signal	What it tells you	Warning sign
`ruok` 4lw, TCP 2181 reachable	Process is alive and client port is open	No response or connection refused: process down or JVM hung
`isro` 4lw	Whether the node serves writes (“rw”) or has lost quorum (“ro”)	“ro” sustained for more than 60 seconds in a multi-node ensemble means writes are impossible
`zk_server_state` (`mntr`)	Current role: leader, follower, observer, standalone, or looking	No leader in the ensemble, or more than one node reporting leader
`zk_uptime` (`mntr`)	Time since process start	Unexpected reset indicates crash or OOM kill
Disk free on `dataDir` and `dataLogDir` (host-level)	Whether ZooKeeper can write transaction logs and snapshots	`dataLogDir` full: all writes fail immediately

ruok is necessary but not sufficient. “imok” only confirms the process is alive and the client port is open. A node that lost quorum and fell into read-only mode still returns “imok”. Always pair ruok with isro, and confirm zk_server_state across the ensemble: exactly one node should report “leader”, the rest “follower” (plus any configured observers). If isro returns “ro”, writes are impossible and dependent services that need coordination (Kafka leader election, HBase region assignment) are already failing.

Suppress alerts during cold starts. After a restart, ZooKeeper loads the latest snapshot and replays the transaction log. For large data trees this can take minutes, during which ruok may fail and isro may return “ro”. Suppress non-critical alerts when zk_uptime is below 300 seconds.

Level 2: operational - what a competent team monitors

Signals to catch saturation and capacity problems before they become outages. This is the level most teams plateau at.

Signal	What it tells you	Warning sign
`zk_avg_latency` / `zk_max_latency`	Aggregated request latency since last `srst` or restart	Sustained avg more than 2x rolling baseline. Max is a high-water mark, not a rate
`zk_outstanding_requests`	Request pipeline backlog	Sustained non-zero means the server cannot keep up. Approaching `globalOutstandingLimit` (default 1000) means throttling is imminent
`zk_num_alive_connections`	Active client connections to this node	Sudden drop over 50 percent in one minute: mass disconnect or session expiration storm
`zk_connection_rejected`	Connections refused due to `maxClientCnxns` per source IP	Any non-zero rate: clients are being silently turned away
`zk_connection_drop_count`	Connections closed by server or broken	Sustained rate above a fraction of total connections per minute
`zk_open_file_descriptor_count` / `zk_max_file_descriptor_count`	FD usage versus OS limit	open/max ratio above 80 percent
`zk_znode_count`	Size of the in-memory data tree	Linear or unbounded growth: eventual heap exhaustion
`zk_learners` (3.6+; `zk_followers` on 3.5.x) / `zk_synced_followers` (leader-only)	Replication health from the leader’s view	`synced_followers` below `ensemble_size - 1`. At floor(N/2), one more failure breaks quorum
Disk free on `dataDir` and `dataLogDir` (host-level)	Capacity headroom for logs and snapshots	`dataLogDir` below 20 percent free. `dataDir` below 30 percent

Leader-only metrics are invisible on followers. zk_learners (3.6+; zk_followers on 3.5.x) and zk_synced_followers appear only on the node currently reporting zk_server_state = "leader". Monitoring that queries a single follower or load-balances across nodes will never see them. Query all nodes and filter for the leader, or target the leader explicitly. The same applies to zk_pending_syncs and zk_quorum_ack_latency at higher levels.

maxClientCnxns rejects silently. Default is 60 connections per source IP. In containerized environments where many pods share a host IP, this limit is easily exceeded. New connections are refused with no server-side log entry. The only signal is zk_connection_rejected incrementing. zk_num_alive_connections is a total across all IPs, so it cannot tell you which IP hit the limit.

Level 3: mature - deep operational visibility

Signals that separate write-path stalls from GC stalls, and replication lag from leader overload. This is where most preventable ZooKeeper outages get caught early.

Signal	What it tells you	Warning sign
`zk_updatelatency` / `zk_readlatency` (p99, ZK 3.6+)	Write and read latency separated	Update p99 above 100ms or growing monotonically. Read p99 above 50ms is abnormal for memory lookups
`zk_fsynctime` (max/avg, ZK 3.6+; fsync has no percentile variant in mntr)	Transaction log fsync latency, the write-path root cause	Above 10ms sustained. Dedicated SSD should be under 2ms at max
`zk_jvm_pause_time_ms` (p99, ZK 3.6+)	JVM stop-the-world pause duration	Approaching a third of `minSessionTimeout` (~1333ms with defaults): sessions start expiring
`zk_throttled_ops`	Operations throttled at `globalOutstandingLimit`	Any non-zero rate: server is saturated and applying backpressure
`zk_watch_count`	Active watch registrations	Unbounded growth: thundering-herd risk when a popular znode changes
`zk_approximate_data_size`	Bytes of znode data in memory	Growing faster than `znode_count`: clients storing too much data per node
`zk_looking_count`	Leader election events	More than one per hour outside maintenance: recurring instability
`zk_follower_sync_time`	Time for followers to catch up with the leader	Approaching `syncLimit x tickTime` (10 seconds with sample `tickTime=2000` ms / `syncLimit=5`): follower about to be ejected
`zk_pending_syncs` (leader-only)	Followers waiting to sync	Sustained non-zero: write rate exceeds follower sync capacity
`zk_stale_sessions_expired`	Sessions that expired from missed heartbeats	Any non-zero rate outside maintenance: session expiration storm

Separate reads from writes. zk_avg_latency aggregates reads (local memory lookups, fast) and writes (quorum round-trip, slow). A write stall can be hidden in the average when read volume dominates. ZooKeeper 3.6+ exposes zk_updatelatency and zk_readlatency with p50, p95, p99, and p999 percentiles. Track both, alert on percentiles, not cumulative averages. The same applies to zk_fsynctime and zk_jvm_pause_time_ms.

fsync latency is the leading write-path signal. Every write blocks until the transaction log is fsync’d to disk. When zk_max_fsynctime climbs, zk_updatelatency follows. On dedicated SSD, fsync max should be under 2ms. If it is not, check whether dataLogDir shares a disk with snapshots, whether cloud burst credits are exhausted, and whether a colocated workload is saturating the device.

GC pauses are the leading election trigger. zk_jvm_pause_time_ms tells you how long the JVM froze. If p99 approaches a meaningful fraction of minSessionTimeout (2 x tickTime = 4000ms with the sample tickTime=2000 ms), sessions start expiring. If it approaches syncLimit x tickTime (10 seconds with the sample tickTime=2000 ms and syncLimit=5), followers miss heartbeats and trigger elections. Correlate pause spikes with zk_looking_count and zk_stale_sessions_expired to confirm GC as the cause rather than network or disk.

Level 4: expert - full operational mastery

Signals for data integrity, consensus health, and write availability accounting. These catch silent corruption and failure modes that only surface after an incident.

Signal	What it tells you	Warning sign
`zk_quorum_ack_latency` (p99, leader-only)	Time from PROPOSE to quorum ACK	Above 50ms sustained: followers or network are the write bottleneck
`zk_proposal_count` / `zk_commit_count`	Write throughput pipeline health	Proposals climbing while commits stall: quorum cannot ACK fast enough
`zk_digest_mismatches_count`	Data tree checksum mismatches	Any increase: data integrity compromised. Investigate and rebuild from a healthy snapshot
`zk_unrecoverable_error_count`	Critical internal errors	Any increase: server integrity may be compromised. Page unconditionally on delta
`zk_snapshot_error_count`	Errors during snapshot create or load	Any increase: recovery safety at risk
`zk_sum_leader_unavailable_time`	Cumulative ms with no leader able to serve writes	Any non-zero delta in a recent window: writes were impossible for that duration
`zk_ensemble_auth_fail`	Server-to-server authentication failures	Any non-zero increment: threatens quorum
`zk_non_mtls_remote_conn_count`	Remote connections without mutual TLS	Non-zero in mTLS-required environments
`zk_observer_sync_time`	Observer lag versus the leader	Sustained high: observers serving stale reads to clients

Data integrity is delta-only. zk_digest_mismatches_count and zk_unrecoverable_error_count are counters. Alert on any increase, never on the absolute value, or you will re-page for historical events. A digest mismatch means the in-memory tree diverged from the expected checksum. Investigate and potentially rebuild the node from a healthy snapshot. If mismatches appear on multiple nodes, escalate immediately.

Track write availability directly. zk_sum_leader_unavailable_time measures cumulative milliseconds during which no leader could serve writes. In steady state this counter should not grow. Any non-zero delta in a recent window means writes were impossible for that duration. Pair it with zk_cnt_leader_unavailable_time to see how many episodes contributed to the total.

Quorum ACK latency isolates consensus cost. zk_quorum_ack_latency (leader-only) measures the time between the leader sending PROPOSE and receiving quorum acknowledgments. If it climbs while zk_fsynctime stays flat, the bottleneck is follower processing or network, not your leader’s disk. It should stay well below syncLimit x tickTime.

Common monitoring gaps that bite production

Mistakes that show up most often in ZooKeeper outage postmortems. If your checklist stops at Level 1 or 2, several of these are probably open right now.

Using ruok as the only health check. It confirms the process is alive but not that the node can serve requests or is part of quorum. A read-only node returns “imok”.
Relying on zk_avg_latency alone. Reads and writes are aggregated, so write stalls hide behind high read volume unless you split zk_updatelatency and zk_readlatency.
Not collecting zk_fsynctime. Disk latency is the root cause of most write stalls. Without it, root cause analysis turns into guesswork.
Forgetting maxClientCnxns. Default 60 per source IP silently rejects clients under containerized NAT. Watch zk_connection_rejected.
Not tracking zk_znode_count growth. The data tree lives entirely in heap. Slow growth over months ends in simultaneous OOM across all nodes, because they all hold the same tree.
Missing leader-only metrics. zk_synced_followers, zk_pending_syncs, and zk_learners (3.6+; zk_followers on 3.5.x) only appear on the leader. A follower-only collector never sees them.
Alerting on cumulative latency extremes. zk_max_latency is a high-water mark since last srst or restart. A spike from days ago stays elevated forever unless you reset stats or use 3.6+ percentiles.
Skipping data integrity signals. zk_digest_mismatches_count is the only early warning for in-memory data divergence, and almost no one monitors it.
Not separating dataLogDir from dataDir. A shared disk means snapshot I/O competes with transaction log fsync. This is a one-line config change that prevents intermittent write spikes.
No autopurge. Without autopurge.purgeInterval and autopurge.snapRetainCount configured, snapshots and transaction logs accumulate until the disk fills and ZooKeeper crashes with no warning.

How Netdata helps

Per-second mntr collection means every level of this checklist is continuously populated, not spot-checked at scrape intervals that miss transient GC pauses or election spikes.
Correlating zk_jvm_pause_time_ms with zk_connection_drop_count and zk_stale_sessions_expired separates GC-caused session storms from network-caused ones in a single timeline.
Correlating zk_fsynctime p99 with zk_updatelatency p99 confirms whether a write stall is disk-bound before it escalates into quorum loss.
Leader-only metrics (zk_synced_followers, zk_pending_syncs, zk_quorum_ack_latency) are collected per-node and tagged by zk_server_state, so the leader is always identifiable without a separate collector path.
Host-level disk free and disk latency on the dataDir and dataLogDir mounts sit next to ZooKeeper process metrics in the same dashboard, closing the gap between application signals and the storage they depend on.
Anomaly detection on slow movers like zk_znode_count, zk_watch_count, and heap usage flags creeping exhaustion weeks before the cliff edge.