$ guides / zookeeper / zookeeper-monitoring-maturity-model ▌

Operations Guides

ZooKeeper monitoring maturity model: from survival to expert

Most ZooKeeper monitoring stops too early. Teams run ruok as their only health check, collect zk_avg_latency without understanding it is cumulative since the last reset, and never look at fsync latency until a write stall cascades into a Kafka outage. The ensemble looks healthy in dashboards until it suddenly does not, and the postmortem reveals the signals were there all along, uncollected.

This article maps ZooKeeper monitoring across four maturity levels: Survival, Operational, Mature, and Expert. Each level answers a harder question than the one before. Survival asks whether the process is alive. Operational asks whether it is serving clients well. Mature asks why it is degrading. Expert asks whether the data is correct and whether the next failure will be silent.

Use this as a gap assessment. If your monitoring stops at Level 1, you will catch crashes but miss the GC death spiral that caused them. Level 3 covers most production incidents. Level 4 is where you catch data corruption, silent read staleness, and the OS-level pathologies that turn one slow disk into an ensemble-wide outage.

flowchart TD
    L1["Level 1: Survival
ruok, isro, server_state, disk"] --> L2["Level 2: Operational
latency, connections, followers, FDs"]
    L2 --> L3["Level 3: Mature
fsync p99, GC pauses, watches, zxid"]
    L3 --> L4["Level 4: Expert
quorum_ack, digest mismatches, THP/NUMA"]

How to use this model

Read each level as a checklist. If you are missing signals from your current level, fill those gaps before adding signals from the next. Jumping straight to Expert metrics without the Operational foundation means you will have rich data on quorum ACK latency but no idea whether the cluster is even accepting connections.

Most metrics listed come from the mntr four-letter command or its equivalents: the AdminServer HTTP endpoints on port 8080 (ZooKeeper 3.5+), or the Prometheus MetricsProvider on port 7000 (3.6+). Since ZooKeeper 3.5.3, four-letter commands must be whitelisted via 4lw.commands.whitelist in zoo.cfg. If mntr is not whitelisted, monitoring systems receive empty responses and interpret them as healthy, which is worse than no monitoring at all.

Level 1: Survival

Question: Is ZooKeeper alive?

At this level, you can detect process crashes, quorum loss, and full disks. You cannot detect latency degradation, GC issues, replication lag, or capacity exhaustion.

Signal	What it tells you
`ruok` returns `imok`	The process is alive and accepting commands on the client port
`isro` returns `rw`	The node has quorum and can accept writes
`zk_server_state` from `mntr`	Exactly one node reports `leader`, the rest report `follower`
`zk_uptime`	Detects unexpected restarts; gates cold-start false positives
Disk space on `dataDir` and `dataLogDir`	A full transaction log partition is immediately fatal

What you catch: process death, OOM kill, port unreachable, disk full, quorum loss via isro returning ro.

What you miss: everything else. A node returning imok can be in LOOKING state with no leader. isro returning rw does not mean latency is acceptable. This is the most common monitoring state in the industry, and it is the source of most “monitoring did not catch it” postmortems.

Common mistake: using ruok as the sole health check. A server in read-only mode, unable to serve any writes, still returns imok. Always pair ruok with isro, and verify zk_server_state rather than trusting either four-letter command alone.

Level 2: Operational

Question: Is ZooKeeper serving clients well?

This is what a competent production team monitors. You add request pipeline visibility, connection health, replication state, and resource pressure.

Signal	What it tells you
`zk_avg_latency`, `zk_max_latency`	Request processing latency (cumulative since last reset)
`zk_outstanding_requests`	Request pipeline backlog; sustained non-zero means saturation
`zk_num_alive_connections`	Client connectivity; sharp drops indicate mass disconnection
`zk_connection_rejected`	Clients hitting `maxClientCnxns` per-IP limit (default 60), silently refused
`zk_connection_drop_count`	Connection instability rate
`zk_open_file_descriptor_count`	FD pressure; approaching the OS limit blocks new connections
`zk_znode_count`	Data tree size; unbounded growth leads to heap exhaustion
`zk_learners`, `zk_synced_followers` (leader only)	Replication health and quorum margin
JVM heap usage and GC pause frequency	Memory pressure and pause impact

What you catch: most operational degradation. Sustained high latency, pipeline saturation, connection storms, FD exhaustion, data tree bloat, and replication degradation.

What you miss: the root cause of latency spikes. zk_avg_latency aggregates reads and writes, so a write stall can hide behind high read volume. Without separated read/write latency and fsync timing, you know something is slow but not why. Cumulative latency metrics (pre-3.6) never reset unless you call srst, so dashboards showing max_latency of 15000ms may reflect a days-old GC pause.

Common mistakes at this level:

Cumulative latency confusion. zk_avg_latency and zk_max_latency are cumulative since the last reset or restart, not sliding windows. Without delta computation or 3.6+ histogram metrics, latency monitoring is misleading.
Leader-only metric blindness. zk_learners (renamed from zk_followers in 3.6.0), zk_synced_followers, and zk_pending_syncs only appear on the leader. Monitoring that queries a follower will never see replication health.
Silent connection rejection. In containerized environments where many pods share a host IP, the default maxClientCnxns of 60 is easily exceeded. Rejections are silent from the server side. The only evidence is zk_connection_rejected incrementing.

Level 3: Mature

Question: Why is ZooKeeper degrading?

This is full operational visibility for a senior SRE. You separate read and write latency, track the disk write path directly, monitor GC impact on sessions, and watch for watch storms and session expiry.

Signal	What it tells you
`zk_updatelatency` p99, `zk_readlatency` p99 (3.6+)	Separated write and read tail latency; isolates write pipeline from local memory lookups
`zk_fsynctime` p99 (3.6+)	Transaction log disk write latency; the root cause metric for write stalls
`zk_jvm_pause_time_ms` p99 (3.6+)	GC pause duration; correlates with session expirations and elections
`zk_throttled_ops` (3.7+)	Operations throttled at `globalOutstandingLimit` (default 1000); any non-zero rate means saturation (3.6 exposes `zk_request_throttle_wait_count`)
`zk_watch_count`	Active watch registrations; unbounded growth is watch-storm risk
`zk_approximate_data_size`	In-memory data footprint; combined with znode count, estimates heap consumption
`zk_looking_count`	Election frequency; any increment outside maintenance indicates instability
`zk_follower_sync_time`	How long followers take to sync; approaching `syncLimit * tickTime` means ejection risk
`zk_pending_syncs` (leader only)	Replication lag; sustained non-zero means followers cannot keep up
`zk_stale_sessions_expired`	Session expiration rate; any non-zero increment outside maintenance is abnormal
Per-server `zk_zxid` comparison	Replication lag detection across ensemble members
OS-level: iowait, disk await, swap usage	Physical root causes behind application-level symptoms

What you catch: the specific subsystem causing degradation. Disk (fsynctime), JVM (pause time), network (quorum ACK), or data structure (znode count, watch count). You can answer this in minutes, not hours.

What you miss: data integrity violations, silent read staleness on followers, and the deepest quorum consensus signals. You also miss the OS-level tuning issues (THP, NUMA) that amplify GC pauses.

Key correlations that define this level:

zk_fsynctime p99 spikes alongside zk_updatelatency p99: disk is the write bottleneck.
zk_jvm_pause_time_ms p99 spikes alongside zk_stale_sessions_expired and zk_looking_count: GC is driving instability.
zk_outstanding_requests growing with zk_throttled_ops incrementing: pipeline is saturated and applying backpressure.
Per-server zxid delta growing between leader and a follower: that follower is falling behind and may require a full SNAP sync.

Level 4: Expert

Question: Is the data correct, and will the next failure be silent?

This is full operational mastery. You add quorum consensus internals, data integrity verification, recovery safety, security signals, and the OS-level pathologies that amplify JVM pauses.

Signal	What it tells you
`zk_quorum_ack_latency` p99 (leader only)	Time for followers to ACK proposals; isolates network from disk causes
`zk_proposal_count`, `zk_commit_count`	Write throughput pipeline; divergence means quorum problems
`zk_digest_mismatches_count`	Data tree checksum mismatch; any increment means corruption
`zk_unrecoverable_error_count`	Critical internal errors; any increment means integrity is compromised
`zk_snapshot_error_count`	Snapshot creation or loading errors; recovery safety at risk
`zk_sync_processor_queue_size`	Write pipeline depth; growing means write throughput exceeds processing
`zk_sum_leader_unavailable_time` (3.7+)	Cumulative write unavailability; any non-zero delta means writes were impossible
`zk_ensemble_auth_fail`	Server-to-server auth failures; threaten quorum
`zk_non_mtls_remote_conn_count`	Remote connections without mutual TLS; encryption compliance
`zk_observer_sync_time` (3.7+)	Observer lag; stale reads on observers
SNAP-sync detection (leader logs)	Followers requiring full snapshot transfer; expensive for the leader
THP status on ZooKeeper hosts	Transparent Huge Pages can multiply GC pause duration significantly
NUMA topology for JVM placement	Cross-node GC memory operations add pause time
Client-side session events	Reconnection loops and expiry events as experienced by clients

What you catch: data corruption before clients notice (zk_digest_mismatches_count). Write unavailability measured directly (zk_sum_leader_unavailable_time). Security violations (zk_ensemble_auth_fail, zk_non_mtls_remote_conn_count). OS-level amplifiers of JVM pauses (THP, NUMA). The gap between server-side health and client-side experience.

What this requires beyond metrics scraping: ZooKeeper does not expose all of these signals natively. SNAP-sync detection requires parsing leader logs for “Sending snapshot.” THP status requires reading /sys/kernel/mm/transparent_hugepage/enabled on each host. Client-side session events require application-level instrumentation in client libraries, since ZooKeeper has no client-side metrics endpoint.

Why this level matters: zk_digest_mismatches_count is the only early warning for data corruption. A single mismatch means a node is serving incorrect data to clients. Without this signal, you discover corruption when applications fail in ways that make no sense, and by then the corrupted state may have propagated.

Where most teams get stuck

The gap between Level 2 and Level 3 is where most teams stall. The reasons are structural:

zk_avg_latency looks sufficient until it is not. Reads are fast (local memory lookups) and writes are slow (quorum consensus plus fsync), so a high read volume masks write stalls in the aggregate. Teams see a healthy average and miss writes timing out.
fsync latency is not a prominent metric. Pre-3.6, fsync timing comes from ZooKeeper logs or OS-level iostat. Many teams never correlate the two. On 3.6+, zk_fsynctime is available with percentiles but is still rarely collected.
Percentile data requires 3.6+. Teams on 3.4.x or 3.5.x collect avg, min, and max and miss tail latency. A p99 fsync of 50ms with an average of 2ms tells a different operational story.
Leader-only metrics are missed by follower-only monitoring. zk_learners, zk_synced_followers, and zk_pending_syncs only appear on the leader. Monitoring that queries a random node or only followers will never see replication health. Identify the leader and query it, or query all nodes and filter for leader-reported values.
JVM GC is treated as a black box. zk_jvm_pause_time_ms is available in the Prometheus MetricsProvider (3.6+) and is also surfaced in mntr output (as zk_avg_jvm_pause_time_ms, zk_p99_jvm_pause_time_ms, etc.) but is rarely collected. GC pauses are a leading cause of session expirations and unnecessary leader elections, yet most teams monitor heap usage without monitoring pause duration.

How Netdata helps

Correlation is where ZooKeeper monitoring earns its keep. A single metric in isolation rarely identifies the problem. Per-second collection across all four maturity levels connects signals that a 60-second scrape interval would miss.

Liveness and function on one timeline. zk_server_state, isro status, and zk_outstanding_requests together distinguish a dead process from a live one that has lost quorum.
Write path root cause isolation. zk_fsynctime p99 next to zk_updatelatency p99 separates disk-caused stalls from quorum-ACK delays, the two most common write degradation causes.
GC and election correlation. zk_jvm_pause_time_ms p99 overlaid with zk_looking_count and zk_stale_sessions_expired reveals whether GC is the root cause of ensemble instability.
Replication margin in real time. zk_synced_followers, zk_pending_syncs, and per-server zxid comparison show how close the ensemble is to quorum loss before it happens.
OS context on the same host. Disk await, iowait CPU, and swap usage alongside ZooKeeper metrics connect application symptoms to their physical cause without switching tools.