ZooKeeper stale reads from followers: zxid lag and read-after-write surprises

A follower can pass every standard health check, serve reads at sub-millisecond latency, and still return data dozens of transactions behind what the leader just committed. No error is logged. No alert fires. The ensemble reports a leader, all followers are synced, and quorum is intact.

This is not a bug. ZooKeeper offers sequential consistency for reads, not linearizability. Reads are served locally from each server’s in-memory data tree, and that tree may lag behind the leader’s committed state by anywhere from a few transactions to far more under load. For configuration data or service discovery where eventual consistency is tolerable, this is fine. For distributed locks, leader election fencing, or read-after-write logic, it can cause duplicate work, lock violations, or lost updates.

The standard monitoring stack does not flag it. You need to actively compare zxid values across the ensemble and know which workloads are sensitive to the gap. This article covers why stale reads happen, how to detect zxid lag, and what mitigations actually work.

What sequential consistency means in practice

The ZAB protocol guarantees that writes are linearizable. Every mutation goes through the leader, is assigned a globally unique, monotonically increasing zxid (a 64-bit value with an epoch in the upper 32 bits and a counter in the lower 32 bits), is written to the transaction log, broadcast to followers, and committed only after a quorum acknowledges. Writes are totally ordered and durable.

Reads are different. Each server handles read requests locally from its own in-memory copy of the data tree. There is no quorum round-trip and no check that the server has applied the latest commits. The result is sequential consistency within a client session: a single client sees its own writes in order, but different clients connected to different servers can see divergent views of the data at the same instant.

This is a deliberate tradeoff. Local reads are fast (pure memory lookups, typically well under a millisecond) and scale horizontally across followers and observers. If every read required a quorum round-trip, ZooKeeper would lose the throughput that makes it useful as a coordination service. The cost is that followers can serve data behind the leader’s committed state.

How stale reads happen

A write commits on the leader after quorum ACK, but individual followers apply the commit to their in-memory tree asynchronously. A client that writes to the leader and then immediately reads from a follower that has not yet applied that commit sees the old value.

sequenceDiagram
    participant C as Client
    participant L as Leader
    participant F as Follower

    C->>L: setData("/app/leader", value2)
    L->>L: assign zxid=100, fsync txnlog
    L->>F: PROPOSE zxid=100
    F->>L: ACK zxid=100
    L->>L: quorum reached, commit
    L->>F: COMMIT zxid=100
    L->>C: success (zxid=100 committed)
    Note over F: applying commit to memory tree
    C->>F: getData("/app/leader")
    F->>C: returns stale value (zxid=99)
    Note over F: no error, sub-ms latency

Under normal conditions, the lag is small. The follower sync pipeline applies commits quickly, and the zxid gap between leader and follower is zero or one transaction. But under write bursts, follower GC pauses, or follower disk contention, the gap widens. A follower struggling to fsync proposals falls behind. It is still synced in the ZAB sense because it has acknowledged recent proposals, but its in-memory tree reflects an earlier zxid than the leader’s committed state.

The lag is invisible to most monitoring because the standard health checks all pass. ruok returns “imok”. isro returns “rw”. zk_server_state reports “follower”. Read latency stays low because reads are still memory lookups. The follower is not failing. It is serving correct data for a slightly earlier point in time.

Why this is the most insidious consistency issue

Nothing is technically broken. There is no error to grep for, no counter incrementing, no log line warning that data is stale. The only evidence is that a client received a value that does not reflect the latest committed write.

For most ZooKeeper use cases, this is harmless. Configuration data changes infrequently, and a few milliseconds of staleness is irrelevant. Service discovery registries tolerate eventual consistency by design. Ephemeral node existence checks for liveness are coarse enough that small lag does not matter.

The danger is in workloads that assume read-after-write consistency across different servers.

Distributed locks. A client acquires a lock by creating an ephemeral sequential node, then reads the lock directory to check whether it is the lowest-numbered node. If that read goes to a follower that has not yet applied the lock creation or a competitor’s node deletion, the client can make a wrong decision about whether it holds the lock. This is not ZooKeeper-level split-brain: ZAB ensures only one leader commits writes, so two leaders accepting conflicting writes is not the risk. The risk is application-level inconsistency where two clients believe they hold the same lock based on stale reads from different followers.

Leader election fencing. A system writes “I am the leader” to a znode, then another component reads that znode to decide whether to proceed. If the read is stale, the reader may act on outdated leadership state, undermining fencing logic that depends on freshness.

Read-modify-write cycles. A client reads a counter, increments it, and writes it back. If the read is stale, the increment is based on an old value, producing a lost update. The correct pattern is to use versioned writes with setData and a version check, which fails with BADVERSION if the znode was modified since the client last read it. Naive implementations that read from followers without version checks can silently lose updates.

Detecting zxid lag across the ensemble

The definitive signal is the zxid delta between the leader and each follower. mntr does not expose the zxid in any 3.x release; every server prints its last-processed zxid on the Zxid: line of the srvr/stat four-letter words (and in the AdminServer /commands/server_stats JSON).

# Check zxid on each ensemble member (mntr has no zk_zxid; use srvr)
echo srvr | nc zk1 2181 | grep Zxid
echo srvr | nc zk2 2181 | grep Zxid
echo srvr | nc zk3 2181 | grep Zxid

In steady state, all members should report the same zxid or differ by no more than a few transactions during active writes. A follower whose zxid is consistently behind the leader is serving stale reads.

Under production burst loads, staleness of several transactions is observable even on healthy clusters, and can grow far larger on slow or GC-stalled followers.

Read the zxid carefully. The upper 32 bits are the epoch, incremented on each leader election, and the lower 32 bits are the counter. A higher epoch always wins, even if the counter is lower. When comparing across servers, first confirm they are in the same epoch, then compare the counter portion.

The leader-only metrics add context about whether followers are struggling to keep up.

# On the leader: synced follower count and pending syncs
echo mntr | nc leader 2181 | grep -E 'zk_(synced_followers|pending_syncs)'

zk_pending_syncs greater than zero means the leader has sync operations in flight to followers. Sustained non-zero values mean followers cannot keep up with the write rate. zk_synced_followers below the expected count means some followers are connected but lagging. Both metrics are leader-only. A monitoring setup that scrapes only followers will never see them. Identify the leader first via zk_server_state, or scrape all members and filter for the leader’s values.

Verification: standard mntr in ZooKeeper 3.5 provides only aggregate latency (zk_avg_latency, zk_min_latency, zk_max_latency) plus the leader-side sync gauges (zk_synced_followers, zk_pending_syncs, zk_follower_sync_time). Since 3.6.0 the metrics framework adds read/write breakdowns (zk_readlatency, zk_updatelatency with zk_p50_* through zk_p999_* variants) and the zk_follower_sync_time / zk_observer_sync_time summaries (exposed as zk_avg_*/zk_max_*), available from both mntr and the Prometheus endpoint.

For read-scaled deployments using observers, observer lag does not threaten quorum, but clients reading from observers see whatever lag exists. The same zxid comparison applies: if the observer’s zxid lags the leader, its reads are stale.

Since ZooKeeper 3.5.3, four-letter commands must be whitelisted via 4lw.commands.whitelist in zoo.cfg (or the zookeeper.4lw.commands.whitelist system property). If mntr is not whitelisted, the command returns nothing and monitoring silently breaks. At minimum, whitelist mntr, stat, srvr, isro, and ruok for operational visibility.

Mitigations and their tradeoffs

Three approaches, in order of strength.

sync() before critical reads. The standard mitigation. A client calls sync() on a path before reading it. This tells the follower to catch up to the leader’s latest committed state before serving the read. For most production scenarios, this reduces staleness to a level that is practically safe.

The official documentation endorses sync() for exactly this case (“Client B should call the sync() method from the ZooKeeper API … before it performs its read”), but sync() is not a quorum operation: it only asks the single server the client is connected to to catch up. During a partition the server that answers the sync may itself be behind, so sync() is a strong practical mitigation, not a formal proof of linearizability; a quorum write before the read is the only way to get that guarantee.

Route sensitive reads to the leader. Connect the client directly to the leader for reads where freshness matters. This avoids the follower catch-up window entirely. The tradeoff is that the leader already handles all writes, so adding read load increases saturation risk. Reserve this for low-volume, high-criticality reads such as lock checks or fencing decisions.

Use versioned writes for read-modify-write. Instead of relying on read freshness, structure updates so staleness is detected at write time. setData with a version number fails with BADVERSION if the znode was modified since the client last read it. This turns a silent stale-read bug into an explicit exception the client must handle. This is the correct pattern for counters, configuration updates, and any mutation based on a prior read.

For workloads where none of these are sufficient and true linearizability is a hard requirement, ZooKeeper may be the wrong tool for that specific path.

Signals to watch

How Netdata helps

• Per-second zk_zxid collection across all members. Netdata scrapes mntr every second, making zxid divergence between leader and followers visible as a real-time trend rather than a manual nc exercise during an incident.
• Correlation with root causes. Overlaying zk_pending_syncs, leader vs follower zxid delta, and follower-side disk and JVM metrics on the same timeline shows whether lag is caused by follower disk, GC, or network.
• Separating stale-read scenarios from real failures. When zk_server_state, isro, and quorum metrics are all healthy but zxid diverges, the dashboard pattern points directly at the sequential-consistency issue rather than sending you down a quorum-loss investigation path.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper pending syncs growing: followers can’t keep up with the write rate
• ZooKeeper proposals not committing: proposal_count outpacing commit_count