ZooKeeper synced_followers below ensemble size: degraded fault tolerance

zk_synced_followers reports how many followers are currently synced with the leader. In a healthy ensemble it equals ensemble_size - 1 (voting members only; observers are excluded). When it drops, a follower is disconnected or lagging, and your fault tolerance margin has shrunk.

This metric is emitted only by the leader. Followers, observers, and standalone nodes do not report it. If your collector scrapes a fixed node or only followers, you have a blind spot: identify the leader dynamically, or scrape every node and keep only the values from the node reporting leader in zk_server_state.

Severity depends entirely on ensemble size and where the new value sits relative to the quorum floor. The critical edge is floor(ensemble_size / 2). At that point the leader plus its synced followers exactly form quorum, and one more failure breaks it. In a 3-node ensemble that edge is reached the moment you lose a single follower, so any drop is already a page.

What this means

zk_synced_followers counts followers fully synchronized with the leader and participating in the proposal-ack-commit pipeline. The expected value is ensemble_size - 1 (voting members only; observers are not counted). Anything below means a follower is not connected, or is connected but still catching up and not yet acknowledging proposals.

zk_learners counts connected learners (followers plus observers) whether or not they are synced. Comparing the two tells you which case you are in:

• zk_learners below the expected count (all followers plus observers): a learner is disconnected. The leader has no TCP session to it.
• zk_learners at the expected count but zk_synced_followers below the number of followers: a follower is connected but lagging. It has not finished catching up and is not part of the ack quorum.

Page thresholds:

• zk_synced_followers == floor(ensemble_size / 2) and ensemble_size > 1 and zk_uptime > 600s: page. The leader plus the synced followers exactly form quorum. One more follower loss means no quorum, no leader election success, no writes.
• One follower missing in a larger ensemble (a 5-node cluster at synced_followers = 3): ticket. Redundancy is degraded but the cluster can still survive another loss.
• Brief drop during a rolling restart: info, expected.

The following diagram shows how a single dropped follower maps to severity across common ensemble sizes.

flowchart TD
  A[Follower disconnects or lags] --> B[zk_synced_followers drops]
  B --> C{Ensemble size and loss}
  C -->|"3 nodes, 1 lost"| D["synced = 1 = floor(3/2)
PAGE: zero fault tolerance"]
  C -->|"5 nodes, 1 lost"| E["synced = 3
TICKET: degraded, still tolerant"]
  C -->|"5 nodes, 2 lost"| F["synced = 2 = floor(5/2)
PAGE: at quorum edge"]
  C -->|"7 nodes, 1-2 lost"| G["TICKET: still tolerant"]
  C -->|"7 nodes, 3 lost"| H["synced = 3 = floor(7/2)
PAGE: at quorum edge"]

A 3-node cluster deserves special attention. Quorum is floor(3/2)+1 = 2, and the leader is always one of those two. When synced_followers drops from 1 to 0, you are sitting exactly at quorum. There is no “degraded but okay” window in a 3-node ensemble the way there is in a 5- or 7-node ensemble. If you run 3 nodes, every missing follower is a page.

Common causes

Quick checks

These are read-only and safe to run during an incident. The leader-only metrics return nothing on a follower, so run them against every node and filter.

# Identify the leader and its replication metrics, across all ensemble members
for host in zk1 zk2 zk3; do
  echo "=== $host ==="
  echo mntr | nc -w 2 $host 2181 | grep -E 'zk_server_state|zk_learners|zk_synced_followers|zk_pending_syncs'
done

# Confirm which node is leader (followers fields are empty on non-leaders)
echo srvr | nc localhost 2181 | grep Mode

# Compare last-processed zxid across nodes. They should match or be within a few transactions.
for host in zk1 zk2 zk3; do
  printf "%s " "$host"; echo srvr | nc -w 2 $host 2181 | grep '^Zxid'
done

# Check pending syncs on the leader. Sustained non-zero means followers cannot keep up.
echo mntr | nc localhost 2181 | grep zk_pending_syncs

# If running 3.6+, check for non-voting followers that inflate synced_followers.
echo mntr | nc localhost 2181 | grep -E 'zk_synced_non_voting_followers|zk_learners'

# Verify the four-letter-word whitelist is not silently blocking mntr (3.5.3+).
echo ruok | nc localhost 2181
echo mntr | nc localhost 2181 | head -1

If mntr returns nothing or returns mntr is not executed because it is not in the whitelist, your monitoring is likely blind to all of these metrics. Check 4lw.commands.whitelist in zoo.cfg.

How to diagnose it

1. Confirm you are looking at the leader. Filter all nodes by zk_server_state and keep only the leader’s values for zk_learners, zk_synced_followers, and zk_pending_syncs. A collector pulling from a fixed node may be scraping a follower and seeing nothing.

2. Distinguish disconnected from lagging. Compare zk_learners and zk_synced_followers.

   • If zk_learners is also low: a learner is not connected at all. Move to network/process checks.
   • If zk_learners is at the expected count but zk_synced_followers is low: a follower is connected but lagging. Move to disk and sync checks.

3. Find which follower is missing. Compare zxid across all nodes. The lagging or disconnected follower will have a lower zxid, or be unreachable. For a follower mid-SNAP sync, the zxid will be far behind and the leader’s logs will show snapshot transfer activity.

4. Check the missing follower locally. On that node, look at zk_jvm_pause_time_ms, zk_fsynctime, and host-level iowait. A follower that cannot fsync proposals fast enough will fall behind and drop out of the synced set.

5. Check inter-node connectivity. ZooKeeper uses two ports between members: one for follower-to-leader traffic and one for leader election. A firewall blocking only the election port will not show up until the next election, at which point the ensemble may fail to re-elect. Verify both ports are reachable between every pair.

6. Account for non-voting followers (3.6+). zk_synced_followers counts all forwarding followers, including non-voting ones (for example, members removed via dynamic reconfig but still connected). If zk_synced_non_voting_followers is non-zero, subtract it to get the true voting synced count. Otherwise the metric can mask a quorum risk.

7. Gate on uptime. If zk_uptime on the leader is under 600 seconds, suppress non-critical alerts. Leader-only metrics only appear after election completes, and a cold start can produce transient low values.

Metrics and signals to monitor

Fixes

Treat the fix as specific to the cause. Do not restart services as a first move; restarting the leader loses leader-only metrics context and can trigger an election that worsens things.

Follower process down or unresponsive

Check ruok and srvr on the suspect node. If the process is gone, look at zk_uptime history and the JVM GC log for an OOM or long pause. Restarting that single follower is fine, but identify why it died first. If it OOM-killed, the data tree is likely too large for the heap; address that or it will die again.

Follower GC pause

If zk_jvm_pause_time_ms p99 on the follower is elevated, GC is freezing it. Java 9+ defaults to G1GC (the ZooKeeper startup scripts do not set a collector; ZooKeeper 3.6+ still supports JDK 8, where the JVM default is Parallel GC); older deployments may run CMS. Check heap sizing against zk_znode_count and zk_approximate_data_size. If the follower has the same data tree as the leader (it should), the same heap pressure that eventually hurts the leader is already hurting the follower.

Follower disk stall

If the follower’s zk_fsynctime is elevated, its transaction log disk cannot keep up. Confirm with host-level iostat -x on that node. Common causes: dataLogDir not separated from dataDir, shared storage, cloud storage burst credit exhaustion, or a colocated workload. The durable fix is dedicated low-latency storage for the transaction log. As a stopgap, identify and remove the competing I/O.

Network partition isolating one follower

If the follower looks healthy to itself but the leader has no session to it, suspect the network. Check both inter-node ports between the leader and that follower. A server-to-server auth problem can present the same way, so also check the quorum auth counters: zk_ensemble_auth_fail (mntr, 3.6+) reports ensemble (server-to-server) authentication failures.

Follower in SNAP sync

A follower that was offline long enough that the leader’s transaction log no longer covers its catch-up needs will receive a full snapshot transfer. This is expensive for both sides and can briefly degrade the leader. Verify the follower’s zxid is advancing. If it is not advancing at all, the sync has stalled and you need to investigate why (network, disk on either side). Do not interrupt a progressing SNAP sync.

Non-voting followers inflating the count (3.6+)

If zk_synced_non_voting_followers is non-zero, you have members still connected but no longer voting (common after dynamic reconfig). Your alerting math must subtract these. If you want them fully gone, complete the reconfig removal so they disconnect. Do not leave stale non-voting members in place; they make the quorum math misleading.

Prevention

• Query the leader, not a random node. Leader-only metrics (zk_learners, zk_synced_followers, zk_pending_syncs) are invisible on followers. Your collector must either identify the leader dynamically or scrape all nodes and filter by zk_server_state.
• Alert on the quorum edge, not just on “below expected”. A 5-node cluster at synced_followers = 3 is degraded; at synced_followers = 2 it is one failure from outage. Use floor(ensemble_size / 2) as the page threshold.
• Prefer odd-sized ensembles. A 4-node cluster needs 3 for quorum, the same as a 5-node cluster, but tolerates only 1 failure. You pay for 4 nodes and get the fault tolerance of 3.
• Gate alerts on uptime. Suppress during cold starts (zk_uptime < 600s) and brief rolling-restart transitions.
• Watch zk_synced_non_voting_followers on 3.6+. If your alerting uses zk_synced_followers directly, non-voting members can mask a real quorum risk.
• Separate transaction log storage on every node. Follower disk stalls are a leading cause of followers dropping out of the synced set.
• Verify the 4lw whitelist includes mntr. Since 3.5.3, a missing whitelist silently returns nothing, and many monitors interpret empty responses as healthy zeros.

A note on version drift: in 3.6.0, the mntr gauge zk_followers was renamed to zk_learners because the new gauge counts every connected learner (followers plus observers), which is what the old name actually measured (verified in LeaderZooKeeperServer.registerMetrics: 3.5.x exposed followers; 3.6.0 replaced it with learners). zk_synced_followers was not renamed and retains its semantics. Dashboards and alerts written against pre-3.6 metric names will silently return no data after upgrade.

How Netdata helps

• Netdata collects zk_server_state per node, so you can build alerts that automatically use only the leader’s values for zk_synced_followers, zk_learners, and zk_pending_syncs without hardcoding which host is the leader.
• Per-second resolution on zk_synced_followers and zk_pending_syncs catches transient drops that a 60-second scraper misses entirely, which matters for followers that flap in and out of the synced set.
• Correlating zk_synced_followers with zk_jvm_pause_time_ms and zk_fsynctime on each follower pinpoints whether a drop was caused by GC, disk, or network, without switching tools.
• Anomaly detection on zk_learners minus zk_synced_followers surfaces the “connected but lagging” case before it becomes a quorum edge.
• Tracking zk_synced_non_voting_followers alongside zk_synced_followers keeps the quorum math honest on 3.6+ ensembles that use dynamic reconfig.
• Alerting on the floor(ensemble_size / 2) edge, gated by zk_uptime, avoids both cold-start false positives and the far worse failure of not paging when you are one loss from outage.

Related guides

• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper outstanding requests growing: the request pipeline is backing up
• ZooKeeper proposals not committing: proposal_count outpacing commit_count
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals