Ceph monitor election storm: monitors that cannot hold a stable quorum

A Ceph monitor election storm is what happens when the MON cluster cannot complete and hold an election. Each round of Paxos leader election starts, partially completes, then restarts before the new leader can commit any map updates. The election epoch counter climbs, the leader name changes moment to moment, and ceph -s itself starts taking several seconds to return because even reading the current map requires a responsive leader.

Client I/O often keeps working for a while. Existing clients hold cached OSD and CRUSH maps and continue pushing reads and writes to the OSDs they already know about. What is frozen is the control plane: no OSD up/down changes commit, no PG remaps are accepted, no pool changes take effect, no new client maps are handed out. As the storm persists, the gap between the frozen maps and reality widens until clients and OSDs begin to time out.

This page covers how to recognize the storm quickly, what the stable causes are, and how to break the loop without making the quorum loss worse. It assumes a 3-MON or 5-MON deployment on Reef (18.x) or Squid (19.x), though the mechanics are similar back through Pacific.

What this means

Ceph monitors form a Paxos quorum. A strict majority (floor(n/2) + 1 of the MON count) must agree on every cluster map update before it is committed. Elections exist to pick the single leader that runs Paxos rounds. Under normal conditions an election completes in seconds and the cluster never thinks about it again.

An election storm is a sustained failure to keep that leader. The classic signature, drawn from operator reports and the upstream MON code, is the lease timeout loop. The leader grants itself a lease, the peons (non-leader quorum members) must acknowledge that lease before it expires, and when they cannot reach the leader in time they call a new election. The new election hands the lease to another monitor, which also cannot hold it, and the cycle repeats faster than any single round can commit a map update.

Verified for Reef/Squid: mon_lease defaults to 5 seconds, and the lease-ack interval is mon_lease_ack_timeout_factor (default 2), i.e. the leader waits 10 s of lease time before peons call a new election.

flowchart TD
    A[Leader elected] --> B[Leader grants Paxos lease]
    B --> C{Peons ack lease in time?}
    C -- yes --> D[Paxos round commits map update]
    D --> A
    C -- no, timeout --> E[Peon calls new election]
    E --> F[New leader elected]
    F --> G{Leader can keep lease?}
    G -- clock skew / slow store / partition --> E
    G -- yes --> B

The visible symptoms from the CLI:

• ceph -s takes several seconds (sometimes 10 or more) because the tool is waiting on a MON that keeps cycling.
• ceph quorum_status shows an election_epoch that ticks upward across repeated samples.
• The quorum_leader_name field changes between runs.
• ceph health detail shows MON_CLOCK_SKEW or, in the worst case, never returns a clean health check.

During the storm, OSD map epoch updates stall. OSDs that fail heartbeats do not get marked down. New OSDs that join do not get marked up. PGs that need to re-peer cannot, because peering decisions require committed map updates. If an OSD is also failing at the same time, the cluster cannot respond to it, which compounds the problem.

Common causes

On recent Ceph releases, clock skew is the first suspect. The upstream troubleshooting guidance is explicit that there is no obvious reason other than clock skew that explains why an electing state would persist.

Quick checks

Run these read-only. None change cluster state.

# Time a basic status call. Several seconds indicates MON trouble.
time ceph -s

# Quorum state, election epoch, leader name. Run it twice, seconds apart,
# and compare election_epoch and quorum_leader_name.
ceph quorum_status -f json | jq '{epoch: .election_epoch, leader: .quorum_leader_name, quorum: .quorum_names}'

# Per-MON state and election_epoch from each daemon's own view.
ceph daemon mon.<name> mon_status

# Clock skew view from the lead monitor.
ceph time-sync-status

# Health detail, filtered for the clock skew check.
ceph health detail | grep -i clock

# Paxos and election counters on each MON.
ceph daemon mon.<name> perf dump | jq '{paxos: .paxos, mon: .mon}'

# MON store size on each MON host.
du -sh /var/lib/ceph/mon/ceph-$(hostname -s)/store.db

# NTP/chrony state on each MON host.
chronyc tracking        # or: ntpq -p

If ceph quorum_status itself hangs, that confirms the MON cluster is not serving requests reliably. Sample election_epoch a few times. In a healthy cluster it barely moves. In a storm it climbs across every sample.

How to diagnose it

1. Confirm it is actually a storm. Sample ceph quorum_status three times, 5 seconds apart. If election_epoch increases between every sample and quorum_leader_name changes, you have a storm, not a one-off election.

2. Check clock sync first. Run ceph time-sync-status and chronyc tracking (or ntpq -p) on every MON host. The default mon_clock_drift_allowed is 0.05 seconds. Anything beyond that triggers MON_CLOCK_SKEW and is enough to destabilize elections. VM-hosted MONs are particularly prone to this.

3. Check MON-to-MON network reachability. From each MON host, ping every other MON host. Look for packet loss, not just latency. A partial partition where two MONs can reach each other but not the third produces exactly the ping-pong leader pattern of an election storm. Check switch error counters and NIC drop counters on the MON hosts.

4. Find the slow MON. On each MON, run ceph daemon mon.<name> perf dump and look at the paxos.commit_latency and paxos.accept_latency values. One outlier with commit latency in the hundreds of milliseconds (or worse) is the one that cannot keep Paxos pace. In a 3-MON cluster the leader needs only one peon ack to commit, but a slow peon will fail to ack its Paxos lease in time and trigger a new election. One slow MON destabilizes the whole quorum.

5. Check the MON store on each host. A bloated or corrupt store slows every Paxos round. Compare du -sh /var/lib/ceph/mon/ceph-<name>/store.db across MONs. A store that is an order of magnitude larger than the others, or that shows RocksDB errors in the MON log, is a candidate for removal and rebuild.

6. Check MON host resource pressure. Run top, iostat -xz 1, and free -m on each MON host. A MON co-located with busy OSDs, RGW, or other VMs can be starved of CPU or disk I/O at the exact moment it needs to respond to a Paxos round.

Metrics and signals to monitor

The Netdata Ceph collector surfaces ceph_mon_quorum_status and ceph_health_detail signals directly. The host-level NTP offset, CPU, and disk I/O on each MON host come from the standard Netdata system collectors. Correlating cluster-side signals with host-side signals tells you whether the storm is a Ceph problem or a host problem.

Fixes

Clock skew

Fix the underlying time sync. Do not raise mon_clock_drift_allowed to make the warning go away. Upstream is explicit that this masks the problem and the cluster will still misbehave, just without telling you.

1. Verify chronyd (or ntpd) is running on every MON host: systemctl status chronyd.
2. Verify each MON can reach its time source: chronyc sources.
3. If MONs are VMs, verify the hypervisor is not stealing time. VM-virtualized clocks are explicitly called out as unsuitable for steady timekeeping on monitor nodes. Prefer bare metal for MON hosts, or at minimum ensure the hypervisor is not oversubscribed.
4. Once clocks are within 50ms of each other and stable, elections should settle within a minute or two.

MON-to-MON network partition

A partition that splits the MONs into two groups that cannot reach each other produces unstable elections. The fix is network-level: find the bad link, switch port, or NIC and repair it. Do not try to tune Ceph around a broken network.

If the partition is asymmetric and one MON is isolated, you can stop that MON’s daemon to let the remaining MONs form a stable quorum. In a 3-MON cluster this leaves you with 2 MONs and zero fault tolerance, so treat it as a stopgap while you repair the network, not a permanent state.

One MON on slow storage

A MON whose store sits on spinning rust, or whose DB device is shared and saturated, cannot keep up with Paxos rounds. The symptom is one MON with paxos.commit_latency far higher than its peers.

1. Move the MON store to SSD-backed storage. The path is /var/lib/ceph/mon/ceph-<name>/store.db.
2. If the store has bloated, trigger a manual compaction with ceph daemon mon.<name> compact. This causes a brief latency spike during compaction, so run it during a maintenance window if possible.
3. If compaction does not help and the store is corrupt, rebuild the MON from a healthy peer.

MON store corruption

If a MON is logging RocksDB errors or refusing to join quorum, it may have a corrupt store. The safe recovery path is to remove that MON from the quorum, wipe its store, and let it rejoin from a healthy peer.

Current official recovery path (see “Removing Monitors from an Unhealthy Cluster” in the docs): stop the MON daemons, extract the monmap with ceph-mon -i <id> --extract-monmap, edit it with monmaptool --rm, re-inject it with ceph-mon -i <id> --inject-monmap, and let the surviving monitors rebuild quorum; ceph-monstore-tool is no longer the documented mechanism.

As a general pattern: stop the daemon, remove the MON from the MON map if needed, wipe the data directory, and redeploy the MON so it bootstraps from an existing quorum member. This is disruptive to that one MON but does not risk the quorum as long as the other MONs are healthy.

In extreme cases where all MONs are stuck in electing state and no quorum can form, operators have recovered by reducing to a single MON (removing all others from the MON map) so the survivor boots without waiting for quorum, then re-adding the others. This is a last resort with real risk: a single MON has no redundancy. Do it only when the alternative is a permanently frozen cluster, and re-add the second and third MONs immediately.

Overloaded co-located MON host

A MON that shares a host with busy OSDs, RGW instances, or other workloads can be starved at the wrong moment. The clean fix is to give MONs dedicated hosts, or at minimum dedicated CPU and disk resources. If that is not possible in the short term, reduce the load on the co-located services during the storm: throttle recovery, pause scrubs, or move workloads off the MON host.

Prevention

• Run MONs on dedicated, bare-metal hosts. Co-location with OSDs or RGW is the most common preventable cause of MON starvation. VMs are workable but riskier.
• Put the MON store on SSD. Spinning rust is not fast enough for reliable Paxos rounds on a busy cluster.
• Monitor NTP offset on every MON host continuously. Catch drift before it crosses the 50ms threshold.
• Monitor election_epoch rate. A healthy cluster barely increments it. More than a handful of elections per hour warrants investigation.
• Monitor MON store size and Paxos commit latency per MON. Outliers are the ones that will eventually destabilize quorum.
• Keep the MON network simple and redundant. Avoid asymmetric routing or flaky links between MON hosts. The classic election algorithm is known to misbehave under netsplit conditions.

Verified: on Reef and Squid the default election strategy is still classic (the docs recommend staying in it), connectivity remains opt-in for conventional clusters, and the documented stretch-mode procedure requires running monitors in connectivity mode.

On Reef and later, the connectivity election strategy (ceph mon set election_strategy connectivity) is available as an alternative to the classic lowest-rank algorithm and is designed to handle netsplit conditions better. It remains opt-in for conventional clusters, but the documented stretch-mode procedure requires it. If you operate a stretch cluster or one with historically flaky MON networking, evaluate it in a test environment before relying on it in production.

How Netdata helps

• ceph_mon_quorum_status per MON shows immediately which monitors are in and out of quorum, and whether membership is flapping. A graph that toggles every few seconds is the visual fingerprint of a storm.
• ceph_health_detail{name="MON_CLOCK_SKEW"} surfaces the most common root cause directly, with per-second resolution so you can see when skew started and whether your NTP fix took hold.
• Host-level CPU, I/O, and memory collectors on each MON host let you correlate a storm with co-located workload pressure. If commit latency spikes line up with OSD disk utilization spikes on the same host, you have your cause.
• Anomaly detection on election_epoch rate and per-MON quorum membership flags an emerging storm before operators notice that ceph -s is slow.

Related guides

• Ceph monitor quorum lost: the cluster can no longer update its maps
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD down: telling a dead disk apart from a network blip