Ceph monitor quorum lost: the cluster can no longer update its maps

Ceph monitor quorum loss is a PAGE condition. Without a majority of MONs agreeing through Paxos, the cluster cannot commit any map update: no OSD up/down transitions, no PG state changes, no pool edits, no CRUSH adjustments. Existing clients keep running on cached maps for a while, but new client connections fail immediately, and as soon as those cached maps go stale, in-flight I/O stalls.

The trigger formula is sum(ceph_mon_quorum_status) < floor(count(ceph_mon_quorum_status) / 2) + 1. For 3 MONs you need 2 in quorum. For 5 MONs you need 3. The standard alert sustain is 300 seconds, long enough to ride out a normal sub-second leader election and short enough to page before clients start timing out en masse.

MONs are not on the data path. That is the only reason this incident is not immediately catastrophic. But MONs are on the metadata path: every map change, every OSD failure, every rebalance decision goes through them. A cluster without quorum cannot react to anything else going wrong. Treat this as a stop-the-line incident.

What this means

Paxos requires a majority quorum to commit any map update. When quorum is lost, each MON daemon enters one of three states: probing (looking for peers), electing (trying to choose a leader), or synchronizing (a rejoining MON catching up on map history). A healthy MON is either leader or peon. Persistent electing is the classic signature of quorum loss.

While quorum is down:

• Existing clients continue serving I/O against their cached OSD maps. Reads and writes against already-known PGs keep working until those maps go stale.
• New client mounts, RBD attach operations, and any client that needs a fresh map fail immediately.
• Any concurrent failure (an OSD going down, a host reboot) cannot be recorded. The cluster cannot mark the OSD down, cannot start recovery, cannot protect itself from the next failure.
• CLI commands that require MON agreement (ceph osd ..., ceph pg ..., pool changes) hang or error out. Read-only inspection via the admin socket may still work if the MON process is alive on the host.

The blast radius widens with time. After the OSD map cache window expires on clients, in-flight operations stall. RBD-attached VMs may hit guest I/O watchdogs. CephFS clients may see cap recalls fail. RGW requests that need new pool metadata fail. Recovery time matters.

Common causes

Quick checks

Start with read-only inspection. None of these change cluster state.

# Top-level status. If this hangs, MONs are unreachable from your admin host.
timeout 10 ceph status

# Quorum membership and leader. Slow responses indicate MON stress.
ceph quorum_status -f json | jq '{epoch: .election_epoch, quorum: .quorum_names, leader: .quorum_leader_name}'

# Per-MON state from the admin socket. Run on each MON host.
ceph daemon mon.<id> mon_status

# Health detail, focused on MON checks.
ceph health detail | grep -E 'MON_|CLOCK|quorum'

# Clock skew as the cluster sees it.
ceph time-sync-status

# Current MON map (addresses the MONs believe are correct).
ceph mon dump

# Paxos and election counters on the local MON.
ceph daemon mon.<id> perf dump | jq '.paxos, .mon'

On each MON host:

# Is the daemon actually running?
systemctl status ceph-mon@<id>

# Time sync health.
chronyc tracking    # or: ntpq -p
timedatectl status

# Disk space on the MON data dir.
df -h /var/lib/ceph/mon/ceph-<id>
du -sh /var/lib/ceph/mon/ceph-<id>/store.db

# Recent MON log lines.
journalctl -u ceph-mon@<id> --since "30 min ago" | tail -100

How to diagnose it

1. Confirm the alert is real. Run ceph status with a short timeout. If the CLI itself hangs past a few seconds, the MONs your admin host talks to are unreachable or stuck. That is itself diagnostic.

2. Identify which MONs are out. From a working MON host, run ceph daemon mon.<id> mon_status. The state field tells you whether that MON sees itself as leader, peon, probing, electing, or synchronizing. Cross-check with ceph quorum_status if any MON is responsive.

3. Check the election epoch trend. A rapidly incrementing election epoch in ceph mon stat means MONs are fighting over leadership. This is typical of clock skew or a flapping peer, not of a clean host outage.

4. Verify time sync on every MON host. chronyc tracking shows the offset. Anything beyond mon_clock_drift_allowed (default 0.05s) shows up as MON_CLOCK_SKEW in ceph health detail. The upstream troubleshooting guide is blunt on this point: persistent electing without clock skew is rare and warrants deeper investigation.

5. Verify MON-to-MON connectivity. From each MON host, test TCP reachability to the others on port 3300 (msgr2) or 6789 (legacy msgr1), whichever the cluster uses. Check monmap addresses in ceph mon dump to be sure you are testing the addresses Ceph actually uses. A MON stuck in probing with peers reachable on ICMP but unreachable on the messenger port usually means a firewall or a stale monmap.

6. Check MON disk space. The MON store lives in /var/lib/ceph/mon/ceph-<id>/store.db. A full filesystem prevents Paxos writes and drops the MON out of quorum silently. This is common on MONs co-located with other services.

7. Inspect logs on any down MON. journalctl -u ceph-mon@<id> and the MON log file show assertion failures, RocksDB errors, store corruption messages, or OOM kills.

flowchart TD
  Q[Quorum-lost alert fires] --> C{ceph -s responds within 10s?}
  C -->|No, CLI hangs| H[Check MON hosts: alive? reachable on 3300/6789?]
  C -->|Yes, slowly| D[ceph daemon mon.X mon_status on each MON]
  D --> St{Persistent electing or probing?}
  St -->|Electing cycle| Sk[Check chronyc tracking on all MONs]
  St -->|Probing only| Net[Check MON-to-MON reachability vs monmap]
  Sk --> Skew{MON_CLOCK_SKEW active?}
  Skew -->|Yes| NTP[Fix time sync, wait one election cycle]
  Skew -->|No| Disk[Check MON data dir disk space]
  Disk --> Full{Any MON disk full?}
  Full -->|Yes| Free[Free space, restart MON]
  Full -->|No| Net
  Net --> Part{Partition or wrong monmap?}
  Part -->|Yes| FixNet[Resolve network or inject corrected monmap]
  Part -->|No| Crash[Investigate MON logs and store]

Metrics and signals to monitor

Fixes

The fix depends on which cause you found. In all cases, do not add or remove MONs mid-incident unless you understand exactly what you are doing. Changing MON membership while quorum is broken can leave the cluster with no valid monmap source and make recovery harder. The official guidance is to add new MONs first, wait for quorum to reform, and only then remove broken ones.

Clock skew

This is the most common root cause and the easiest to fix.

1. Identify the skewed MON via ceph time-sync-status and ceph health detail | grep CLOCK.
2. On the affected MON host, check chronyc tracking (or ntpq -p). Confirm the time source is reachable and the offset is shrinking.
3. If chrony or ntpd is stopped or misconfigured, fix the configuration and restart the daemon: systemctl restart chronyd (or ntpd).
4. Wait for the offset to drop below mon_clock_drift_allowed. The MON will rejoin quorum on its own once elections stabilize.
5. Do not manually restart MONs to “force” rejoining. Let Paxos settle.

VM-based MON hosts are the usual culprit. Live migration, overloaded hypervisors, and missing tsc clock sources all cause drift. Bare-metal MON hosts with working chrony rarely skew past a few milliseconds.

MON disk full

1. Identify which MON host has a full data volume via df -h /var/lib/ceph/mon/ceph-<id>.
2. Free space. Common offenders: rotated but unflushed journals, core dumps left in /var/lib/ceph, log bloat, other services co-located on the MON host.
3. If the store itself is bloated (large store.db), compact it: ceph daemon mon.<id> compact. This causes a brief latency spike but reclaims space and improves startup time.
4. Once disk space is available, restart the MON if it did not rejoin automatically: systemctl restart ceph-mon@<id>.

Do not delete files under store.db directly. The store is a RocksDB database; manual deletion corrupts it.

MON-to-MON network partition

1. Confirm with ceph mon dump that the addresses in the monmap are the ones MONs are actually trying to reach. A stale monmap after a host migration or IP renumbering is a common cause.
2. From each MON host, test TCP reachability to peers on the msgr2 port (3300) or legacy msgr1 port (6789). ICMP alone is not sufficient.
3. Resolve the underlying issue: switch config, firewall rules, routing, or a stale monmap.
4. If the monmap itself is wrong on every surviving MON, you need to inject a corrected monmap. This is a destructive operation: it involves stopping MONs, rebuilding the monmap with monmaptool, and replacing it on each MON’s data directory. Follow the upstream “Recovery using a rebuilt monmap” procedure for your deployment type (bare metal, cephadm, or Rook). Do not improvise.

MON crash or store corruption

1. Identify the failed MON from ceph mon stat and systemctl status ceph-mon@<id>.
2. Inspect the journal and log file. Assertion failures, RocksDB corruption errors, and OOM kills all show up here.
3. If the host itself is the problem (failed hardware, kernel panic), bring the host back or evacuate the MON role.
4. If the store is corrupted, the recovery path is to destroy and rebuild the MON. For non-cephadm deployments this is documented upstream as the “Recovery using OSDs” procedure, which builds a fresh MON store from OSD maps. For cephadm and Rook deployments, follow the deployment-specific variant. The rebuild tool has known limitations: it cannot recover MDS maps, partially created pools, or non-OSD keyrings.
5. Once the rebuilt MON is back and in quorum, validate cluster state with ceph health detail before declaring the incident over.

Prevention

• Run at least three MONs on three separate physical hosts in three separate failure domains. Five MONs is appropriate for larger or stretch clusters.
• Put MON data on SSD-backed storage. A slow MON store slows every Paxos round and every election.
• Dedicate the MON host or budget enough CPU and memory that the MON is never starved by a noisy neighbor.
• Configure chrony (or ntpd) on every MON host, point at reliable upstream sources, and alert on offset drift well before the 50ms threshold.
• Monitor MON store size over time. A growth rate above 100MB/day usually means excessive OSD map churn, often from flapping OSDs.
• Track election epoch rate. Even when quorum holds, frequent elections indicate a fragility that will eventually become a real outage.
• Periodically exercise the monstore-recovery procedure in a staging cluster. Operators who have never run it will struggle during a real incident.

How Netdata helps

• The per-second ceph_mon_quorum_status series, broken out by ceph_daemon, lets you see the exact moment each MON dropped out and whether the loss was simultaneous (network or shared dependency) or staggered (host-by-host cascade).
• Correlating quorum state with host-level CPU, memory, disk usage, and disk I/O on the MON hosts distinguishes host resource starvation from a Ceph-internal failure.
• The ceph_health_detail{name="MON_CLOCK_SKEW"} and MON_DOWN signals surface the leading indicators before quorum actually breaks.
• ML anomaly detection on MON host clock offset, disk latency, and network retransmits catches slow drift toward quorum loss before the alert fires.
• Election epoch rate and Paxos commit latency from the MON admin socket, collected per second, show election storms even when the binary quorum signal has not yet flipped.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine