Ceph PGs stuck peering: the slow peering loop after a mass restart

After a full cluster power cycle, mass OSD restart, or wide maintenance event, peering should finish within a minute or two. When it does not, ceph -s shows a long list of PGs in peering or peering+activating, the count is not decreasing, and client I/O for those PGs is hung. HEALTH_WARN or HEALTH_ERR can sit there for 20, 30, or 60 minutes with no visible progress.

The right question is not “why are PGs peering” (peering is the normal response to a topology change), but “why is peering not completing.” When many OSDs return at once, several unrelated conditions can stall the loop: monitor clock skew, an OSD flapping fast enough to re-trigger peering every cycle, missing OSDs the primary cannot probe, a peering_blocked_by_history_les_bound bound from a past interval, or a stuck requested_info_from request to an up-but-unresponsive peer. Some self-resolve when the underlying fault clears. Some never do, and the PGs degrade to down or incomplete, at which point the cluster crosses into the PAGE conditions for those states.

The triage path below is ordered by what resolves the largest fraction of real incidents first: clock skew, then flapping, then per-PG query of the blocking condition.

What this means

peering is the PG state in which the acting set of OSDs negotiates a consistent view of the placement group’s log and past intervals after a topology change. activating is the brief window after peering succeeds but before the PG is fully active. Both are normal during OSD restarts, CRUSH changes, and pool creation. The expected duration after a single OSD restart is a few seconds; after a mass restart, peering typically completes within 60-120 seconds once enough OSDs are back in the acting sets.

“Stuck peering” is an operational label, not a state machine value. Treat PGs that have been in peering (or peering+activating) for more than roughly 10 minutes without reaching active as stuck. At that point the OSDs are not forming consensus for those PGs. The monitor’s ceph pg dump_stuck machinery exists for exactly this case (its default threshold is mon_pg_stuck_threshold, 60 seconds in current releases; pass -t <seconds> to override). If peering never completes and the PGs fall to down or incomplete, the cluster crosses into the PAGE conditions defined for those states: sum(ceph_pg_incomplete) > 0 or sum(ceph_pg_down) > 0, both sustained for more than 300 seconds.

The diagnostic flow below mirrors the playbook’s first-response list for the slow peering loop.

flowchart TD
    A["PGs stuck in peering > 10 min"] --> B{"MON_CLOCK_SKEW active?"}
    B -- Yes --> C["Fix NTP/chrony on MON hosts"]
    B -- No --> D{"OSD_FLAPPING active?"}
    D -- Yes --> E["Stop the flap, set noout on OSD"]
    D -- No --> F["ceph pg pgid query"]
    F --> G{"peering_blocked_by populated?"}
    G -- "down_osds_we_would_probe" --> H["Bring missing OSDs back"]
    G -- "history_les_bound" --> I["Targeted config workaround"]
    G -- "empty" --> J["Restart peer or check for bad authorizer"]

Common causes

Quick checks

All read-only. None of these change cluster state.

# Cluster status and active health checks
ceph -s
ceph health detail

# List PGs stuck in non-active/non-clean states. Threshold defaults to
# mon_pg_stuck_threshold (60s in current releases); override with -t.
ceph pg dump_stuck inactive
ceph pg dump_stuck unclean
ceph pg dump_stuck stale

# Per-PG detail. Replace <pgid> with one of the PGs above.
ceph pg <pgid> query

# Clock skew from the monitor's perspective
ceph time-sync-status
ceph health detail | grep -i clock

# Flapping OSDs and current map epoch
ceph health detail | grep -i flapping
ceph osd stat

# OSD tree to spot missing or down OSDs in acting sets
ceph osd tree

How to diagnose it

1. Check the umbrella first. ceph -s and ceph health detail tell you whether this is a global problem (clock skew, MON discord, mass flapping) or isolated to specific PGs. If MON_CLOCK_SKEW is active, fix time sync before anything else. Clock skew above mon_clock_drift_allowed (default 0.05s) is a documented direct cause of stuck peering.

2. Confirm the OSDs the PGs need are actually up. Peering cannot complete if the primary cannot probe enough peers. ceph pg <pgid> query exposes down_osds_we_would_probe for OSDs the primary would have queried but cannot reach.

3. Look for flapping. Each up/down transition generates a new OSD map epoch and forces every PG on that OSD to re-peer. A single flapping OSD can keep hundreds of PGs peering indefinitely. There is no native flapping health check; watch OSD map epoch churn and up/down cycling in ceph osd stat. The underlying cause is usually heartbeat responses exceeding osd_heartbeat_grace (default 20s).

4. Read the recovery_state. For each stuck PG, ceph pg <pgid> query exposes peering_blocked_by and peering_blocked_by_detail. A populated peering_blocked_by is your answer. An empty one with a persistent requested_info_from osd.X points at a peer that is up but not responding to info requests.

5. Check OSD logs for auth failures. If an OSD cannot establish a session with its peers, peering cannot proceed. Search the OSD and MON logs for bad authorizer. Restarting the OSD may temporarily clear the symptom, but key reconciliation is the durable fix.

Metrics and signals to monitor

Fixes

Grouped by cause, in the order the playbook’s first-response list prescribes.

Clock skew

Fix NTP/chrony on the affected MON hosts. mon_clock_drift_allowed defaults to 0.05s, which is tight. Virtualized MONs are particularly prone to drift after live migration or hypervisor load. Verify with chronyc tracking on each MON. Do not disable the MON_CLOCK_SKEW check to silence the warning; that hides a real cause of stuck peering and a precursor to monitor quorum instability.

Flapping OSDs

Stop the flapping before peering can converge. The general pattern from the playbook’s flapping-cascade guidance is to set noout on the specific OSD (for example ceph osd add-noout osd.<id>) to prevent CRUSH remapping, then mark the OSD down and stop the daemon while you investigate the underlying disk, network, or memory pressure. Warning: marking an OSD down and stopping its daemon halts client I/O for any PG it primaries until recovery remaps them. Each flap re-peers every PG on that OSD, so leaving it flapping guarantees the slow peering loop continues. See the OSD flapping guide for the full cascade behavior.

Missing or unprobed peers

If ceph pg <pgid> query shows down_osds_we_would_probe, bring those OSDs back up or confirm whether they are gone permanently. If they are gone and the acting set cannot be satisfied, the PG eventually transitions to incomplete, which is a PAGE condition and a separate recovery problem.

peering_blocked_by_history_les_bound

When the last_epoch_start bound from a past interval prevents the PG from finding a compatible OSD, the documented operator workaround is to temporarily set osd_find_best_info_ignore_history_les = true on the affected OSDs (a real option, still present on Reef and Squid, default false and documented as extremely dangerous). Treat this as a targeted change. Revert once peering completes so the normal bound checks resume.

Stuck requested_info_from

If all acting-set OSDs are up, peering_blocked_by is empty, and the query shows a requested_info_from osd.X that never resolves, the operator pattern is to restart the named OSD. No formal fix exists for this loop. This is disruptive to that OSD’s other PGs, but it breaks the loop. Investigate the OSD’s logs and admin socket before restarting if you can do so quickly; otherwise restart first and diagnose from logs after.

Auth key mismatch

For bad authorizer entries in OSD or MON logs, verify the OSD’s keyring matches the key in the monitor’s auth database. Restarting the OSD may temporarily clear the symptom, but key reconciliation is the durable fix.

Prevention

• Keep NTP/chrony under monitoring on every MON host, with alerting on offset. MON_CLOCK_SKEW is a leading indicator not only for stuck peering but for monitor quorum problems more broadly.
• Stagger OSD restarts during planned maintenance when feasible. Peering is designed to handle topology changes, but thousands of PGs peering simultaneously stresses CPU and heartbeats, and on overloaded hosts can itself trigger the flap-and-re-peer loop.
• Watch heartbeat latency as a precursor to flapping. Heartbeat responses approaching osd_heartbeat_grace are the early warning, though the latency itself is not exposed through the Prometheus MGR module.
• Track OSD map epoch rate as a steady-state metric. A normally operating cluster produces epochs in single digits per minute at most. Sustained high rates indicate instability that will eventually block peering convergence.
• For erasure-coded pools, plan for slower peering. The larger shard counts make peering more resource-intensive and more sensitive to any single slow or missing peer.

How Netdata helps

• The ceph_pg_peering per-pool metric gives you a continuous, per-second view of how many PGs are in the peering state, so you can see the count flatten instead of relying on ceph -s snapshots.
• ceph_health_detail{name="MON_CLOCK_SKEW"} and ceph_health_detail{name="OSD_FLAPPING"} surface the two leading causes of stuck peering as labeled time series, so you can correlate the moment either check fires with the moment ceph_pg_peering stops decreasing.
• ceph_pg_down and ceph_pg_incomplete, summed across pools, are the escalation signals. Alerting on these with a 300-second sustain window catches the transition from “slow peering” to “data unavailable” without paging on the normal cold-start peering window.
• ceph_healthcheck_slow_ops correlates client impact. Ops stuck in waiting on pg rise at the same time peering stalls, which is the difference between “the cluster is healing” and “users are blocked.”
• ML anomaly detection on the OSD map epoch rate flags churn before the cluster crosses into flapping.
• Per-OSD commit and apply latency let you spot the individual OSD dragging out peering, rather than chasing the cluster-wide average.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine