Ceph PG stale: the monitor has not heard from the PG's OSDs

A stale placement group means the monitor cluster has stopped trusting the last status it received for that PG. The acting primary OSD that should be reporting state has gone silent, so the MON cannot confirm whether the PG is active, recovering, or unavailable. The recorded state is preserved but flagged as untrusted.

This differs from down. A down PG means the cluster has positively confirmed that no replica can serve I/O. A stale PG means the cluster does not know the current state because the reporting chain broke. Both can co-occur, but the response differs: down is confirmed replica loss; stale is a communication or reporting failure.

The MON declares a PG stale when it has not received a status update since the PG mapping changed, typically because the acting primary stopped reporting. The Prometheus signal is ceph_pg_stale per pool. The operational threshold is sum(ceph_pg_stale) > 0 sustained for more than 300 seconds. In Reef and later, stale is folded into the umbrella PG_AVAILABILITY health check rather than being a standalone PG_STALE check.

What this means

The MON builds its PG map from reports sent by OSDs. Each PG has an acting set, and the primary of that set reports PG state periodically. When reports stop arriving, the MON retains the last known state but marks it stale. The data on disk may be fine; you just cannot verify it from the monitor’s view.

Defining behavior:

• The primary OSD has stopped sending PG stats to the MON.
• The cluster cannot confirm whether the PG is still serving I/O.
• The last recorded state is preserved but flagged as untrusted.
• ceph pg dump_stuck stale lists the affected PGs.

Two configuration knobs define when staleness becomes “stuck”:

• mon_pg_stuck_threshold (default 1 minute since Quincy; pre-Quincy releases used 300 seconds): the minimum time a PG must remain stuck before it appears in dump_stuck output.
• mon_osd_report_timeout (default 900 seconds): the grace period before the monitor declares an unresponsive OSD down.

flowchart TD
  A[OSD stops reporting PG state] --> B{MON within grace period?}
  B -- yes --> C[PG marked stale in PG map]
  B -- no --> D[OSD declared down via mon_osd_report_timeout]
  C --> E[ceph pg dump_stuck stale]
  E --> F{Is the OSD actually running?}
  F -- no --> G[OSD failure or host down]
  F -- yes --> H[Network partition or OSD overload]
  G --> I[Recover or replace OSD]
  H --> J[Fix network or throttle recovery]

Common causes

Quick checks

# Cluster summary and active health checks
ceph status

# Full health detail including specific stale PG IDs
ceph health detail

# List PGs in each stuck category; specify a meaningful duration
ceph pg dump_stuck stale 300
ceph pg dump_stuck inactive 60
ceph pg dump_stuck unclean 300

# Inspect OSD up/down and in/out state
ceph osd tree

# Per-OSD commit and apply latency
ceph osd perf

# Confirm noout / norecover / nobackfill flags are unset
ceph osd dump | grep flags

# Query a specific stale PG for acting set and last-peering detail
ceph pg <pgid> query

All of the above are read-only.

How to diagnose it

1. Identify the affected PGs. Run ceph health detail and copy the listed stale PG IDs. Note their pools.
2. List the acting set for each stale PG. ceph pg <pgid> query shows the acting set, up set, and last-known state. If every acting OSD is down, this is an OSD failure, not a reporting issue.
3. Correlate with OSD state. Cross-reference acting sets against ceph osd tree. If the acting primary is down+in, the cluster is waiting on mon_osd_down_out_interval (default 600s) before marking OUT and triggering recovery.
4. Check for active network problems. Look for MON_CLOCK_SKEW, monitor quorum instability, or OSD_FLAPPING. A partition between OSD hosts and MON hosts produces stale PGs even when the OSDs are running.
5. Check for slow or stalled OSDs. ceph osd perf plus ceph health detail | grep SLOW_OPS tells you if the primary is alive but unable to send reports because it is saturated.
6. Check the flags. ceph osd dump | grep flags. If noout, norecover, or nobackfill are set, recovery will not progress and stale PGs can persist past their self-healing window.
7. Inspect the stale state combination. stale alone is usually a communication problem. stale+active+clean is a different scenario and typically follows an OSD removal while still referenced in a PG’s acting set. It is not a documented state in the PG-state reference but is operator-reported (including in Rook clusters after full reboots); Ceph’s PG states are a bitmask, so stale legitimately combines with active+clean.

Metrics and signals to monitor

There is no dedicated PG_STALE alert in the upstream ceph-mixin Prometheus rules (verified against Reef, Squid, and current main; stale is covered by the PG_AVAILABILITY-based CephPGsInactive alert). Alert on ceph_pg_stale > 0 sustained for 300 seconds, or on the PG_AVAILABILITY health check.

Fixes

Stale because the acting OSD is down

The straightforward case. The primary OSD is down and the cluster needs to peer the PG on a surviving replica.

1. Confirm the OSD is genuinely failed: ceph osd tree, then inspect the OSD daemon and host.
2. Decide whether to bring it back (failed host reboot, daemon restart) or let recovery handle it.
3. If recovery should proceed, ensure noout is unset: ceph osd unset noout.
4. Watch ceph pg dump_stuck stale shrink as peering completes on the surviving replicas.

If the OSD is not coming back, mon_osd_down_out_interval (default 600s) eventually marks it out, which triggers CRUSH remapping and full peering on the replacements.

Stale because of a network partition

The OSDs are running but cannot reach the MONs.

1. Verify connectivity from OSD hosts to MON hosts: ping, and check for packet loss on the relevant interfaces.
2. Check TCP retransmits on the cluster network. Persistently high retransmit rates indicate a link-layer problem.
3. Verify NTP/chrony is healthy on MON hosts. Clock skew does not directly cause stale PGs but commonly accompanies partition-related instability.
4. Resolve the partition. Stale PGs should clear once reports start flowing again.

Stale because of an overloaded OSD

The primary OSD is up and reachable but saturated to the point that PG reports are delayed past mon_pg_stuck_threshold.

1. Check ceph osd perf for outlier commit or apply latency.
2. Check slow ops: ceph health detail | grep SLOW_OPS.
3. Identify the cause: recovery storm, deep scrub, BlueStore compaction, or a failing device.
4. Throttle the load. Standard levers are osd_max_backfills (default 1) and the recovery-active limits: osd_recovery_max_active is 0 by default (Ceph picks per device class), with osd_recovery_max_active_hdd = 3 and osd_recovery_max_active_ssd = 10. Temporarily setting noscrub and nodeep-scrub frees I/O if scrub is the trigger.

Stale+active+clean after OSD removal

An operator-reported scenario. The OSD was removed from the cluster but the PG’s last-known acting primary epoch still references it. The PG may be technically active+clean on surviving OSDs, but the MON cannot trust that state because the recorded primary no longer exists. This does not self-heal.

The operator-reported workaround is ceph osd force-create-pg <pgid>. This is destructive: it destroys all data on the affected PG and recreates it from scratch. Use only when:

• The PG is genuinely stuck in stale+active+clean after an OSD was permanently removed.
• The data is recoverable from another source, or its loss is acceptable.
• You have confirmed via ceph pg <pgid> query that the recorded acting primary is no longer a valid OSD.

This pattern is also reported in Rook deployments after all nodes reboot, even when every OSD pod comes back healthy. If you are on Rook, check the project issue tracker for current guidance before reaching for force-create-pg.

Stale on a replicated pool of size 1

With replication size 1, there is no secondary OSD to take over reporting. Once the single acting OSD goes silent, the PG stays stale indefinitely. The fix is structural: either restore the missing OSD, or reconfigure the pool to use replication or erasure coding so future failures have a reporting fallback. Accept data loss on the affected PG if the OSD is unrecoverable.

Prevention

• Monitor ceph_pg_stale directly, not just ceph_health_status. Stale PGs are folded under PG_AVAILABILITY and can be drowned out by other PG state warnings in the umbrella signal.
• Track per-OSD latency outliers. Most stale PGs that are not pure OSD-down events come from a saturated primary. Alert when any OSD’s commit or apply latency exceeds 5x the cluster median for the same device class.
• Catch the noout trap. Stale PGs combined with a forgotten noout flag is a common preventable outage. Alert on ceph_osd_flag_noout == 1 for more than 24 hours.
• Verify network separation. Public and cluster networks should be physically or VLAN-separated. Sharing a single NIC for client and recovery traffic makes partition-induced stale PGs more likely during recovery storms.
• Avoid size-1 replicated pools for any data you care about. They offer no failure recovery and turn any primary OSD silence into a permanently stale PG.
• Track scrub flag state. A noscrub or nodeep-scrub flag left set after maintenance does not directly cause stale PGs, but it tends to coincide with the kind of deferred maintenance window where stale issues go unnoticed for too long.

How Netdata helps

• Netdata surfaces ceph_pg_stale per pool alongside ceph_pg_down, ceph_pg_incomplete, and ceph_pg_degraded, so you can immediately distinguish “we lost reports” from “we lost replicas”.
• Per-second collection on ceph_osd_up, ceph_osd_in, and ceph_healthcheck_slow_ops lets you correlate the moment a stale PG appears with the exact OSD state change or slow-ops spike that caused it.
• Cluster flags like ceph_osd_flag_noout, ceph_osd_flag_norecover, and ceph_osd_flag_nobackfill are exposed as metrics, so a stale PG alert can be cross-referenced against the flag state blocking recovery.
• The same dashboard surfaces ceph_mon_quorum_status and MON_CLOCK_SKEW health detail, so partition-induced stale PGs are visible next to their root cause rather than in isolation.
• Anomaly detection on ceph_pg_stale and per-OSD latency gives early warning before the 300-second sustained threshold is crossed, which is useful for slow-developing network or overload conditions.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine