Ceph PG incomplete: placement groups that cannot serve I/O

An incomplete placement group means the PG cannot find enough authoritative data to serve reads or writes. This is a data-availability failure, and it does not self-resolve the way a transient peering or recovering state does.

Alert on sum(ceph_pg_incomplete) > 0 sustained for more than 300 seconds, summed across all pools. The 300 second sustain filters out cold-start peering after a cluster-wide restart, which typically completes within 60-120 seconds. Anything still incomplete after five minutes is genuinely stuck.

What this means

Ceph marks a PG incomplete when the acting set of OSDs cannot reconstruct a consistent write log to peer from. If too many of the OSDs that hold pieces of that log are gone or unreachable, peering cannot complete. The PG cannot serve client I/O until peering succeeds.

The cluster surfaces this two ways:

• The PG_AVAILABILITY health check fires and pushes the cluster to HEALTH_ERR.
• The per-pool metric ceph_pg_incomplete (label pool_id) becomes non-zero for the affected pool.

flowchart TD
    A["OSDs in PG acting set down or destroyed"] --> B["Peering cannot reach consensus"]
    B --> C["PG marked incomplete"]
    C --> D["PG_AVAILABILITY health check fires"]
    C --> E["Reads and writes to affected objects fail"]
    C --> F["Alert: ceph_pg_incomplete > 0 sustained 300s"]
    G["Down OSDs return"] -.-> H["Peering resumes, PG exits incomplete"]
    I["Down OSDs permanently lost"] -.-> J["Manual recovery or accept data loss"]

The single most important triage question: are the missing OSDs merely down (and will come back) or permanently destroyed? The answer determines whether you wait and watch or enter a recovery scenario.

Common causes

Quick checks

Start with read-only commands. None of these change cluster state.

# Which PGs are stuck, and which pools they belong to
ceph health detail | grep -A2 incomplete

# List PGs in the incomplete state
ceph pg dump_stuck incomplete

# Inspect a specific PG's peering state
ceph pg <pgid> query

# Show the acting and up sets, primary, and blocked-by detail
ceph pg <pgid> query | jq '.recovery_state'

# Inventory of down OSDs and whether they share a host
ceph osd tree down

# Confirm whether down OSDs are destroyed or merely stopped
ceph osd dump | grep " down "

# Check for unfound objects that may also be blocking recovery
ceph health detail | grep unfound

# Verify the PG's pool EC profile if applicable
ceph osd pool ls detail | grep <pool>

The ceph pg <pgid> query output is the centerpiece. The recovery_state section explicitly tells you why peering is blocked. Look for peering is blocked due to down osds, the down_osds_we_would_probe list, and the peering_blocked_by detail. That list of OSD IDs is your work queue.

How to diagnose it

1. Identify the affected PGs and their pools. Run ceph health detail and ceph pg dump_stuck incomplete. Note the pool_id for each stuck PG so you can correlate with ceph_pg_incomplete{pool_id=...} later.

2. Query each stuck PG. Run ceph pg <pgid> query. Read the recovery_state block. The peering_blocked_by entry names the specific OSD IDs that the PG is waiting on, and down_osds_we_would_probe is the set the PG would probe if they were reachable.

3. Cross-reference with OSD status. For each blocked-by OSD ID, run ceph osd tree and ceph osd dump. Determine for each:

   • Is it down but the host is fine? Likely a crashed daemon. Check systemctl status ceph-osd@<id> and /var/log/ceph/ceph-osd.<id>.log.
   • Is the host itself down? Recover the host first.
   • Is the OSD marked destroyed or already removed via ceph osd rm? You are in permanent-loss territory.

4. Classify the failure.

   • All blocked-by OSDs are merely down and recoverable: restart them and let peering complete.
   • One or more blocked-by OSDs are permanently destroyed, but surviving shards exist on disk: use ceph-objectstore-tool export/import.
   • All copies or too many EC chunks are permanently destroyed: choose between accepting loss (mark_unfound_lost delete, ceph osd lost) or recreating the PG empty (ceph osd force-create-pg).

5. Check for compounding factors. Verify the noout flag is not preventing the cluster from healing around the failure. Check capacity with ceph osd df to ensure recovery has somewhere to go. A full or backfillfull target OSD will silently block recovery even when source data exists.

Metrics and signals to monitor

Fixes

Work through these in order. Every step past the first increases the risk of data loss.

Restore the down OSDs

If ceph pg <pgid> query shows blocked-by OSDs that are merely down (host up, daemon stopped), the fix is to bring them back. Restart the ceph-osd daemons, fix the underlying host issue, or replace a failed disk and re-create the OSD on the same ID. Once the missing OSDs rejoin, peering completes and the PG exits incomplete on its own.

This is the only fix that loses no data. Exhaust it before anything else.

Recover surviving shards with ceph-objectstore-tool

If a missing OSD is permanently destroyed but its disk still has readable data, you can extract the surviving PG content and import it onto a healthy OSD. Use ceph-objectstore-tool with --data-path <osd-data-dir> --op export --pgid <pgid> on the source and --data-path <target-osd-data-dir> --op import on the target. Stop both the source and target OSD daemons before operating on their stores.

Two warnings from operators who have done this:

• Do not use cp -r or rsync -aXP to copy PG directories. BlueStore’s internal xattrs and metadata are not preserved by ordinary file copy tools, and the target OSD will crash. Only ceph-objectstore-tool is safe for moving PG data.
• The --skip-journal-replay flag was a FileStore-era option for OSDs with corrupt journals. BlueStore does not use a separate journal, so this flag is not applicable to BlueStore OSDs. Confirm the correct flag set for your Ceph version and backend before running.

When importing on a target OSD whose monitors are unreachable, pass --no-mon-config to prevent ceph-objectstore-tool from trying to contact the monitor. Confirm the exact flag set for your Ceph version before running.

Mark permanently lost OSDs as lost

If an OSD is gone and cannot be recovered, ceph osd lost <osd-id> --yes-i-really-mean-it tells the cluster to stop waiting for it. This can unblock peering for PGs that were waiting on that OSD.

One operational trap: if the OSD has already been removed from the cluster map with ceph osd rm, ceph osd lost will refuse because the OSD no longer exists. The workaround is to recreate the OSD entry with the same ID (empty) so that ceph osd lost can proceed.

Reducing min_size on the pool does not reliably clear incomplete on its own. The PG needs a healthy copy to peer from, not just a lower threshold. Setting min_size to 1 without a healthy source does nothing useful and widens your data-loss exposure for other PGs in the pool. For erasure-coded pools, temporarily reducing min_size may allow recovery only if you still have at least k data chunks to reconstruct from.

Handle unfound objects

After recovery unblocks, some PGs may still be stuck on recovery_unfound because specific objects cannot be located on any surviving OSD. For replicated pools, ceph pg <pgid> mark_unfound_lost revert rolls back to an older version of the object, or delete removes it entirely. The action argument is required.

For erasure-coded pools, revert does not work. Only delete is supported, because EC cannot reconstruct a partial object without all k data chunks. This is a hard constraint of the encoding, not a tooling limitation.

Recreate the PG empty as last resort

ceph osd force-create-pg <pgid> --yes-i-really-mean-it recreates the PG with no data. The objects that lived in that PG are gone. Clients that try to read them will get errors. This is a data-loss operation and should only be used when:

• All recovery paths above have been tried.
• The data is genuinely unrecoverable.
• You have accepted the loss and communicated it to affected users.

There is no undo. Document the decision, the PG ID, and the approximate data scope before running it.

Prevention

• Keep noout time-bound. The most common preventable cause of incomplete PGs is a forgotten noout flag combined with later OSD failures. Treat ceph_osd_flag_noout set for more than 24 hours as a ticket.
• Maintain capacity headroom. Recovery needs somewhere to go. Stay at least 20% below backfillfull on average, and watch per-OSD variance. A cluster that cannot backfill cannot heal around failures, which is exactly when incomplete PGs appear.
• Separate failure domains correctly. Verify CRUSH rules actually place replicas across distinct hosts or racks. A misconfigured rule can leave a PG one host failure away from incomplete.
• Run deep scrubs on schedule. incomplete PGs sometimes emerge from corruption that scrub would have caught earlier. Do not leave nodeep-scrub set indefinitely.
• Test recovery paths before you need them. The first time you run ceph-objectstore-tool export/import should not be during an outage. Walk through the procedure on a test cluster.
• Track unfound objects as a leading indicator. A rising ceph_num_objects_unfound count means redundancy is eroding. Investigate before it becomes incomplete.

How Netdata helps

• Per-second ceph_pg_incomplete per pool lets you see the exact moment a PG enters the state and whether it clears during peering or stays stuck past the 300 second sustain window.
• Correlating ceph_pg_incomplete with ceph_osd_up and ceph_osd_in immediately identifies whether the cause is a transient daemon restart or a permanently lost OSD, which determines your recovery path.
• ML anomaly detection on ceph_num_objects_unfound surfaces a rising unfound-object count before it cascades into incomplete PGs.
• ceph_health_detail{name="PG_AVAILABILITY"} as a labeled metric distinguishes this condition from other ERR-level checks, so the alert you receive names the actual problem rather than the umbrella status.
• Tracking ceph_osd_flag_noout alongside OSD down events catches the noout trap that turns a single disk failure into an incomplete PG.
• Recovery rate correlation (ceph_pool_recovering_bytes_per_sec against degraded PG counts) tells you whether the cluster is healing or stalled once OSDs come back.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine
• Ceph nearfull: the 85% warning that decides whether the cluster can heal