$ guides / ceph / ceph-capacity-death-spiral ▌

Operations Guides

Ceph capacity death spiral: an OSD fails and recovery has nowhere to go

An OSD fails on a cluster you have been running at 80% or higher. ceph -s shows recovery starting, then ceph health detail lists OSD_NEARFULL and BACKFILL_TOOFULL. The degraded PG count climbs, then flatlines. Recovery bytes per second sits near zero. Nothing is healing, and you are one more failure away from data loss.

This is the Ceph capacity death spiral. It is not a single fault: it is the intersection of tight capacity, CRUSH imbalance, and the hard thresholds Ceph uses to protect itself. Recovery needs spare space on the surviving OSDs. When those OSDs are already past the backfillfull ratio (default 0.90), Ceph refuses to push more data onto them, and backfill stalls for every PG that would target them.

The signature is BACKFILL_TOOFULL (or the health check PG_BACKFILL_FULL) alongside OSD_NEARFULL, a flat or rising degraded PG count, and recovery rate near zero. That combination means the cluster has lost its ability to self-heal until you add space, free space, or move the thresholds. This article walks through confirming the pattern and the tradeoffs of each response.

What this means

Ceph uses three capacity thresholds on each OSD, set as ratios of utilized space:

Threshold	Default ratio	What happens when crossed
nearfull	0.85	HEALTH_WARN. Backfill may be throttled.
backfillfull	0.90	New backfill operations to this OSD are refused. PGs targeting it enter backfill_toofull.
full	0.95	HEALTH_ERR. All writes to this OSD stop.

The expected ordering is nearfull < backfillfull < full. Violating that ordering raises OSD_OUT_OF_ORDER_FULL.

Recovery after an OSD failure is not free. The data from the failed OSD has to land on the surviving OSDs in the same pool and failure domain. If those OSDs are already at or above the backfillfull ratio, Ceph will not start new backfill to them. The PGs that need that target OSD sit in backfill_wait with the backfill_toofull flag set.

flowchart TD
    A[OSD fails] --> B[Recovery targets surviving OSDs]
    B --> C{Target OSDs above backfillfull?}
    C -- No --> D[Backfill proceeds, PGs heal]
    C -- Yes --> E[PGs enter backfill_toofull]
    E --> F[Degraded PG count flat or rising]
    F --> G[Recovery rate near zero]
    G --> H[Cluster stuck at reduced redundancy]
    H --> I[Second failure risks data loss]

The danger is the exposure window. With degraded PGs not healing, the cluster runs with fewer replicas than the pool size specifies. For a 3x replicated pool, some objects may be down to a single good copy. If the OSD holding that last copy fails before recovery unblocks, those objects become unfound, which is potential data loss.

A related trap: when OSDs are marked OUT, their capacity is subtracted from the cluster total. A cluster that looked comfortable at 78% can jump past nearfull once several OUT OSDs leave the denominator. Always check ceph osd df tree for the real picture, including which OSDs are IN versus OUT.

Common causes

Cause	What it looks like	First thing to check
Running hot (>70% average)	Cluster average near 80%, single OSDs already at 85%+	`ceph osd df tree`: compare average to max OSD utilization
Uneven CRUSH distribution	A few OSDs at 90%+ while average is 75%	`ceph osd df tree` sorted by utilization variance
Sudden data ingestion	Pool usage spiked over hours or days before the OSD failure	`ceph df detail` and pool growth rate
OUT OSDs reduce total capacity	Nearfull appeared right after OSDs went OUT	`ceph osd dump` for in/out state, recompute real headroom
Forgotten recovery flags	`norecover` or `nobackfill` in flags, recovery rate zero	`ceph osd dump \| grep flags`

Quick checks

Read-only and safe to run during an incident:

# Cluster status and recovery line
ceph -s

# Health detail including BACKFILL_TOOFULL and OSD_NEARFULL
ceph health detail

# Per-OSD utilization, variance, and which are past thresholds
ceph osd df tree

# Stuck PGs - look for backfill_toofull in the state column
ceph pg dump_stuck unclean

# Pool-level usage and max_avail
ceph df detail

# Check for recovery flags that would stall healing
ceph osd dump | grep flags

# Current PG state summary and recovery activity
ceph pg stat

If ceph health detail shows PG_BACKFILL_FULL or lists PGs in backfill_toofull, and ceph osd df tree shows surviving OSDs at or above 90% utilized, the pattern is confirmed.

How to diagnose it

Confirm the thresholds are firing. Run ceph health detail and look for OSD_NEARFULL, OSD_BACKFILLFULL, and PG_BACKFILL_FULL. Note which OSDs are named in each check.
Check per-OSD utilization, not just the average. Run ceph osd df tree. The cluster average can be 80% while three OSDs sit at 92%. Those three OSDs are the ones refusing backfill. Watch the gap between average and maximum utilization.
Confirm recovery is stalled, not just slow. Sample ceph pg stat repeatedly and watch the recovering/backfilling PG count and recovery rate. Then run ceph pg dump_stuck unclean. If the degraded PG count is flat across samples and recovery bytes per second is near zero, recovery is stalled.
Identify which PGs are blocked and why. Look for backfill_toofull or recovery_toofull in the PG states. For a specific stuck PG, run ceph pg <pgid> query to see which target OSD is too full to accept data.
Check for administrative flags. Run ceph osd dump | grep flags. If norecover or nobackfill is set, recovery is intentionally stopped. If noout is set alongside down OSDs, they remain down+in and degraded PGs persist. Clear forgotten flags before assuming capacity is the only problem.
Recompute real headroom. OSDs marked OUT have their capacity subtracted from the total. The cluster’s effective utilization may be higher than ceph df suggests if OSDs were marked OUT recently.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`ceph_pg_backfill_toofull`	PGs blocked because target OSD is past backfillfull	Any nonzero value sustained, especially with degraded PGs
`ceph_pg_recovery_toofull`	PGs blocked because target OSD is past the full ratio	Any nonzero value means the cluster is at the hard stop
`ceph_num_objects_degraded`	Objects with fewer replicas than pool size	Flat or increasing trend instead of decreasing
`ceph_pool_recovering_bytes_per_sec`	Whether recovery is making progress	Near zero while degraded count is high
`ceph_health_detail{name="OSD_NEARFULL"}`	OSDs approaching capacity limits	Active, especially with individual OSDs above 85%
`ceph_health_detail{name="OSD_BACKFILLFULL"}`	OSDs refusing new backfill	Active means recovery will stall for PGs targeting these OSDs
Per-OSD utilization	Distribution imbalance	Max OSD utilization more than 10% above the average
`ceph_osd_flag_norecover` / `ceph_osd_flag_nobackfill`	Forgotten maintenance flags	Set while degraded PGs exist

Fixes

There is no fix that does not involve adding capacity, freeing capacity, or moving thresholds. Each has tradeoffs.

Reweight disproportionately full OSDs

If a few OSDs are at 92% while the rest sit at 80%, lowering the CRUSH weight of the fullest OSDs forces data to move elsewhere:

# Lower the reweight on a specific full OSD (0.0 to 1.0 scale)
ceph osd reweight <osd-id> 0.9

This only helps if the other OSDs have headroom. If the cluster average is already past backfillfull, reweighting just moves the problem. ceph osd reweight-by-utilization flattens distribution automatically, but watch the recovery impact on clients.

Free space

Delete non-critical data, old snapshots, or run RGW garbage collection. Snapshots consume space invisibly. Check ceph df detail for snapshot consumption by pool. For RGW deployments, forcing GC can reclaim space from deleted-but-not-collected objects.

This is the safest fix, but it depends on having data you can actually remove.

Add capacity

Add new OSDs. This is the only fix that increases the denominator, and it is the slowest if you do not have spare hardware. New OSDs receive backfill and reduce pressure on existing OSDs. Plan for the time it takes to provision, weight, and let the cluster rebalance onto them.

Temporarily raise the backfillfull ratio (dangerous, last resort)

If recovery is completely blocked and you cannot add or free space quickly, you can temporarily raise the backfillfull ratio to let Ceph push data onto OSDs it is currently refusing. This is risky. It lets OSDs fill closer to the full ratio, leaving less margin before writes stop entirely. Use this only as a bridge while you add capacity, and lower the ratio again as soon as recovery progresses.

# Temporarily raise the backfillfull ratio on a running cluster
ceph osd set-backfillfull-ratio 0.95

Treat this as buying time, not as fixing the problem. If an OSD hits the full ratio (0.95) while you are in this state, all writes to that OSD stop and the cluster enters HEALTH_ERR.

Prevention

The death spiral is preventable with capacity discipline and per-OSD monitoring.

Keep average utilization below 70%. Stay at least 20% below the backfillfull ratio (0.90) to leave room for recovery after a failure. That puts the operational ceiling around 70% average. Small clusters with few failure domains need even more headroom because losing one host removes a larger fraction of total capacity.
Monitor per-OSD utilization, not just cluster average. CRUSH does not guarantee even distribution. A cluster at 70% average with several OSDs at 85% is already at risk. Alert on maximum OSD utilization and on the variance between the fullest and average OSD.
Model failure scenarios explicitly. Ask whether the surviving OSDs can receive recovery data if the largest host fails. At 75% average with a host holding 10% of capacity, the remaining OSDs would need to absorb that data at roughly 83%, already past nearfull.
Do not dismiss nearfull warnings. Nearfull at 85% is the signal that a single failure will push surviving OSDs past backfillfull. Treat it as a capacity planning trigger, not background noise.
Watch recovery rate during any OSD failure. Degraded PGs with active recovery are expected. Degraded PGs with zero recovery rate mean the cluster cannot heal. That is the early signal of the death spiral, often before backfill_toofull even appears.
Track OSD IN/OUT state against total capacity. OSDs marked OUT reduce the cluster total. A cluster can tip into nearfull simply from OUT OSDs leaving the denominator, even without new writes.

How Netdata helps

Netdata’s Ceph collector surfaces the per-second signals that distinguish a normal recovery from a stalled one:

Per-OSD utilization shows CRUSH imbalance directly, not just the cluster average. The OSD at 92% while the average is 78% is the one that will block recovery.
ceph_pg_backfill_toofull and ceph_pg_recovery_toofull as gauges mark the exact moment recovery loses its target space. Correlate with degraded PG count to confirm the stall.
Degraded object count alongside recovery rate reveals whether healing is progressing. A flat degraded count with near-zero ceph_pool_recovering_bytes_per_sec is the death spiral signature.
ceph_health_detail labels (OSD_NEARFULL, OSD_BACKFILLFULL, PG_BACKFILL_FULL) map health checks to specific capacity conditions without parsing CLI output.
Recovery flag state (norecover, nobackfill, noout) on the same dashboard as PG states prevents a forgotten flag from consuming an hour of diagnosis.

Correlating per-OSD capacity with PG recovery state and recovery rate in one view shortens the path from “OSD failed” to knowing whether the cluster can still heal.