Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine

Cluster writes are blocked. ceph status shows HEALTH_ERR with OSD_FULL active. But ceph df reports the cluster at 65% utilized. Both are correct.

The Ceph full ratio (default 0.95) is enforced per-OSD, not cluster-wide. When any single OSD crosses that threshold, Ceph refuses writes for every PG that OSD serves. Because CRUSH distributes PGs across OSDs, one full OSD can block writes to a large fraction of PGs even when ninety-nine other OSDs have ample free space.

What this means

Three thresholds matter, all enforced per-OSD:

The cluster-wide utilization metric (ceph_cluster_total_used_raw_bytes / ceph_cluster_total_bytes) is an average. A single OSD at 96% in a 100-OSD cluster averaging 60% produces a cluster metric that looks healthy. The OSD_FULL health check fires on the individual OSD, and the resulting write block is cluster-wide.

CRUSH places PGs on OSDs proportional to OSD weights. It does not guarantee byte-level evenness. If weights do not match actual disk capacities, or if the balancer is off or not converging, some OSDs fill faster. The first OSD to hit the full ratio stops writes for every PG it serves as primary or replica.

The metric ceph_cluster_by_class_total_bytes (label device_class) shows capacity per class. If one device class is significantly fuller than another, the imbalance is structural, not random.

Common causes

The last row is the most subtle. The built-in balancer (in upmap mode on modern releases) optimizes by PG shard count, not by actual data bytes. An OSD with the “correct” number of PGs can still be fuller than peers if its PGs happen to contain more data. This is a documented limitation: the balancer cannot solve byte-level imbalance directly.

Quick checks

# Which OSDs are full, nearfull, or backfill_toofull
ceph health detail | grep -iE 'full|nearfull|toofull'

# Per-OSD utilization, hierarchical view
ceph osd df tree

# Balancer status
ceph balancer status

# Recovery flags that block rebalancing (nobackfill, norecover, norebalance)
ceph osd dump | grep flags

# Stuck PGs that may be blocked by full target OSDs
ceph pg dump_stuck unclean

The %USE column in ceph osd df tree shows per-OSD utilization. The VAR column shows each OSD’s deviation from the device-class average. The summary line reports STDDEV across the cluster.

The STDDEV threshold is vendor-documented Red Hat guidance (RHCS 6/7/8 Troubleshooting Guide): if ceph osd df ssd or ceph osd df nvme reports a standard deviation greater than 2.0 for a device class, the balancer may not be enabled or functioning correctly.

How to diagnose it

flowchart TD
    A[Writes blocked, OSD_FULL health check] --> B[Run ceph osd df tree]
    B --> C{One or few OSDs at 95%+?}
    C -->|No| D[Cluster-wide capacity issue]
    C -->|Yes| E[Fullness imbalance confirmed]
    E --> F{STDDEV or VAR high?}
    F -->|Yes| G[Check balancer status]
    F -->|No| H[Check CRUSH weights vs disk sizes]
    G --> I{Balancer on and mode upmap?}
    I -->|No| J[Enable balancer]
    I -->|Yes| K[Check for override reweights]
    K --> L{REWEIGHT not 1.0 on any OSD?}
    L -->|Yes| M[Reset override reweights]
    L -->|No| N[Consider PG count vs bytes limitation]

1. Confirm the imbalance. Run ceph osd df tree. Identify the OSDs above 85% (nearfull), 90% (backfillfull), or 95% (full). A VAR above 1.5 means that OSD is roughly 50% fuller than the device-class average.

2. Check the balancer. Run ceph balancer status. Confirm it is active and in upmap mode. If it is on but the cluster was recently degraded (OSDs down), note that the balancer does not run while the cluster is degraded. After recovery completes, it should begin correcting the imbalance.

3. Check for override reweights. In ceph osd df tree, look at the REWEIGHT column. If any value is not 1.0000, someone set a manual override on that OSD, likely during a past incident. Override reweights conflict with the upmap balancer.

4. Check CRUSH weights against disk sizes. Compare the SIZE column with the WEIGHT column. CRUSH weight should be proportional to disk size. If a 4TB OSD and an 8TB OSD both show weight 1.0, the 4TB OSD will fill roughly twice as fast.

5. Check for backfill_toofull PGs. Run ceph pg dump_stuck unclean. PGs stuck in backfill_toofull mean recovery is blocked because the target OSD is above the backfillfull ratio (90%). Even after you free space on the full OSD, recovery may not proceed until target OSDs drop below 90%.

Metrics and signals to monitor

The cluster-wide utilization metric is necessary but not sufficient. If your monitoring pipeline only collects ceph_cluster_total_used_raw_bytes, you will not see this problem coming.

Fixes

Immediate: unblock writes

If writes are blocked right now, you need to get the full OSD below 95% quickly.

• Reweight the full OSD down to reduce new CRUSH placements on it. This triggers recovery to move data off gradually:

# Temporary override reweight; range 0.0 to 1.0
# Triggers PG migration away from this OSD; expect recovery traffic
ceph osd reweight <osd-id> 0.80

Existing data does not move instantly, but the OSD stops accepting new placements and recovery gradually moves data off.

• Delete data or snapshots. Run ceph df detail to find pools with high snapshot usage. Deleting old RBD or CephFS snapshots can free significant space immediately.

• Force RGW garbage collection if you use RGW:

radosgw-admin gc process

Deleted RGW objects are queued for GC, not removed immediately. A backlogged GC queue consumes space invisibly.

• As a last resort, temporarily raise the full ratio. This unblocks writes immediately but risks the OSD filling completely:

# WARNING: if the OSD reaches 100%, it will crash and may not recover cleanly
# Lower it back as soon as writes are unblocked
ceph osd set-full-ratio 0.97

Short-term: correct the imbalance

Once writes are unblocked, fix the structural imbalance.

1. Reset any override reweights to 1.0:

ceph osd reweight <osd-id> 1.0

Override values fight the balancer.

2. Ensure the balancer is on and in upmap mode:

ceph balancer on
ceph balancer mode upmap

3. Wait. The upmap balancer moves PGs incrementally. It will not fix a large imbalance instantly. Monitor ceph osd df tree over hours. VAR values should converge toward 1.0.

4. If the balancer is not converging and the cluster has mixed disk sizes, check whether upmap_max_deviation (default 5 PG shards, minimum 1) is too coarse for your environment. Lowering it below the default is not something the official documentation recommends; operator reports of excessive PG churn from low values are version-specific, so test any reduction on a non-production cluster first.

Structural: fix CRUSH weights

If the root cause is mismatched CRUSH weights (typically from mixed disk sizes), set CRUSH weights proportional to disk capacity.

CRUSH weights are in units of tebibytes. A 4TB disk should have a weight of approximately 4.0. An 8TB disk should have a weight of approximately 8.0. If all disks were assigned weight 1.0 regardless of size, small disks fill first.

# Check current CRUSH weights and hierarchy
ceph osd tree

# Set CRUSH weight for an OSD (weight is in TiB units)
ceph osd crush reweight osd.<id> <weight>

This changes the placement weight, which triggers PG migration. Do this gradually on production clusters. Changing CRUSH weights on many OSDs simultaneously causes a large recovery burst that competes with client I/O.

What not to do

Do not use ceph osd reweight-by-utilization as a substitute for the upmap balancer: it sets override reweights, and the Ceph docs state that any override reweight value conflicts with the balancer (with the balancer in use, all override reweights should be 1.0000). If you have been using it, stop, reset all override reweights to 1.0, and enable the upmap balancer.

Prevention

• Monitor per-OSD utilization, not just cluster-wide averages. The ceph_cluster_total_used_raw_bytes metric will not warn you. The per-OSD view from ceph osd df tree is the only reliable leading signal.

• Keep the balancer on in upmap mode at all times. The balancer does not run when the cluster is degraded. After any degradation event (OSD failures, maintenance), verify that the balancer resumes and the imbalance corrects.

• Match CRUSH weights to disk capacity at deployment time. Mixing disk sizes without adjusting weights is the most common structural cause.

• Clear override reweights after incidents. Any ceph osd reweight command sets a temporary override that persists until reset. If you reweight an OSD down during an incident and forget to reset it, the balancer cannot function correctly on that OSD.

• Track the VAR column trend. A gradually increasing VAR on specific OSDs indicates drift the balancer is not correcting. Investigate before the OSD crosses nearfull.

How Netdata helps

• Alerting on ceph_health_detail{name="OSD_NEARFULL"} catches the 85% per-OSD condition before the cluster-wide write block at 95%. OSD_FULL alerting is necessary but already too late to prevent the outage.

• Correlating ceph_pg_backfill_toofull with OSD_NEARFULL or OSD_FULL health checks distinguishes a capacity cliff from a recovery stall. If backfill_toofull is non-zero while OSD_FULL is active, recovery is also blocked, not just writes.

• Per-device-class metrics (ceph_cluster_by_class_total_bytes) surface structural imbalance between HDD and SSD tiers that the cluster-wide average hides.

• The ceph_osd_weight metric exposes CRUSH weight per OSD. Correlating this with disk capacity reveals weight mismatches that cause uneven fill rates on mixed-size clusters.

• Cluster-wide capacity metrics (ceph_cluster_total_used_raw_bytes, ceph_cluster_total_bytes) provide context. When per-OSD signals fire but the cluster-wide metric looks normal, the diagnosis is fullness imbalance.

Per-OSD utilization as a time-series metric depends on the Netdata Ceph collector revision (out of scope for this factual review); ceph osd df tree remains the authoritative per-OSD source of truth.

Related guides

• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph nearfull: the 85% warning that decides whether the cluster can heal
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• Ceph health detail: mapping ceph_health_detail checks to a cause
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert