Ceph BlueStore allocator fragmentation: rising latency at moderate fullness

Write latency climbs on OSDs that are only moderately full. Capacity dashboards look healthy, the cluster sits well below nearfull, and SMART is clean. But a subset of OSDs shows rising commit and apply latency, slow ops begin accumulating, and the affected OSDs are the long-lived ones carrying heavy overwrite workloads. This is BlueStore allocator fragmentation: the free-space structure BlueStore walks on every allocation has become shredded into small, non-contiguous free extents, so each new write needs a longer search before a suitable run of blocks is found.

The signal is not capacity. The OSD may sit at 50% or 70% utilization and still be badly fragmented. Two OSDs at the same utilization can have very different allocation costs: utilization measures how much is used, fragmentation measures how the free space is shaped. A cluster that looks fine on ceph osd df can be quietly losing write throughput on a handful of aged OSDs.

The most reliable check is the BlueStore allocator fragmentation score, a single float in the range 0.0 to 1.0. A score near 0 means the free space is mostly one contiguous chunk. A score approaching 1 means the free space is shredded into tiny extents and each allocation walk is expensive. Since Ceph 19.2.3 (Squid) the cluster surfaces this as a BLUESTORE_FREE_FRAGMENTATION health warning when the score crosses 0.8. On older releases you have to ask each OSD directly through its admin socket.

This article covers how to identify fragmented OSDs, distinguish fragmentation from disk failure and BlueStore DB spillover, and apply remediation that actually works given that Ceph has no online BlueStore defragmenter.

What this means

BlueStore does not write through a filesystem. It manages raw block space itself, using an in-memory allocator (bitmap, stupid, avl, btree, or hybrid depending on version and configuration) to track which blocks are free. Every object write, every overwrite, and every RocksDB and BlueFS metadata write asks the allocator for a run of contiguous blocks. When free space is contiguous, that request is cheap. When free space is scattered, the allocator walks longer to find a fit, and on dense, heavily overwritten OSDs that walk becomes a measurable component of write latency.

Three things make this pattern hard to spot:

• It is invisible to capacity metrics. ceph osd df, ceph df, and the Prometheus capacity gauges report utilization, not fragmentation. Two OSDs at 60% can have scores of 0.2 and 0.85.
• It is per-OSD. Cluster-average latency looks fine while a small number of long-lived OSDs drag tail latency for the PGs they host.
• It looks like disk failure. Rising apply latency, slow ops, and occasional client-visible stalls are exactly the symptoms of a dying disk. SMART may be clean.

The fragmentation score is stable per-OSD and only changes as data is written, deleted, and rewritten. It does not spike transiently. A score above 0.9 is a structural condition that has built up over weeks or months, not something that appeared in the last hour.

Score interpretation, per the official Ceph health-checks documentation:

Operators on long-running clusters routinely see scores in the 0.8 to 0.9 band without measurable performance impact. Performance degradation and the risk of bluefs enospc assertions on the bitmap allocator typically appear above 0.9.

flowchart TD
  A["Write latency rising on subset of OSDs"] --> B{"Capacity below nearfull?"}
  B -- No --> G["Capacity issue, treat as backfill_toofull"]
  B -- Yes --> C["Check SMART and iostat"]
  C --> D{"Device healthy?"}
  D -- No --> E["Replace or reweight failing OSD"]
  D -- Yes --> F["Pull allocator score"]
  F --> H{"Score at or above 0.8?"}
  H -- No --> I["Investigate RocksDB or DB spillover"]
  H -- Yes --> J["Fragmentation: reweight-down or redeploy"]

Common causes

Quick checks

All read-only and safe to run during normal operation.

# Per-OSD fragmentation score (OSD must be running)
ceph daemon osd.<id> bluestore allocator score block

# Detailed free-extent dump for the same OSD
ceph daemon osd.<id> bluestore allocator dump block

# Cluster-wide view of any fragmentation health warnings
ceph health detail | grep -i BLUESTORE

# Offline check via ceph-bluestore-tool (requires the OSD stopped)
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<id> --allocator block free-score

# Offline free-extent dump
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<id> --allocator block free-dump

# Confirm capacity is not the real problem
ceph osd df tree

# Check the OSD's configured allocator type
ceph config get osd.<id> bluestore_allocator

# OSD apply and commit latency for context
ceph osd perf

# Slow ops and which OSDs they sit on
ceph health detail | grep -i slow

There is no cluster-wide aggregate. Script the score command across all OSD IDs and sort by score.

How to diagnose it

1. Identify candidate OSDs. Start with the OSDs that have been in the cluster longest and that show elevated apply or commit latency in ceph osd perf. Fragmentation accumulates with age and overwrite churn.
2. Confirm the OSD is not actually full. Check ceph osd df tree. If the OSD is at or above nearfull, capacity is the more pressing problem and you should treat it as a backfill-toofull issue, not fragmentation.
3. Pull the fragmentation score. Run ceph daemon osd.<id> bluestore allocator score block. Anything below 0.7 is background noise. 0.7 to 0.9 warrants tracking but is usually not the dominant latency contributor. A score at or above 0.9 on an OSD with rising latency is a strong signal.
4. Rule out device failure. Run iostat -xz 1 on the backing device and smartctl -A against it. Rising await with clean SMART is consistent with fragmentation. Reallocated or current pending sectors mean the device is failing and should be replaced regardless of fragmentation score.
5. Rule out BlueStore DB spillover. DB spillover produces a very similar latency pattern. Confirm with ceph daemon osd.<id> perf dump and inspect the bluefs section for nonzero slow_used_bytes. Any nonzero value means the DB has spilled and that is almost certainly the dominant cause, not allocator fragmentation.
6. Inspect the free-extent distribution. ceph daemon osd.<id> bluestore allocator dump block shows the shape of the free space. A long tail of tiny extents confirms the diagnosis quantitatively.
7. Correlate with slow ops. Pull ceph health detail and confirm slow ops land on the same OSDs that have high scores. If slow ops are scattered across OSDs with low scores, fragmentation is not the common cause.

Metrics and signals to monitor

The allocator score itself is not exported by the Prometheus manager module. Clusters that deploy the ceph-exporter daemon export the BlueStore perf counter fragmentation_micros (the allocator score multiplied by 1000) alongside their other perf counters; the exact allocator score still has to be collected out-of-band through the admin socket. The latency, slow-ops, and health-detail signals around it are exported normally.

Fixes

There is no online BlueStore defragmenter. The allocator state is rebuilt only when the OSD’s data is repopulated, which means every remediation is a variation on “move the data off and let it come back.” An upstream defrag-on-scrub proposal (ceph PR #57631) remains unmerged.

Reweight-down and let recovery repopulate

The standard remediation: temporarily reduce the OSD’s CRUSH weight so the balancer migrates data away, then restore the weight. When the OSD empties and refills, the allocator is rebuilt from a fresh state.

# WARNING: this triggers a full backfill of the OSD's PGs to peers.
# On a busy cluster it can roughly double write load on receiving OSDs.
# Do not run against many OSDs at once.

# Drop the OSD's weight to bleed data off
ceph osd reweight osd.<id> 0

# Wait for backfill to complete (monitor with ceph -s and ceph pg stat)

# Restore once the OSD is nearly empty
ceph osd reweight osd.<id> <original_weight>

Consider lowering osd_max_backfills and osd_recovery_max_active during the operation. Confirm target OSDs are not already at backfillfull, or recovery will stall with backfill_toofull PGs.

Destroy and redeploy the OSD

For OSDs that are deeply fragmented, or that also need a different min_alloc_size, the cleaner option is to destroy and redeploy. This is also the only way to change min_alloc_size, which is fixed at OSD creation.

Tradeoffs: full backfill of the OSD, the same I/O cost as reweight-down plus the operational overhead of OSD removal and re-add. Reserve this for cases where reweight-down is insufficient or where you are also rethinking allocator configuration.

Temporary allocator switch to recover a crashing OSD

If an OSD using the bitmap allocator is asserting with bluefs enospc at extreme fragmentation (documented in Ceph tracker #45519), operators have recovered the OSD by temporarily switching BlueStore and BlueFS to the stupid allocator, letting the OSD stabilize, then switching back. This is a recovery workaround, not a fix for fragmentation; the underlying allocator state still needs to be rebuilt via reweight or redeploy.

# WARNING: requires an OSD restart. Targeted at a single misbehaving OSD.
ceph config set osd.<id> bluestore_allocator stupid
ceph config set osd.<id> bluefs_allocator stupid
# Restart the OSD, verify it stays up, then plan reweight-down or redeploy

What does not work

• Restarting the OSD. The allocator state is persisted; a restart does not reflow free space. On legacy StupidAllocator clusters a restart could reset the in-memory iterator and briefly mask the symptom, but the fragmentation returns within hours.
• Tuning bluestore_warn_on_free_fragmentation. Setting the threshold to 1.0 disables the health warning but does not change allocator behavior. It only hides the signal.
• Adding capacity. Free space helps only because new OSDs have fresh allocators. Existing fragmented OSDs stay fragmented.

Prevention

• Track the score on long-lived OSDs. Sample bluestore allocator score block periodically for the oldest OSDs in the cluster. Trending the score is the only way to catch this before it shows up as slow ops.
• Prefer the hybrid allocator on Octopus and later. The default allocator changed from bitmap to hybrid in Nautilus 14.2.11, in part to improve fragmentation behavior. Older clusters still on stupid (the pre-Luminous default) are the worst case.
• Avoid extreme overwrite churn on a single OSD. Workloads that overwrite the same objects repeatedly (some database-on-RBD patterns, heavy journaling) accelerate fragmentation on the OSDs hosting them. Distribute such workloads where you can.
• Size min_alloc_size correctly at OSD creation. min_alloc_size is immutable after the OSD exists. For small-object workloads, the default 4KiB min_alloc_size is usually right (HDD moved from 64KiB to 4KiB in Pacific; SSDs have used 4KiB since Octopus and earlier). Changing it later requires destroying and redeploying the OSD.
• Keep bluefs_alloc_size and bluefs_shared_alloc_size aligned with min_alloc_size (4096). Mismatch has been observed to make available_from_bluestore report zero and to trigger BlueFS allocation failures on fragmented OSDs.
• Plan OSD redeploy cycles. On clusters with very long-lived OSDs and heavy overwrite, treat periodic reweight-down or redeploy as a planned maintenance activity, not an incident response.

How Netdata helps

• Per-OSD apply and commit latency (ceph_osd_apply_latency_ms, ceph_osd_commit_latency_ms) surfaces the outlier OSD dragging tail latency while cluster averages stay flat. Fragmentation is per-OSD, so per-OSD signals are the only thing that catches it early.
• Slow ops count (ceph_healthcheck_slow_ops) tells you when latency has crossed from “slow” into “stuck past osd_op_complaint_time.” Correlating slow ops with the specific OSDs that have high allocator scores is the fastest path to confirmation.
• BLUESTORE_FREE_FRAGMENTATION check label surfaces the cluster-level warning on Squid (19.2.3+) alongside the latency signals, so you see the structural cause and the symptom in one view.
• BlueFS slow-device usage (the same bluefs section you inspect for DB spillover) is the cleanest differential against allocator fragmentation, since the two produce nearly identical latency symptoms.
• Per-second granularity on OSD latency makes the slow, steady climb characteristic of fragmentation visible against the sharper spikes of compaction stalls or device failure.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn
• Ceph MON_DOWN: a monitor out of quorum and reduced redundancy
• Ceph monitor election storm: monitors that cannot hold a stable quorum
• Ceph monitor quorum lost: the cluster can no longer update its maps