Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device

A small subset of OSDs shows periodic commit-latency spikes and slow ops while the rest of the cluster looks healthy. Capacity metrics are normal. SMART is clean. ceph -s reports HEALTH_WARN, and ceph health detail returns something like:

BLUEFS_SPILLOVER
    3 OSDs spilled over ~18 GiB metadata from 'db' device
    (e.g. osd.12 spilled over 6.1 GiB metadata from 'db' device)

BlueStore’s RocksDB metadata has outgrown its dedicated fast DB partition (SSD/NVMe) and is spilling onto the slow HDD data partition. Compaction that took milliseconds on flash now takes seconds on spinning disk. Between compaction cycles the OSD looks fine; during compaction it stalls. This is a cliff edge, not gradual degradation: the moment slow_used_bytes goes nonzero, latency steps up by one to two orders of magnitude on the affected OSDs.

The trap is that spillover is invisible in normal capacity views. ceph osd df does not surface DB partition usage by default. Cluster-wide latency averages mask the few affected OSDs. The only reliable signals are BLUEFS_SPILLOVER in ceph health detail, per-OSD commit latency, and the bluefs perf counters themselves.

What this means

BlueStore stores object metadata, omap data, and allocator state in RocksDB. RocksDB lives in BlueFS, a tiny log-structured filesystem that can span up to three locations: a fast DB device (SSD/NVMe), an optional WAL device, and the main slow data device. When the DB partition fills, BlueFS does not stop writing. It extends RocksDB SST files onto the slow device. The BLUEFS_SPILLOVER health check fires whenever slow_used_bytes > 0 in the bluefs perf dump.

The performance impact is severe and discontinuous. On HDD, random RocksDB reads that took tens of microseconds on NVMe now take milliseconds, and compaction jobs that completed in milliseconds stretch into seconds. Because BlueStore serializes writes through RocksDB via the kv_sync thread, every write op on the affected OSD pays the new latency during compaction. The result is periodic latency spikes on specific OSDs, perfectly correlated with RocksDB compaction events, on a cluster that otherwise looks healthy.

flowchart TD
    A[DB partition SSD/NVMe fills] --> B[BlueFS extends RocksDB to slow device]
    B --> C[slow_used_bytes greater than 0]
    C --> D[BLUEFS_SPILLOVER health check]
    C --> E[Compaction reads hit HDD]
    E --> F[kv_sync thread stalls]
    F --> G[Periodic commit-latency spikes on specific OSDs]

Spillover can persist even when the DB device reports free space. RocksDB levels are sized by the compaction strategy. If a single level is larger than the DB partition, older Ceph releases allocated the entire level to the slow device, wasting fast capacity. Granular allocation via bluestore_volume_selection_policy was added in Nautilus 14.2.10 and Octopus 15.2.0; the use_some_extra policy uses fast space for partial RocksDB levels and became the default in Nautilus 14.2.12 (Octopus 15.2.x still defaulted to rocksdb_original). Confirm the policy in effect before assuming a too-small DB partition is the only cause.

Common causes

Quick checks

These are read-only and safe to run on a production cluster.

# List which OSDs have spillover
ceph health detail | grep -A2 BLUEFS_SPILLOVER

# Per-OSD bluefs usage (slow_used_bytes is the cliff signal)
ceph daemon osd.<id> perf dump | jq '.bluefs | {db_used_bytes, db_total_bytes, slow_used_bytes, slow_total_bytes, wal_used_bytes}'

# BlueFS volume selector matrix (SLOW row should be empty)
ceph tell osd.<id> bluefs stats

# Per-OSD commit latency outliers (spillover signature)
# ceph osd perf reports commit_latency and apply_latency columns;
# sort by the commit_latency column to surface spillover OSDs
ceph osd perf | sort -k2 -n

# RocksDB level statistics
ceph daemon osd.<id> perf dump | jq '.rocksdb | keys'

# Volume selection policy in effect
ceph daemon osd.<id> config get bluestore_volume_selection_policy

# Related health signals
ceph health detail | grep -E 'BLUEFS_SPILLOVER|LARGE_OMAP_OBJECTS|OSD_NEARFULL'

# Confirm whether spillover warning has been suppressed
ceph config get osd bluestore_warn_on_bluefs_spillover

If bluestore_warn_on_bluefs_spillover returns false, the warning has been muted. The spillover is still happening; you have just hidden the signal. Re-enable before diagnosis.

How to diagnose it

1. Confirm the cluster-level signal. ceph health detail lists each affected OSD with the spill volume. Note the OSDs: spillover is a per-OSD condition, not cluster-wide.

2. For each affected OSD, confirm with bluefs stats. Run ceph daemon osd.<id> perf dump | jq '.bluefs' and check slow_used_bytes. Any nonzero value is the cliff edge. Cross-check with ceph tell osd.<id> bluefs stats to see the volume selector matrix; the SLOW row shows what is on the slow device.

3. Correlate with commit latency. Run ceph osd perf and sort by commit latency. Spillover OSDs sit at the top of the list with periodic spikes that match RocksDB compaction. Sustained high commit latency on a few OSDs with normal apply latency is the classic signature.

4. Check DB partition sizing. From ceph daemon osd.<id> bluefs stats, compare db_total_bytes to the data device size. The Ceph documentation recommends the DB partition be between 1% and 4% of the data device for typical workloads (RBD usually needs no more than 1-2%), and at least 4% for RGW or workloads with heavy omap usage. Undersized DB partitions are the most common root cause.

5. Check for omap growth. If spillover is concentrated on OSDs hosting RGW bucket index pools or CephFS metadata pools, omap data is likely the driver. ceph health detail | grep LARGE_OMAP_OBJECTS surfaces oversized index objects. radosgw-admin bucket stats --bucket=<bucket> shows shard counts via the num_shards field in the bucket metadata. Each undersharded bucket with millions of objects pushes omap data into RocksDB.

6. Check the version and policy. Confirm the bluestore_volume_selection_policy in effect: use_some_extra has been the default since Nautilus 14.2.12 and on all current releases, but Octopus 15.2.x shipped with rocksdb_original as the default. Older releases or clusters that pre-date the fix may be spilling unnecessarily because RocksDB allocated entire levels to the slow device. On Reef 18.2.0+, changed bluestore_rocksdb_options defaults may have triggered spillover that did not exist on Pacific.

7. Check whether compaction will help. ceph tell osd.<id> compact can reduce spillover, but it may not eliminate it. If the DB partition is genuinely undersized or RocksDB levels cannot fit, compaction alone is not a fix. Multiple compactions or migration is required.

Metrics and signals to monitor

Per-OSD granularity is essential. Cluster-wide commit latency averages will mask the few affected OSDs. The signature is periodic latency spikes on specific OSDs, perfectly correlated with RocksDB compaction events, invisible in capacity metrics.

Fixes

Plan before touching anything

All fixes that move data off the slow device require OSD downtime. Plan a maintenance window. Reweight the affected OSD down first (ceph osd reweight osd.<id> 0) so the cluster rebalances off it before you stop the daemon, or set noout if you want to preserve placement and accept degraded PGs during the work. Reweighting to zero first avoids double I/O from recovery happening concurrently with the migration.

Move spilled data back with bluefs-bdev-migrate

Expanding the DB partition alone does not move spilled data back. The required sequence on an LVM-deployed OSD:

# 1. Stop the OSD
systemctl stop ceph-osd@<id>

# 2. Expand the DB LV
lvextend -L +<size>G /dev/<vg>/<db-lv>

# 3. Tell BlueFS about the new space
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<id> bluefs-bdev-expand --dev-target db

# 4. Migrate spilled data back to the DB device
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<id> bluefs-bdev-migrate --devs-source slow --dev-target db

# 5. Restart and verify
systemctl start ceph-osd@<id>
ceph daemon osd.<id> perf dump | jq '.bluefs.slow_used_bytes'

slow_used_bytes should read 0 after migration and the BLUEFS_SPILLOVER health check should clear on the next poll.

If you used bluefs-bdev-new-db to add a DB device to an OSD that did not have one, you must still run bluefs-bdev-migrate afterward. A residual 64-128 KiB of metadata remains on the slow device otherwise, producing a permanent BLUEFS_SPILLOVER warning that no amount of compaction will clear.

Try compaction first when DB headroom exists

If the DB device has headroom and the spillover is recent, compaction can shrink RocksDB and may eliminate the spill without downtime:

# Trigger RocksDB compaction on a single OSD (online)
ceph tell osd.<id> compact

Watch slow_used_bytes after compaction completes. If it does not drop to zero, the DB partition is undersized for the level structure and migration is required. Do not loop compaction hoping it eventually works; that just generates I/O.

Reduce the DB working set

When the DB device cannot be enlarged, reduce the metadata footprint:

• Enable LZ4 compression on RocksDB. On Squid (19.x; shipped since Squid 19.0.0) this is the default (compression=kLZ4Compression in bluestore_rocksdb_options). Reef and earlier default to kNoCompression; set it manually on those releases. Existing SST files are not recompressed until compacted, so run a full compaction after enabling to realize the savings.
• Reshard undersharded RGW bucket indexes. radosgw-admin bucket reshard reduces per-shard omap size and the corresponding RocksDB pressure. Resharding is online but stresses the cluster; schedule during low load and read the documentation for bucket instance idempotency before running it.
• Reweight the affected OSD down to reduce its object count and migrate omap data away over time.

Suppress the warning only with documented intent

ceph config set osd bluestore_warn_on_bluefs_spillover false silences the health check. It does not fix the spillover. Use it only as a temporary measure while planning migration, and track that it is set. Long-term muting is how teams end up chasing “mystery latency” months later.

Prevention

• Size DB partitions correctly at deploy time. Between 1% and 4% of the data device for typical workloads (RBD usually needs no more than 1-2%), 4% or more for RGW, EC pools, or workloads known to generate heavy omap. Larger is always safer; the DB device is cheap insurance against a 100x latency cliff.
• Monitor bluefs.slow_used_bytes per OSD, not just the health check. The health check fires only after spillover occurs. Trending db_used_bytes / db_total_bytes lets you migrate before the cliff.
• Track RocksDB level growth on OSDs hosting RGW index pools. OMAP growth is the most common driver of unexpected DB pressure.
• Re-enable the warning if it has been muted. A muted BLUEFS_SPILLOVER is a silent performance cliff.
• Test migrations in a staging cluster. bluefs-bdev-migrate is safe but requires OSD downtime and a correct device specification. Errors here are recoverable but expensive.
• On Reef upgrades, watch for new spillover on clusters that were clean on Pacific. Changed RocksDB defaults can push borderline DB partitions over the edge. Enabling LZ4 compression and running a full compaction typically resolves it.

How Netdata helps

• Per-second commit and apply latency per OSD surfaces the spillover signature (commit latency outliers with normal apply latency) without scraping ceph osd perf by hand.
• The health check dimension with the name label brings BLUEFS_SPILLOVER alongside related checks like LARGE_OMAP_OBJECTS, so you can correlate spillover with omap-driven root causes in one view.
• ML anomaly detection on per-OSD commit latency flags the periodic spikes that precede operator awareness, before they cascade into slow ops.
• Per-OSD granularity (not cluster averages) is what surfaces the few affected OSDs against the healthy majority.
• Correlating commit latency with recovery rate, slow ops, and OSD up/down state on a single timeline compresses diagnosis from “mystery latency” to “DB spillover on osd.12” in minutes rather than hours.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn
• Ceph MON_DOWN: a monitor out of quorum and reduced redundancy
• Ceph monitor election storm: monitors that cannot hold a stable quorum
• Ceph monitor quorum lost: the cluster can no longer update its maps
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert