Ceph recovery storm: rebuild traffic starving client I/O

A Ceph recovery storm occurs when the cluster’s self-healing machinery starves the workloads it is supposed to serve. After an OSD failure, host loss, or bulk OSD addition, recovery and backfill traffic floods the same disks, network links, and OSD CPU that client I/O depends on. Client latency climbs 10x to 100x, applications time out, and retries add more load on top of the recovery stream.

The signature is recognizable in seconds: ceph -s shows a high recovery line, many PGs sit in recovering or backfilling, and client latency is elevated cluster-wide rather than on one slow OSD. Health is usually HEALTH_WARN, not ERR, which makes it easy to underestimate.

The danger is not the recovery itself. Ceph is doing what it should: rebuilding redundancy after a topology change. The danger is the feedback loop. Recovery consumes the resources clients need, clients slow and retry, retries consume more resources, and surviving OSDs begin to miss heartbeat deadlines. Left unchecked, a recovery storm cascades into OSD flapping, peering loops, and in the worst case a capacity or availability death spiral.

What this means

A recovery storm is a resource contention failure, not a data integrity failure. Data is not lost; the cluster simply cannot serve clients within any reasonable SLA while rebuilding.

Three resources are saturated simultaneously:

• Disk I/O: Recovery reads objects from source OSDs and writes them to target OSDs. On HDD clusters especially, this consumes the sequential and random I/O budget that client writes and reads also depend on.
• Network bandwidth: Replication, recovery, and backfill traffic cross the cluster network. If public and cluster networks share a NIC or VLAN, recovery traffic directly steals bandwidth from clients.
• OSD CPU: Each OSD runs an async messenger, a sharded op queue, BlueStore I/O threads, and RocksDB work. Recovery ops hit the same threads that serve clients, and the OSD must also peer the PGs being recovered.

The recovery machinery has throttles, but the defaults are aggressive enough that on a busy cluster a single host failure can saturate everything. On Quincy and later, the mClock scheduler changes how throttling behaves, which is the single biggest source of operator confusion during a storm.

flowchart TD
    A[OSD failure or bulk OSD add] --> B[PGs go degraded / peering]
    B --> C[Recovery + backfill start]
    C --> D[Disk I/O, cluster net, OSD CPU saturate]
    D --> E[Client latency spikes 10-100x]
    E --> F[Clients time out and retry]
    F --> D
    D --> G[Heartbeat grace exceeded]
    G --> H[More OSDs marked down]
    H --> B

Common causes

Quick checks

These are safe read-only commands. Run them in order; together they tell you whether you are in a recovery storm and how bad it is.

# Cluster summary: look for recovery io line and degraded/backfilling PG counts
ceph -s

# Per-PG state totals: high recovering/backfilling/degraded confirms storm
ceph pg stat

# Down and failed OSDs, plus any flapping
ceph health detail

# Per-pool recovery rate and client io rate, side by side
ceph osd pool stats

# Current recovery throttle settings (effective values)
ceph config dump | grep -E 'osd_max_backfills|osd_recovery_max_active|osd_recovery_sleep|osd_op_queue|osd_mclock'

# Cluster flags: norecover / nobackfill / noout intentionally set?
ceph osd dump | grep flags

# Per-OSD commit and apply latency: look for systemic elevation
ceph osd perf

If ceph -s shows a non-trivial recovery line, ceph pg stat shows many PGs in recovering or backfilling, and ceph osd perf shows elevated latency across many OSDs at once, you are in a recovery storm.

How to diagnose it

1. Confirm the recovery is the load source. Correlate the recovery rate from ceph osd pool stats with client latency. If both rise together and fall together when you throttle, recovery is the cause. If client latency is elevated but the recovery line is zero, you have a different problem (see the related guides on blocked ops and slow OSDs).
2. Identify the trigger. Check ceph osd tree for down OSDs and ceph health detail for OSD flapping. If no OSDs are down, ask whether someone added OSDs, changed CRUSH weights, or modified a pool’s replication factor. Bulk changes show up as remapped PGs.
3. Check whether recovery is actually progressing. Degraded PG count should be monotonically decreasing. Run ceph pg stat twice, a minute apart. If degraded count is flat or rising while recovery rate is high, recovery is spinning (often due to flapping) rather than healing.
4. Check the throttle stack. On Quincy and later, mClock is the default OSD scheduler for BlueStore, and the old sleep-based throttles are ignored. Confirm with ceph config dump | grep osd_op_queue. If you are on mClock and you tune osd_recovery_sleep and nothing changes, that is why.
5. Check for secondary failures. Recovery storms often trigger cascading OSD failures as surviving OSDs miss heartbeats. Watch the down OSD count and the slow ops count. If either is climbing, the storm is escalating and you need to throttle immediately, not investigate further.

Metrics and signals to monitor

Fixes

The goal is not to stop recovery; the goal is to cap its share of cluster resources so client I/O recovers while healing continues in the background. Every throttle you tighten extends the redundancy exposure window. That tradeoff is the whole decision.

Throttle recovery (pre-Quincy, or Quincy+ with mClock override)

On clusters using the classic WPQ scheduler, recovery throttling is done with sleep and concurrency parameters.

# Reduce concurrent recovery and backfill streams per OSD
ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'

# Force recovery to yield between ops; 0.5s is aggressive, lower if needed
ceph tell 'osd.*' injectargs '--osd_recovery_sleep 0.5'

# Pause scrub and deep-scrub to free I/O for client traffic
ceph osd set noscrub
ceph osd set nodeep-scrub

osd_max_backfills defaults to 1 (no per-device-class variant). osd_recovery_max_active defaults to 0, which resolves to the per-device-class limits: 3 concurrent recoveries on HDD OSDs (osd_recovery_max_active_hdd) and 10 on SSD OSDs (osd_recovery_max_active_ssd).

During a storm, setting both to 1 plus a non-zero osd_recovery_sleep is a reasonable starting point. Watch client latency and the slow ops count after each change; the effect should be visible within a minute.

Remember to unset noscrub and nodeep-scrub once the storm is over. Forgotten scrub flags are a common cause of silent data integrity debt.

Use mClock profiles (Quincy and later)

On Quincy and later, mClock is the default scheduler for BlueStore OSDs. The old sleep-based parameters are ignored when mClock is active, and osd_max_backfills and osd_recovery_max_active are managed by the scheduler. Tuning them with injectargs will appear to work and then be reset.

Switch profiles instead:

# Favor client ops during recovery; recovery slows but clients recover first
ceph config set osd osd_mclock_profile high_client_ops

The profile change takes effect without an OSD restart. The balanced profile is the default; high_client_ops is the right choice when a recovery storm is starving clients. Move back to balanced once the storm is over.

If you need finer control than the built-in profiles provide, set osd_mclock_override_recovery_settings=true (verified parameter name; default false) and then tune osd_max_backfills and osd_recovery_max_active_hdd/osd_recovery_max_active_ssd directly. Without this override, attempts to modify those limits under mclock are reset to their defaults.

This is advanced territory; profile switching covers most operational needs.

The mclock custom profile is a valid enumeration value on Reef and Squid, but the built-in profiles (balanced, high_client_ops, high_recovery_ops) are the supported path; the official documentation recommends testing a custom profile before relying on it in production.

norecover and nobackfill as a last resort

If throttling is not enough and client impact is critical, you can pause recovery entirely.

# Pause recovery and backfill; understand the tradeoff before doing this
ceph osd set norecover
ceph osd set nobackfill

This is a deliberate, understood last resort. The moment you set these flags, the cluster stops healing. Degraded PGs stay degraded. If another OSD fails before you unset the flags, you may lose data. Set them, fix whatever is driving the storm (usually a cascading failure or a network saturation), and unset them as soon as client I/O stabilizes.

# Resume recovery as soon as the immediate crisis is past
ceph osd unset nobackfill
ceph osd unset norecover

If you set noout to prevent further CRUSH remapping during the storm, track it. The noout trap is one of the most common preventable Ceph outages.

Do not add capacity mid-storm

Adding OSDs during a recovery storm increases PG remapping and makes the storm worse before it gets better. Bring up any down OSDs, stabilize what you have, then plan capacity changes once health is back to HEALTH_OK.

Prevention

• Separate public and cluster networks. Set cluster_network in the [global] section of ceph.conf so replication, recovery, and heartbeat traffic use a dedicated subnet. This single change prevents most recovery-storm-induced client impact on clusters with adequate disk headroom.
• Size the cluster network for recovery bursts. A single OSD failure can consume 30-50% of a 10GbE link for recovery alone. Plan for 2x peak client throughput on the cluster network.
• Tune recovery throttles ahead of time. Decide whether you want minimum exposure window (aggressive recovery) or minimum client impact (conservative recovery) and configure the cluster accordingly. Revisit the settings whenever cluster size changes significantly.
• On Quincy+, choose the right mClock profile proactively. balanced is fine for most clusters, but high_client_ops should be your standard response during any topology-driven recovery event.
• Add OSDs in small batches. Bulk OSD additions look like a recovery storm to the cluster. Add a host or a rack at a time, let the cluster settle, and continue.
• Keep capacity headroom. Recovery requires spare space. A cluster near backfillfull (default 90%) cannot heal, which converts a recovery storm into a capacity death spiral.

How Netdata helps

• The Ceph collector surfaces ceph_pool_recovering_bytes_per_sec, ceph_pg_recovering, ceph_pg_backfilling, and ceph_num_objects_degraded per second, so you can watch recovery progress and client impact on the same timeline instead of polling ceph -s manually.
• Per-second ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms let you see cluster-wide latency elevation as it develops, and identify whether the storm is systemic or concentrated on a few OSDs.
• ceph_healthcheck_slow_ops and ceph_health_detail labels correlate slow ops with the specific health checks firing, which shortens the gap between “clients are slow” and “this is a recovery storm.”
• OSD flag metrics (ceph_osd_flag_norecover, ceph_osd_flag_nobackfill, ceph_osd_flag_noout) show whether recovery is paused and whether noout is still set from a previous maintenance window.
• Host-level network interface metrics on the same dashboards let you confirm whether the cluster network is saturated and whether public and cluster traffic are actually separated.
• Anomaly detection on recovery rate and OSD latency helps catch the storm before client latency has climbed the full 10-100x.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn
• Ceph MON_DOWN: a monitor out of quorum and reduced redundancy