Ceph deep scrub performance impact: I/O saturation that mimics an incident

A Ceph cluster suddenly shows elevated apply latency across many OSDs. Slow ops tick up. Client write latency degrades. The dashboard looks like the start of a real incident. But cluster health is HEALTH_OK and all PGs are active+clean. Before you start chasing a failing disk or a network partition, check whether a deep scrub is running.

Deep scrub reads and checksums every byte of every object in a PG. On HDD-backed OSDs, this is a sequential read pass across the entire device. With default scheduling, it happens to every PG roughly once a week. If the schedule concentrates scrubs onto the same OSDs at the same time, or if your scrub window is narrow, deep scrub will saturate disk I/O and produce symptoms that look identical to a real performance incident.

The distinguishing signal is PG state. A PG in active+clean+scrubbing+deep is doing exactly what it should. The latency spike is the cost of verification. The wrong response is to treat this as a fault, suppress the scrub with nodeep-scrub, and move on. That trades immediate relief for verification debt: corruption that would have been caught this week goes undetected until the next scrub.

What this means

Deep scrub is Ceph’s integrity check. A light scrub compares object metadata across replicas. A deep scrub reads the full contents of every object, checksums them, and compares results across replicas. BlueStore already checksums every block on client reads, so corruption on frequently accessed data is caught at access time. Deep scrub is the only background mechanism that catches corruption on cold data and detects divergence between replicas that single-copy checksums miss.

The cost is I/O. A deep scrub of a PG reads every object from the primary and each replica. On an HDD OSD, this is largely sequential read traffic, but it competes with client I/O and recovery for the same device. On a fully populated HDD OSD with several terabytes of data, a single deep scrub pass can take tens of minutes to hours. If multiple PGs on the same OSD are scheduled for deep scrub in the same window, the device is saturated for that entire period.

The default deep scrub interval is 7 days (osd_deep_scrub_interval = 604800 seconds). With osd_scrub_interval_randomize_ratio at the default 0.5, scrubs are spread across a randomized window, but on clusters with many PGs per OSD, several will still land in the same hour. osd_deep_scrub_randomize_ratio (default 0.15) causes some PGs to be deep-scrubbed before their interval expires, increasing total scrub I/O above the naive “one pass per week” estimate.

The latency symptoms are real. Clients do experience slower I/O. But the root cause is scheduled maintenance work, not a failing component.

flowchart TD
    A[Apply latency spikes across OSDs] --> B{PGs show scrubbing+deep?}
    B -- Yes --> C{Spike correlates with scrub schedule?}
    B -- No --> D[Investigate as real fault]
    C -- Yes --> E[Scrub-induced I/O saturation]
    C -- No --> D
    E --> F[Tune schedule, do not suppress indefinitely]
    F --> G[Verify deep scrubs complete within interval]

Common causes

Quick checks

These are safe, read-only commands. Run them before changing any configuration.

# Check daemon responsiveness and overall cluster health
ceph health detail

# See which PGs are currently scrubbing or deep-scrubbing
# Note: ceph pg dump can be slow on clusters with many PGs
ceph pg dump | grep -E 'scrubbing|deep'

# Check whether scrub-suppression flags are set
ceph osd dump | grep flags

# Check the configured scrub window
ceph config get osd osd_scrub_begin_hour
ceph config get osd osd_scrub_end_hour

# Check how many concurrent scrubs an OSD will run
ceph config get osd osd_max_scrubs

# Check the scrub load threshold (default 0.5)
ceph config get osd osd_scrub_load_threshold

# Check per-OSD apply and commit latency to confirm the spike pattern
ceph osd perf

# Check the active mClock profile (Quincy and later)
ceph config get osd osd_mclock_profile

# Check the last deep-scrub timestamp for a sample of PGs
ceph pg dump -f json-pretty | jq '.pg_stats[] | {pgid: .pgid, last_deep_scrub: .last_deep_scrub_stamp}' | head -40

How to diagnose it

1. Confirm the latency spike is correlated with scrub state. Look at per-OSD apply latency via ceph osd perf or the ceph_osd_apply_latency_ms metric. Identify which OSDs are affected. Then check whether those same OSDs have PGs in active+clean+scrubbing+deep. If the sets match, scrub is the likely cause.

2. Rule out concurrent recovery. Recovery competes for the same I/O. Check ceph pg stat for recovering or backfilling PGs, and check ceph osd dump | grep flags for norecover or nobackfill. If recovery is also active, the latency is compounded. The default osd_scrub_during_recovery is false, so new scrubs should not start during recovery, but a scrub that started before recovery began will continue to completion.

3. Check whether the spike is periodic. If you have per-second metrics, overlay apply latency with the count of PGs in scrubbing state. A scrub-induced spike rises and falls with the scrub count. A hardware-induced spike does not. This is the single most reliable discriminator.

4. Check for overdue scrubs catching up. If nodeep-scrub was set recently, or if the cluster was under heavy load that prevented scrubs from running, the scheduler will aggressively schedule deep scrubs once conditions allow. Look at ceph health detail for PG_NOT_DEEP_SCRUBBED. A large number of overdue PGs means the cluster will run concentrated scrub bursts that look like incidents.

5. Check the mClock profile on Quincy or later. The mClock scheduler is the default since Quincy. When it is active, osd_scrub_sleep is ignored (the OSD sets it to 0 under mClock). Scrub I/O is governed by the background best-effort class of the active mClock profile. If you tuned osd_scrub_sleep on an older release and upgraded, your previous tuning may have no effect. Verify with ceph config get osd osd_mclock_profile.

6. On Squid, check for the reservation queuing bug. If PGs are stuck in scrubbing state for extended periods and PG_NOT_DEEP_SCRUBBED warnings accumulate, check the Squid scrub reservation queuing work done in ceph GitHub PRs #56750 (disable) and #57303 (reinstate). The temporary workaround is osd_scrub_disable_reservation_queuing = true, but this option is intended to be removed in a future release.

Metrics and signals to monitor

Fixes

All commands below are cluster-wide configuration changes. Test on a staging cluster or apply during a maintenance window where you can observe the effect on client latency before committing.

Narrow the concurrent scrub count

osd_max_scrubs controls how many scrubs (light and deep combined) a single OSD runs concurrently. The default changed from 1 to 3 in a Reef point release (18.2.3 or 18.2.4), was backported to Quincy (17.2.8), and ships in Squid. On HDD OSDs, 3 concurrent scrubs on one OSD can saturate the device. Lowering it to 1 or 2 reduces peak I/O at the cost of longer total scrub wall-clock time.

# Check current value
ceph config get osd osd_max_scrubs

# Set cluster-wide
ceph config set osd osd_max_scrubs 1

Tradeoff: lower concurrency means scrubs take longer. If the scrub window is narrow, scrubs may not complete, leading to PG_NOT_DEEP_SCRUBBED warnings and growing verification debt.

Define the scrub window

osd_scrub_begin_hour and osd_scrub_end_hour define when scrubs may start. The default is 0 to 24 (always). Restricting scrubs to off-peak hours is the most effective lever for keeping scrub I/O out of the client path.

Important caveat: the window controls when scrubs may start, not when they must finish. A scrub that starts at 23:59 continues past midnight. Also, if the maximum scrub interval is exceeded, a scrub will run regardless of the configured window. The default osd_scrub_max_interval in current releases is 7 days.

ceph config set osd osd_scrub_begin_hour 22
ceph config set osd osd_scrub_end_hour 6

Tune the load threshold

osd_scrub_load_threshold prevents scrubs from starting when OSD load is high. The default is 0.5 (Red Hat Ceph Storage documents the same value). Verify your actual value with ceph config get osd osd_scrub_load_threshold before tuning.

The threshold is expressed as a load ratio. A low threshold means scrubs only start when the OSD is nearly idle. On busy clusters, this can prevent scrubs from ever starting, which causes verification debt. Raising the threshold lets scrubs run more aggressively, at the cost of more client impact.

Suppress scrubs temporarily, with a calendar entry

Setting nodeep-scrub is the right move during a critical customer-facing window. It is the wrong move as a permanent state.

# Suppress deep scrubs immediately (cluster-wide, halts all deep scrub scheduling)
ceph osd set nodeep-scrub

# Resume deep scrubs
ceph osd unset nodeep-scrub

If you set nodeep-scrub, also consider setting noscrub if you want to suppress light scrubs too. Then track the duration. If either flag stays set for more than 24 hours without an active incident or maintenance ticket, treat it as a configuration error.

On mClock (Quincy and later), tune the profile instead of osd_scrub_sleep

If your cluster runs the mClock scheduler, osd_scrub_sleep has no effect. Scrub I/O is governed by the background best-effort class of the active mClock profile. To reduce scrub impact on clients, use a profile that prioritizes client I/O.

# Check the active profile
ceph config get osd osd_mclock_profile

# Options include balanced, high_client_ops, high_recovery_ops
ceph config set osd osd_mclock_profile high_client_ops

Profiles in current Squid: balanced (default; client 50%/weight 1, recovery 50%/weight 1, background best-effort 0/weight 1/limit 90%), high_client_ops (client 60%/2, recovery 40%/1, best-effort 0/1/limit 70%), high_recovery_ops (client 30%/1, recovery 70%/2, best-effort 0/1), and custom.

Tradeoff: high_client_ops protects client latency but slows recovery and scrub. If your cluster also has degraded PGs, this extends the data-loss exposure window.

On Squid, apply the reservation queuing workaround if needed

If PGs are stuck in scrubbing state and PG_NOT_DEEP_SCRUBBED warnings are accumulating, the Squid scrub reservation queuing bug (ceph GitHub PRs #56750/#57303; option osd_scrub_disable_reservation_queuing) may be the cause. The temporary workaround:

ceph config set osd osd_scrub_disable_reservation_queuing true

This reverts to the pre-Squid behavior where scrub replica reservations respond immediately. It is intended to be removed in a future release, so track upstream before relying on it long-term.

Prevention

• Correlate latency with scrub state before paging. The single most effective prevention is an alert rule that checks PG scrubbing state alongside apply latency. If latency spikes and scrubbing PG count spikes simultaneously, do not page. If latency spikes and scrubbing PG count is flat, page.

• Monitor the age of the oldest un-deep-scrubbed PG. Track the maximum last_deep_scrub_stamp age across all PGs. If it exceeds 2x the configured interval (14 days at the default 7-day interval), scrubs are falling behind and verification debt is growing.

• Track nodeep-scrub and noscrub flag duration. Alert if either flag is set for more than 24 hours. The most common preventable scrub-related problem is a forgotten suppression flag left over from maintenance.

• Schedule scrub windows to match your traffic pattern. If your cluster has a clear off-peak period, align the scrub window to it. If traffic is flat 24/7, spread the window wider and lower osd_max_scrubs instead.

• Test scrub behavior after upgrades. The transition from WPQ to mClock (Quincy) silently disabled osd_scrub_sleep for many clusters. After any major upgrade, verify that scrubs still run on schedule and that I/O impact is within tolerance.

• Verify the actual load threshold after deployment. The default for osd_scrub_load_threshold is 0.5 both upstream and in Red Hat Ceph Storage. Confirm the effective value on your cluster rather than relying on documentation.

How Netdata helps

• Per-second apply and commit latency per OSD. Netdata collects ceph_osd_apply_latency_ms and ceph_osd_commit_latency_ms per OSD at one-second resolution. This is fine enough to see the start and end of a scrub pass and to confirm that latency returns to baseline when the scrub completes.

• PG state counts with scrubbing states broken out. The Netdata Ceph collector exposes PG state metrics. Overlaying the scrubbing PG count with apply latency makes the correlation immediate: if the two traces move together, scrub is the cause.

• Anomaly detection on latency without static thresholds. Netdata’s ML-based anomaly detection flags unusual latency for a given OSD relative to its own baseline. This catches the OSD whose latency spikes during scrub even when the cluster-wide average looks normal, and it avoids false positives on OSDs whose normal latency is higher than the cluster median.

• Slow ops as a correlating health signal. ceph_healthcheck_slow_ops is collected as a gauge. During a scrub, expect brief nonzero values that clear when the scrub ends. If slow ops persist, the anomaly advisor surfaces the mismatch.

• Scrub flag state as a configuration signal. Netdata exposes ceph_osd_flag_nodeep-scrub and ceph_osd_flag_noscrub. Track these alongside the PG_NOT_DEEP_SCRUBBED health check to catch the common pattern of a suppression flag left set after maintenance.

• Health detail with per-check labels. ceph_health_detail metrics with the name label let you distinguish PG_NOT_DEEP_SCRUBBED from OSD_SCRUB_ERRORS and other scrub-related checks without parsing CLI output in the dashboard.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators