Ceph noout flag left set: the most common preventable outage

The noout flag is the single most common source of preventable Ceph outages. An operator sets it before maintenance to stop Ceph from marking down OSDs as out, then forgets to unset it. The cluster keeps serving I/O, health stays at HEALTH_WARN (not ERR), and nothing visible breaks. But recovery is now disabled by policy. The next OSD failure stacks on top of the first, and degraded PGs that should have healed days ago are still degraded.

The trap is silent because noout does not break anything by itself. It only changes what happens when something else breaks. A cluster with noout set and every OSD up is indistinguishable from a healthy cluster on most dashboards. The damage appears only when an OSD goes down and fails to recover. By the time someone notices, the cluster may be one failure away from data loss.

What this means

The noout flag tells the monitors: do not automatically mark down OSDs as out. Normally, when an OSD daemon stops, Ceph waits mon_osd_down_out_interval (default 600 seconds) and then marks the OSD out. Marking out triggers CRUSH remapping and recovery. With noout set, that countdown never fires. The OSD stays down+in indefinitely. Its PGs stay degraded. No recovery starts.

The flag exists for good reason. During planned maintenance such as rebooting a host, upgrading, or replacing a disk, you do not want Ceph to start an expensive full backfill the moment an OSD restarts. noout holds the OSD’s place in CRUSH so that when it comes back, recovery is a quick PG log replay instead of a full data copy.

The problem is that noout has no built-in expiry. It stays set until someone manually unsets it. And because the cluster looks healthy while noout is set and all OSDs are up, the flag is easy to forget.

flowchart TD
    A[Operator sets noout] --> B[Maintenance completes]
    B --> C{noout unset?}
    C -- Forgotten --> D[noout stays set]
    D --> E[OSD goes down]
    E --> F[mon_osd_down_out_interval fires?]
    F -- Blocked by noout --> G[OSD stays down+in]
    G --> H[PGs stay degraded]
    H --> I[No recovery starts]
    I --> J[Next OSD failure
= data loss risk]
    C -- Yes --> K[Normal recovery behavior]

Common causes

Quick checks

# Check cluster-wide recovery flags
ceph osd dump | grep flags

# Check which health checks are firing
# OSDMAP_FLAGS = cluster-wide noout, OSD_FLAGS = per-OSD or per-bucket noout
ceph health detail

# See OSD up/down/in/out state at a glance
ceph osd tree

# Quick OSD summary (X osds: Y up, Z in)
ceph osd stat

# Check for all recovery-blocking flags at once
ceph osd dump | grep -E 'noout|norecover|nobackfill|noscrub|nodeep-scrub'

# See PG state summary
ceph pg stat

# Check recovery rate and cluster status
ceph -s

How to diagnose it

1. Confirm noout is set and identify its scope. Run ceph osd dump | grep flags for the cluster-wide flag. If that line does not show noout, check ceph health detail for the OSD_FLAGS health check, which names specific OSDs or CRUSH buckets with per-OSD or per-bucket noout set. Per-OSD noout (set via ceph osd add-noout) and per-bucket noout (set via ceph osd set-group noout) do not appear in the cluster-wide flag line.

2. Estimate how long noout has been set. The OSDMAP_FLAGS health check does not include a timestamp. Cross-reference with your change management system, maintenance tickets, or MON logs. If noout has been set for more than 24 hours without an active maintenance window, treat it as stale.

3. Check whether any OSDs are down. This is the urgent condition. Run ceph osd tree and look for OSDs in the down state. If noout is set and any OSD is down, the cluster is running at reduced redundancy with no recovery in progress. This is the state that precedes data loss.

4. Check for other recovery-blocking flags. A forgotten maintenance window often leaves multiple flags set. Run ceph osd dump | grep flags and look for norecover, nobackfill, noup, nodown, noscrub, and nodeep-scrub. Each disables a different recovery or integrity function.

5. Check degraded PG count and recovery rate. Run ceph pg stat and ceph -s. If degraded PGs exist and recovery rate is zero, recovery is blocked. The block may be noout (OSDs never marked out), norecover or nobackfill (recovery explicitly stopped), or backfill_toofull (target OSDs too full).

6. Verify maintenance is actually complete. Before unsetting noout, confirm that the maintenance it was set for is finished. If an OSD is still being replaced or a host is still being rebooted, unsetting noout prematurely will trigger unnecessary recovery I/O.

Metrics and signals to monitor

Fixes

Clear the cluster-wide noout flag

If noout was set cluster-wide and maintenance is complete:

# Verify maintenance is done before running this
ceph osd unset noout

After unsetting, Ceph resumes normal behavior. Any OSD that has been down longer than mon_osd_down_out_interval will be marked out, and recovery will begin. This recovery I/O competes with client traffic, so monitor client latency after unsetting if the cluster was heavily degraded.

Clear per-OSD or per-bucket noout

Per-OSD noout and per-bucket noout do not respond to ceph osd unset noout. Clear them with the matching command:

# Per-OSD noout (set with ceph osd add-noout)
ceph osd rm-noout osd.<id>

# Per-bucket noout, e.g. a host or rack in CRUSH
ceph osd unset-group noout <bucket-name>

The per-OSD and per-bucket forms are the recommended approach since Luminous (12.x) because they limit the blast radius. A forgotten per-OSD noout affects one OSD. A forgotten cluster-wide noout disables recovery for the entire cluster.

Handle a stuck per-host noout (Rook)

If you run Ceph under Rook with disruption management enabled, a removed host can leave a stale per-bucket noout on a CRUSH node that no longer exists. The cluster-wide flag is unset, but the OSD_FLAGS health check persists. Clear it from the toolbox:

ceph osd unset-group noout <hostname>

This is a known Rook issue (rook/rook #11100). If it recurs after node removal, check the Rook disruption controller logs.

Clear other forgotten flags

If ceph osd dump | grep flags shows norecover, nobackfill, noscrub, or nodeep-scrub alongside noout, clear them all once maintenance is confirmed complete:

ceph osd unset norecover
ceph osd unset nobackfill
ceph osd unset noscrub
ceph osd unset nodeep-scrub

Each flag disables a different function. norecover and nobackfill block data movement. noscrub and nodeep-scrub block integrity verification. Leaving them set indefinitely means the cluster is neither healing nor verifying data integrity.

Verify recovery starts

After clearing flags, confirm recovery is actually progressing:

# Watch recovery activity
ceph -s

# Check that degraded PG count starts decreasing
ceph pg stat

If recovery does not start, check for other blockers: backfill_toofull (target OSDs too full), unfound objects, or remaining flags.

Prevention

Alert on recovery-blocking flag duration. Ticket when any of ceph_osd_flag_noout, norecover, nobackfill, noscrub, or nodeep-scrub has been set for more than 24 hours. The same forgotten-flag pattern applies to all of them. Maintenance should not take more than a day without an explicit extension, so a duration-based alert catches the forgotten case without firing during legitimate short windows.

Alert urgently on noout plus a down OSD. The combination of noout set and any OSD down is the direct precursor to data loss. This should be an urgent ticket at minimum, and a page if the cluster is already running near capacity or has other degraded PGs.

Prefer per-OSD or per-bucket noout. Since Luminous, ceph osd add-noout <osd-id> and ceph osd set-group noout <bucket-name> scope the flag to the OSD or host you are actually maintaining. A forgotten per-OSD noout affects one OSD. A forgotten cluster-wide noout affects the entire cluster.

Pair every flag set with an unset reminder. Use a runbook checklist, a calendar reminder, or an automation step that fires when noout is set and alerts if it is still set after the maintenance window closes. The TripleO fast-forward upgrade bug, where automation set noout in step 1 but never ran the unset in step 5, is one example of why this matters.

Audit flags regularly. Run ceph osd dump | grep flags as part of routine cluster checks. Any flag that is set without an active maintenance ticket is stale and should be cleared.

How Netdata helps

Netdata collects ceph_osd_flag_noout and the other recovery-blocking flags (norecover, nobackfill, noscrub, nodeep-scrub) as gauges, alongside per-OSD ceph_osd_up state. The urgent condition (noout set plus a down OSD) is visible on a single dashboard without manual correlation, and a single alert rule on any flag set for more than 24 hours catches the whole class of forgotten maintenance flags.

ceph_health_detail metrics surface OSDMAP_FLAGS (cluster-wide noout) and OSD_FLAGS (per-OSD or per-bucket noout) as labeled gauges, so you can distinguish a forgotten cluster-wide flag from a stale per-host flag without parsing CLI output. Recovery rate (ceph_pool_recovering_bytes_per_sec) and degraded PG count (ceph_pg_degraded) are collected per second, making a stalled recovery obvious: a flat degraded count with zero recovery rate is the signature of a blocked recovery.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn