Ceph OSD up/down vs in/out: the four states and mon_osd_down_out_interval

Most Ceph operations references describe an OSD as “down” as if it were a single state. It is not. An OSD carries two independent flags: up/down (is the daemon alive?) and in/out (does CRUSH place data on it?). Combined, that produces four states, and the most dangerous one, down+in, is invisible if you only look at one flag.

The two flags are linked by a clock. After an OSD goes down, the monitors wait mon_osd_down_out_interval (default 600 seconds) before automatically flipping it to out. That 10-minute window is the gap between “OSD stopped” and “recovery starts.” That timer, and the flags on either side of it, are the vocabulary every recovery, flapping, and noout runbook assumes.

This is a reference, not a troubleshooting flow. It maps the four states, explains the timer that transitions between them, and identifies the signals you need to distinguish a transient blip from a recovery in progress.

What it is and why it matters

The up/down and in/out flags live in the OSD map and are maintained by the monitors. They answer different questions.

• up/down (ceph_osd_up): is the OSD daemon running and responding to heartbeats? This is daemon liveness. The monitors mark it down after peer OSDs report the daemon as unresponsive past osd_heartbeat_grace (default 20 seconds), with mon_osd_min_down_reporters (default 2) reporters from different subtrees at mon_osd_reporter_subtree_level (default host).
• in/out (ceph_osd_in): is the OSD participating in CRUSH placement? This is data placement. in means new PGs can be assigned to it and existing PGs expect data to live there. out means CRUSH has excluded it.

The two are independent in principle but linked by mon_osd_down_out_interval. An OSD that goes down is not immediately out. The monitors wait the configured interval first, giving the operator (or systemd) a chance to bring the daemon back before the cluster starts the expensive work of moving its data elsewhere.

This distinction matters because every recovery runbook, every noout warning, and every flapping cascade is described in terms of these four states. If you read “OSD is down” without knowing whether it is also in, you do not yet know whether recovery has started.

How it works

The four states, with the operational meaning of each:

The dangerous state is down+in. The data is “expected” on a daemon that is not answering. PGs served by that OSD are degraded, and the cluster has not yet committed to recovery. This is the window where a second failure in the same failure domain can cause data loss.

stateDiagram-v2
    [*] --> UP_IN
    UP_IN: up + in (normal)
    DOWN_IN: down + in (degraded, no recovery yet)
    UP_OUT: up + out (maintenance)
    DOWN_OUT: down + out (recovering)
    UP_IN --> DOWN_IN: daemon stops / heartbeat timeout
    DOWN_IN --> UP_IN: restart within mon_osd_down_out_interval
    DOWN_IN --> DOWN_OUT: mon_osd_down_out_interval expires (600s), unless noout set
    DOWN_OUT --> UP_OUT: daemon restarts after auto-out
    UP_OUT --> UP_IN: ceph osd in, or mon_osd_auto_mark_auto_out_in
    UP_IN --> UP_OUT: ceph osd out (manual)
    note right of DOWN_IN
        noout flag blocks
        transition to DOWN_OUT
    end note

The transitions, in order of how often you will see them:

1. Daemon stops. OSD moves from up+in to down+in. Peers mark it down via heartbeat. The mon_osd_down_out_interval timer starts.
2. Daemon restarts within the window. OSD moves back to up+in. PGs recover via PG log replay, which is relatively cheap. No backfill.
3. Timer expires without restart. OSD moves to down+out. CRUSH recalculates placement; recovery and backfill begin to other OSDs.
4. Daemon returns after auto-out. OSD moves to up+out. Because mon_osd_auto_mark_auto_out_in defaults to true, it will be marked back in automatically, triggering backfill to rebalance data onto it. This is the expensive path: a brief outage that exceeded 600 seconds produces a full backfill in both directions.
5. Manual ceph osd out. Moves up+in to up+out without taking the daemon down. Used for planned evacuations.
6. Manual ceph osd in. Reverses it.

Verified syntax: per-OSD noout is ceph osd add-noout osd.<id> (and ceph osd rm-noout osd.<id>); per-CRUSH-bucket noout is ceph osd set-group noout <bucket> (comma-separated flags from {noup,nodown,noin,noout}) with ceph osd unset-group to clear it. No preconfigured “group” object is needed; the target is a CRUSH bucket or device class.

The noout flag blocks transition 3. With noout set, a down+in OSD stays down+in indefinitely. PGs remain degraded and recovery never starts. This is intentional during brief maintenance, and the cause of the “noout trap” when it is left set after maintenance ends.

A related but separate concept is ceph_osd_weight. Setting an OSD’s reweight to 0 excludes it from placement regardless of its in/out state. This is a soft eviction used during investigation. It stops new placements without triggering the full OUT transition machinery, and it does not produce the same OSD map churn.

Where it shows up in production

Host reboot during maintenance. You reboot a host with 10 OSDs. All 10 go down+in. You have 600 seconds to bring them back before auto-out fires and the cluster starts moving terabytes of data. This is exactly why operators set noout (or per-OSD ceph osd add-noout osd.<id>) before planned reboots. Set it, do the work, unset it.

Brief network blip on the cluster network. OSDs lose heartbeats to peers, get marked down+in. Network recovers in 30 seconds. OSDs come back up+in. PG log replay handles it. No backfill. The 600-second window did its job.

OSD segfault. Daemon dies, systemd does not restart it (or restarts it slowly). OSD sits at down+in for 600 seconds, then transitions to down+out. Recovery starts. If this was a 3x-replicated pool and another OSD in the same PG’s acting set fails during the 600-second window, you have a data availability problem.

Flapping OSD. Daemon oscillates between up+in and down+in faster than the 600-second timer can expire. Each flap generates a new OSD map epoch, triggering peering across hundreds of PGs. The OSD never reaches down+out, so recovery never starts, but the peering storms damage cluster performance. The fix is ceph osd add-noout osd.<id> to stop the peering cascade, then investigate the underlying cause.

Forgotten noout. Maintenance finishes, noout is left set. Over the next week, three OSDs fail. All three sit at down+in. PGs accumulate in degraded state. The cluster appears to be limping but functional, until a fourth failure tips some PGs into incomplete. This is the most common preventable Ceph outage.

When this matters

The 600-second window is a tuning dial, not a constant. mon_osd_down_out_interval defaults to 600s and most clusters leave it alone. Shortening it speeds recovery start (less exposure) but increases the chance that a transient blip triggers an expensive full backfill. Lengthening it gives transient issues more time to self-resolve but extends the degraded window. Do not tune it without thinking about both sides.

mon_osd_adjust_down_out_interval (default true) scales the interval automatically. When enabled, the monitors increase mon_osd_down_out_interval when an OSD “appears to be laggy” (the option’s documented purpose). This can help clusters with a few slow OSDs but can mask deteriorating hardware. If you are trying to understand why a particular OSD took longer than 600 seconds to be marked out, this is why.

mon_osd_down_out_subtree_limit (default rack) blocks auto-out at scale. If the monitors detect that all OSDs within a rack are down, they will not auto-mark them out. This prevents a rack-wide outage from triggering a recovery storm that would saturate the surviving racks. If your CRUSH hierarchy is flat and lacks a rack level, this safeguard may not apply. Verify your topology if you run flat.

noout is a surgical instrument, not a default. Per-OSD noout (ceph osd add-noout osd.<id>) is safer than the cluster-wide flag because the cluster-wide version is easy to forget. Per-CRUSH-bucket noout is the right tool for maintenance on a whole host or rack.

Setting mon_osd_down_out_interval to 0 disables auto-out entirely. The OSD_NO_DOWN_OUT_INTERVAL health check warns about this. Some operators do it deliberately on clusters where they want manual control over every OUT transition. If you do this, your OSD-down alerting needs to be solid, because the safety net is gone.

Signals to watch in production

The single most useful correlation is ceph_osd_up == 0 AND ceph_osd_in == 1, sustained. That is the down+in window, and its duration is the cluster’s exposure to a second failure without recovery in progress.

How Netdata helps

• The Ceph collector exposes ceph_osd_up and ceph_osd_in per OSD, so you can alert on the four states directly rather than relying on the cluster-wide down count.
• The down+in window (ceph_osd_up == 0 AND ceph_osd_in == 1) is the most operationally meaningful state. Per-second granularity lets you see exactly when an OSD crossed into it and whether it recovered or transitioned to down+out.
• Correlating ceph_osd_flag_noout with down OSDs identifies the noout trap before it causes a cascading failure.
• OSD map epoch churn is visible as a rate on the OSD state metrics. Sustained churn indicates flapping before the OSD_FLAPPING health check fires.
• ML-based anomaly detection on per-OSD state transitions surfaces flapping patterns that simple thresholds miss, useful for the marginal disks that fail slowly.
• Per-pool PG state metrics sit alongside the OSD state metrics, so you can confirm whether a down+in OSD is actually producing degraded PGs and whether recovery has started.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine
• Ceph nearfull: the 85% warning that decides whether the cluster can heal