Ceph OSD down: telling a dead disk apart from a network blip

OSD_DOWN fires. ceph_osd_up flipped to 0. The 600-second countdown to OUT and recovery has begun.

Your first job: figure out whether this is a dead disk (act now, plan a replacement) or a network blip (wait, verify, let the OSD come back). Acting on the wrong diagnosis wastes disk and network I/O on a needless backfill, or worse, leaves a failing disk in service past the point where SMART was already warning you.

The OSD_DOWN alert is a TICKET, not a PAGE, by design. Whether data is actually at risk depends on pool sizes, CRUSH failure domains, and which PGs landed on the missing OSD. The OSD is suspect; the PGs are the verdict.

What this means

ceph_osd_up = 0 means the monitor has stopped hearing from the OSD daemon. Ceph does not take a single peer’s word for it: at least 2 OSDs from different hosts (mon_osd_min_down_reporters default 2, mon_osd_reporter_subtree_level default “host”) must miss heartbeats past the 20-second grace (osd_heartbeat_grace) before the monitor marks the OSD down.

Once down, a 600-second countdown starts (mon_osd_down_out_interval default 600). At the end, the OSD is marked OUT, PGs remap, and Ceph recovers data onto surviving OSDs. This is the cost of misdiagnosing a network blip: once OUT fires, an OSD that returns must backfill its PGs rather than replay the PG log.

The OSD_DOWN health check summary is e.g. “1 osds down” (the monitor prints “ osds down”). It is a TICKET by design. The PG-state signals (ceph_pg_down, ceph_pg_incomplete, ceph_pg_degraded with stalled recovery) carry the actual data-availability verdict and are what you should consider paging on.

Before assuming recovery will run at all, check the noout flag. With ceph_osd_flag_noout set, a down OSD stays DOWN+IN indefinitely, PGs stay degraded, and no recovery runs. Forgotten noout is the single most common preventable Ceph outage.

Common causes

Quick checks

Read-only triage commands. None of these change cluster state.

# Confirm the OSD_DOWN health check and list which OSDs are flagged
ceph health detail | grep -A2 OSD_DOWN

# Show down OSDs in topology context (host, failure domain, weight)
ceph osd tree down

# Check whether noout is blocking recovery
ceph osd dump | grep -E 'noout|flags'

# Per-OSD commit and apply latency as reported by peers
ceph osd perf

# PGs that are degraded, stale, or down because of the missing OSD
ceph pg dump_stuck degraded
ceph pg dump_stuck stale

# On the OSD host: kernel messages for the backing device
dmesg -T | grep -iE 'error|fail|i/o|killed process'

# Is the ceph-osd daemon process actually running?
systemctl status ceph-osd@<id>

# Recent OSD daemon log
tail -n 200 /var/log/ceph/ceph-osd.<id>.log

# MTU sanity check toward a peer OSD host (jumbo-frame payload is 8972)
ping -M do -s 8972 -c 3 <peer_osd_host>

How to diagnose it

The decision flow is: is the daemon process alive, and if so, is the disk healthy?

1. Locate the OSD host and check the daemon. ceph osd tree down tells you which host. SSH in. systemctl status ceph-osd@<id> tells you whether systemd thinks it is running; ps -ef | grep ceph-osd confirms the process exists.

2. If the process is gone, look at how it died.

   • dmesg -T | grep -i 'killed process' for OOM-kill.
   • OSD log tail for had suicide timed out after or the abort message hit suicide timeout (the watchdog fires at 150 seconds, 10x the osd_op_thread_timeout default of 15 seconds).
   • If neither appears and peer OSDs on the same host are also down, suspect host crash or host network failure rather than the daemon.

3. If the process is alive, the disk is the prime suspect. Pull SMART with smartctl -A /dev/<dev>. Look at Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable. Any nonzero and rising means the device is failing. Cross-check with dmesg for medium errors. BlueStore writes the raw block device, but the kernel still logs medium errors at the block layer.

4. If SMART is clean, look at the cluster network. A failed private (cluster) network with a working public network is the classic flapping pattern: OSDs mark each other down on the monitor while still serving clients. Check the cluster-network interface (ip -s link show), peer reachability with the jumbo-frame ping, and switch port counters. If all OSDs on the host are down together, suspect shared infrastructure: NIC, cable, or top-of-rack switch. If only this OSD is down and peers on the same host are up, the cluster network is probably fine; lean toward slow disk or daemon stall.

5. Check what Ceph itself thinks. ceph health detail shows whether the OSD is reported as flapping (OSD_FLAPPING health check active). If it is flapping rather than cleanly down, the diagnosis leans toward network or overload, not a hard disk failure.

6. Decide before 600 seconds. The cheapest action is “let it come back” if you suspect a network blip. The expensive action is “mark it out and recover” if the disk is dead. If you cannot decide in time, set noout to stop the clock and keep investigating. Do not set noout globally unless you mean it; a forgotten global noout prevents recovery cluster-wide.

flowchart TD
    A[ceph_osd_up = 0] --> B{Daemon process alive?}
    B -- No --> C{dmesg: OOM-kill?}
    C -- Yes --> D[Tune osd_memory_target, restart]
    C -- No --> E{OSD log: assert or suicide timeout?}
    E -- Yes --> F[Crash: inspect slow ops, RocksDB]
    E -- No --> G[Peer OSDs on host also down: host or net failure]
    B -- Yes --> H{SMART errors or dmesg EIO?}
    H -- Yes --> I[Disk failure: stop, mark out, replace]
    H -- No --> J{Other OSDs on host also down?}
    J -- Yes --> K[Host failure: power, NIC, switch]
    J -- No --> L{OSD_FLAPPING health check active?}
    L -- Yes --> M[Network: MTU, cluster-network, switch port]
    L -- No --> N[Slow disk overload: slow ops, deep-scrub, RocksDB]

Metrics and signals to monitor

Fixes

The fix depends on the cause. Do not restart the daemon as a first move: in-memory state goes away and you lose the ability to query the admin socket for ops-in-flight and bluefs stats while the OSD is still warm.

Disk failure

If SMART shows reallocated or pending sectors, or dmesg shows medium errors, the disk is dead or dying. Stop the OSD, mark it out (if noout is set, unset it for this OSD), and follow your disk-replacement procedure. These commands commit the OSD to replacement; run ceph osd safe-to-destroy osd.<id> first if you have any doubt.

# Destructive: commits the OSD to replacement
ceph osd out osd.<id>
systemctl stop ceph-osd@<id>
ceph osd destroy osd.<id> --yes-i-really-mean-it   # only when you are sure you will not reuse it as-is

Host crash

If peer OSDs on the same host went down at the same epoch, treat it as a host problem. The OSDs are not the incident. Check IPMI/iLO for power events, journalctl -b -1 for the previous boot, and the host’s network. The OSDs will come back when the host does. If mon_osd_down_out_interval fires first, recovery starts and you eat a backfill on the host’s return. Consider setting noout globally during the investigation if you expect the host back inside an hour, and unset it the moment the host is healthy.

Network loss or MTU mismatch

If the daemon is alive and the disk is healthy, and especially if OSD_FLAPPING is active, suspect the network. Verify MTU end to end with ping -M do -s 8972. A single misconfigured switch port will flap every OSD that talks through it. Check ip -s link show for input and output errors and /proc/net/snmp for rising TCPRetransSegs. Fix the underlying fault and the OSD stops flapping on its own.

OOM-kill

If dmesg shows the OOM-killer targeted the OSD, the host is over-subscribed. Lower osd_memory_target for the OSDs on the host, or add RAM. A one-off OOM during a recovery burst can be survived by systemctl start ceph-osd@<id>, but if it recurs you have a sizing problem, not a transient fault.

Suicide timeout (slow disk or RocksDB stall)

A suicide timeout means an operation thread was stuck long enough to trigger the watchdog. The OSD killed itself rather than stall indefinitely. The cause is almost always disk-side: an uncompacted OMAP, a BlueStore DB spillover event, or a deep-scrub running into bad media. While the daemon is down, the OSD log file still has the trace. Check historical monitoring captures or the OSD log for evidence of slow_used_bytes > 0 before the crash, since the admin socket is unavailable while the daemon is stopped. Check whether a deep-scrub was scheduled on the OSD. If BlueStore DB spillover is the cause, plan a ceph-bluestore-tool DB migration to a larger device before bringing the OSD back, or it will time out again.

Prevention

• Alert on noout for more than 24 hours. Forgotten noout prevents recovery cluster-wide.
• Alert on per-OSD latency outliers, not cluster averages. A dying disk drifts to 5x its device-class median for days before it dies; cluster averages hide that.
• Monitor SMART with rate-of-change alerting. One old reallocated sector is noise; five new ones in a week is a failing disk.
• Separate public and cluster networks. Heartbeat and recovery traffic should not share the client-facing NIC.
• Verify MTU consistency after switch firmware upgrades. Mismatched MTU masquerades as disk failure via flapping.
• Alert on rising slow ops before they become suicide timeouts. A sustained rise on one OSD in the hour before it dies is the precursor pattern.
• Page on PG states, not on ceph_osd_up. ceph_pg_down and ceph_pg_incomplete are the actual availability verdict.

How Netdata helps

• The Ceph collector exposes ceph_osd_up and ceph_osd_in per OSD with per-second resolution, so the exact second of the down transition and the transition to OUT roughly 600 seconds later are both visible on the same chart.
• ceph_health_detail is scraped as a labelled gauge, so you can chart the moment OSD_DOWN and OSD_FLAPPING fire next to the OSD up/down signal. The discrimination between clean-down and flapping is visible at a glance.
• Per-OSD ceph_osd_apply_latency_ms and ceph_osd_commit_latency_ms make the slow-disk precursor pattern visible hours before the OSD dies.
• The ceph_osd_flag_noout metric lets you alert on the noout trap directly: ticket on noout == 1 for more than 24 hours without a maintenance window.
• PG-state metrics (ceph_pg_down, ceph_pg_incomplete, ceph_pg_degraded with recovery-rate correlation) give you the actual data-availability verdict alongside the daemon-state signal, so you can decide TICKET versus PAGE on the same dashboard.
• Host-level disk and memory collectors on the OSD host put block-device latency, OOM events, and process RSS next to the Ceph metrics, so OOM-kill and disk-failure causes are separable without switching tools.

Related guides

• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph nearfull: the 85% warning that decides whether the cluster can heal