Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows

OSD flapping is a failure cascade. One OSD misses heartbeats, peers mark it down, its PGs start peering and recovering elsewhere, the OSD comes back, peering reverses, and the cycle repeats. Each flap mints a new OSD map epoch that every OSD in the cluster must process. The peering overhead from a single flapping OSD can slow dozens of healthy OSDs enough that they also miss heartbeats, and the cascade spreads.

The signature: the OSD map epoch incrementing multiple times per minute, SLOW_OPS appearing across many OSDs (not just the problematic one), and PGs cycling through peering states without reaching active+clean. Client latency spikes, retries add load, and the cluster appears to be failing in multiple places at once when the root cause is often a single marginal disk or a network issue on one host.

This article covers how to recognize flapping early, stabilize the cluster before the cascade spreads, and diagnose the underlying cause. The stabilization step is time-critical: the longer an OSD flaps, the more OSD map epochs are minted, the more peering work the cluster does, and the more likely healthy OSDs are to join the flap.

What this means

Ceph detects OSD failure through peer heartbeats, not through monitor polling. Each OSD sends heartbeat pings to a small set of peers at intervals shorter than osd_heartbeat_interval (default 6 seconds). If a peer does not receive a heartbeat response within osd_heartbeat_grace (default 20 seconds), it may report the OSD as down to the monitors.

A single report is not enough. By default, mon_osd_min_down_reporters is 2 and mon_osd_reporter_subtree_level is host, meaning two OSDs in different host failure domains must report the OSD down before the monitors mark it down. This prevents false positives from a single misbehaving peer. But once the threshold is met, the monitor commits a new OSD map epoch with that OSD marked down.

That epoch is the expensive part. Every OSD in the cluster must receive, process, and acknowledge the new map. Every PG whose acting set includes the flapping OSD must re-peer. Recovery I/O starts moving data to surviving replicas. Then the OSD comes back up (the underlying issue was transient), the monitors commit another epoch, peering reverses, and recovery I/O reverses with it. The net data movement is near zero, but the CPU, disk, and network cost of the peering churn is enormous.

The cascade begins when this peering load slows healthy OSDs enough that their heartbeat responses are delayed past the grace period. Those OSDs then get reported down too, multiplying the problem.

flowchart TD
    A[OSD misses heartbeat grace] --> B[2 peers report it down]
    B --> C[MON mints new OSD map epoch]
    C --> D[PGs re-peer and recover]
    D --> E[Peering load slows OSDs]
    E --> F[Original OSD returns]
    F --> C
    E --> G[Surviving OSDs miss heartbeats]
    G --> B

Common causes

Quick checks

# Ceph has no dedicated OSD_FLAPPING health check; watch OSD up/down churn
watch -n 10 'ceph osd tree | grep -E "down|out"'

# See which OSDs are down and their location in the CRUSH tree
ceph osd tree | grep down

# Check cluster status, recovery line, and slow ops
ceph -s

# Check for admin flags that may be affecting recovery behavior
ceph osd dump | grep flags

# Watch the OSD map epoch increment rate
watch -n 5 'ceph osd stat | grep "epoch"'

# Check slow ops detail and where ops are stuck
ceph health detail | grep -A20 SLOW_OPS

# On the flapping OSD host: check for OOM kills
dmesg -T | grep -i "killed process"

# On the flapping OSD host: check disk SMART health
smartctl -H /dev/<device>

# Verify cluster network MTU end-to-end (jumbo frames)
ping -c 10 -s 8972 -M do <peer-osd-ip>

How to diagnose it

1. Confirm flapping is active. Ceph has no dedicated OSD_FLAPPING health check. Watch ceph osd tree repeated over a minute and look for OSDs alternating between up and down; the OSD map epoch counter in ceph osd stat climbing is corroborating evidence.

2. Identify the flapping OSDs. Sample the OSD list repeatedly (for example watch -n 10 'ceph osd tree | grep -E "down|out"') and record which OSDs alternate state. If only one OSD is flapping, the root cause is likely local to that OSD (disk or memory). If multiple OSDs on the same host are flapping, suspect the host (network, CPU, or memory). If OSDs across multiple hosts are flapping, suspect the network fabric or a systemic issue like conntrack exhaustion.

3. Check the OSD map epoch rate. This is the single most important signal during a flap. Each epoch represents a map commit and full cluster-wide propagation. More than one or two epochs per minute sustained is a sign of active flapping. Several per minute means the cascade is at risk of spreading.

4. Examine the slow ops distribution. Run ceph health detail | grep SLOW_OPS and read the detail. If slow ops appear on many OSDs, not just the flapping one, the peering storm is already consuming cluster resources. The stuck-operation description is the most valuable diagnostic: waiting on pg points to peering stalls, waiting for subops from points to replication or network stalls.

5. Investigate the host. On the flapping OSD host, check:

   • dmesg -T | grep -i "killed process" for OOM kills
   • smartctl -A /dev/<device> for reallocated or pending sectors
   • iostat -xz 1 for disk saturation (high %util, high await)
   • ip -s link show <iface> and ethtool -S <iface> for network errors
   • cat /proc/sys/net/netfilter/nf_conntrack_count versus _max for conntrack exhaustion

6. Check whether it is cascading. If OSDs that were previously healthy are now also cycling up and down, the cascade has begun. Stabilize immediately (see Fixes) before more OSDs join.

Metrics and signals to monitor

OSD heartbeat latency (the precursor to flapping) is not directly exposed via the Prometheus MGR module. The osd_ping_latency counter is available through ceph daemon osd.<id> perf dump on the host, but it is not in the standard metrics pipeline. The first externally visible signal of heartbeat failure is the OSD being marked down.

Fixes

Stabilize first (universal first response)

Before diagnosing the root cause, stop the peering storm. Every additional flap makes the cluster worse and risks cascade.

Pin the flapping OSD with per-OSD noout. This prevents CRUSH from remapping PGs away from the OSD, which stops the peering reversal:

# Prevent the specific OSD from being marked out
ceph osd add-noout osd.<id>

Then mark it down and stop the daemon so it stops participating in heartbeats entirely:

# Mark the OSD down and stop the daemon
ceph osd down osd.<id>
systemctl stop ceph-osd@<id>

After stabilization, investigate the root cause. Clear the per-OSD noout once the OSD is fixed and back up: ceph osd rm-noout osd.<id>.

Emergency brake: cluster-wide noup and nodown

If multiple OSDs are flapping and the cascade is spreading, you can freeze all OSD state transitions cluster-wide as an emergency measure:

# Emergency: freeze all OSD up/down state transitions
ceph osd set nodown
ceph osd set noup

This stops the monitors from marking any OSD down or up, halting the peering storm. Client I/O to healthy OSDs continues. Once the root cause is addressed, unset both:

ceph osd unset nodown
ceph osd unset noup

Use this with care. While noup and nodown are set, the cluster cannot respond to real OSD failures. Clear them as soon as the situation is stable.

Disk failure

If SMART shows reallocated sectors, pending sectors, or offline uncorrectable counts, the disk is failing. Replace it. The standard procedure:

1. Mark the OSD out: ceph osd out osd.<id>
2. Wait for backfill to complete (monitor with ceph -s)
3. Stop the OSD, destroy it, replace the disk, redeploy

Do not leave a marginal disk in service hoping it stabilizes. It will flap again.

Network issues

MTU mismatch. If you use jumbo frames (MTU 9000), verify the path MTU end-to-end with ping -c 10 -s 8972 -M do <peer-ip>. A single switch port or bond member without jumbo frames causes silent packet fragmentation and severe latency that looks like disk slowness.

nf_conntrack exhaustion. If the kernel connection tracking table fills, OSD connections fail intermittently. Check dmesg for “nf_conntrack: table full” messages. Raise net.netfilter.nf_conntrack_max or blacklist the module if Ceph traffic does not need connection tracking.

Cluster network design. If your public and cluster networks share a NIC, recovery and replication traffic competes with client I/O. The Reef-era docs softened the historical recommendation to always run separate public and cluster networks, noting that a separate cluster network which fails while the public network works is itself a primary cause of flapping. Test both paths.

NIC or cable faults. Check ethtool -S <iface> for receive or transmit errors, drops, and CRC failures. A failing NIC or dirty fiber can cause intermittent packet loss that is just enough to miss heartbeat grace.

Memory pressure

If the OOM killer is terminating the OSD, the host does not have enough memory for the number of OSDs and their configured osd_memory_target. Check:

# OSD process RSS
ps -eo pid,rss,args | grep ceph-osd
# Configured memory target
ceph config get osd osd_memory_target

Either add RAM to the host, reduce the number of OSDs, or lower osd_memory_target at the cost of cache performance. On hosts with many HDD OSDs, the aggregate memory demand is significant.

mClock scheduler on HDDs

On HDD clusters using the mClock scheduler (default since Quincy), the default shard configuration can cause slow requests that manifest as flapping. On Quincy, Reef, and Squid 19.2.0 the defaults are osd_op_num_shards_hdd=5 and osd_op_num_threads_per_shard_hdd=1; the officially recommended change is to invert the shard and thread counts, which upstream adopted as the new default in Squid 19.2.1. On 19.2.1 and later, verify the current values before changing them (ceph config get osd.X osd_op_num_shards_hdd). The classic fix is:

# Adjust shard config for HDD OSDs with mClock
ceph config set osd osd_op_num_shards_hdd 1
ceph config set osd osd_op_num_threads_per_shard_hdd 5

Restart the affected OSDs after applying.

The markdown backstop and containerized deployments

Ceph has a built-in mechanism to stop an OSD from flapping indefinitely. The osd_max_markdown_count and osd_max_markdown_period parameters control how many times an OSD can mark itself down within a window before Ceph shuts it down instead of restarting. The defaults are 5 markdowns within 600 seconds, after which the OSD logs “marked down … > osd_max_markdown_count … shutting down” and stops.

In containerized deployments (Rook on Kubernetes), this backstop may not behave as expected. Kubernetes restart policies can cause the OSD pod to restart even after Ceph’s markdown limit is reached, perpetuating the flap. Rook v1.13.0 and later includes a fix that makes the OSD pod sleep instead of immediately restarting when flapping is detected. If you run Ceph under Rook, verify your Rook version.

Prevention

• Monitor the OSD map epoch rate. This is the leading indicator of flapping. Alert on sustained rates above baseline (more than 1 or 2 per minute outside of planned maintenance).
• Monitor per-OSD latency outliers. A marginal disk shows rising commit or apply latency days before it starts missing heartbeats. Alert on any OSD at 5x the cluster median for its device class.
• Verify MTU consistency across all OSD hosts and switch ports after any network change, including NIC replacements.
• Size nf_conntrack_max appropriately for your cluster, or blacklist the module if Ceph traffic does not need connection tracking.
• Keep NTP or chrony disciplined on monitor hosts. Clock skew destabilizes Paxos elections, which slows map commits and worsens flapping impact.
• Test recovery throttling periodically. Recovery that was safe with 10 OSDs may cause cascading failures with 100 OSDs. Review osd_max_backfills and osd_recovery_max_active when the cluster grows significantly.
• Prefer per-OSD noout over cluster-wide noout. Use ceph osd add-noout osd.<id> for a single problem OSD. Cluster-wide noout is the trap that prevents recovery from any failure and is one of the most common preventable Ceph outages.

How Netdata helps

• Per-second OSD state metrics (ceph_osd_up, ceph_osd_in) catch the up/down cycling pattern faster than longer polling intervals, which can miss rapid flapping between samples entirely.
• The OSD_FLAPPING health check (ceph_health_detail{name="OSD_FLAPPING"}) surfaces as a dedicated signal, so you see the specific failure pattern rather than inferring it from OSD down counts.
• Correlation with slow ops (ceph_healthcheck_slow_ops) and PG peering counts (ceph_pg_peering) on the same timeline shows whether the peering storm is spreading beyond the original OSD.
• Host-level signals (disk latency, OOM events, network errors, TCP retransmits) on the same dashboard as the Ceph metrics let you identify the root cause without switching tools.
• MON Paxos commit latency and OSD map epoch trends reveal when the monitor cluster is being stressed by rapid map commits.
• Anomaly detection on per-OSD latency and OSD map epoch rate can flag a marginal disk or rising peering overhead before the first heartbeat miss.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph nearfull: the 85% warning that decides whether the cluster can heal