Ceph OSD heartbeat timeouts: the osd_heartbeat_grace precursor to flapping

Ceph OSDs declare each other down based on a simple rule: if a peer does not respond to heartbeat pings within osd_heartbeat_grace (default 20 seconds), it is reported to the monitors as unresponsive. Two peers from different failure domains must agree before the monitor marks the OSD down. This is the mechanism that turns a transient stall into an OSD state change, and repeated near-misses against this 20-second window are the most reliable leading indicator of OSD flapping.

The problem for operators is that the precursor signal, heartbeat ping latency, is not exposed by the Prometheus MGR module. By the time you see an OSD transition to down in your metrics, the grace period has already been exceeded and the cluster is already paying the cost. You have to infer rising heartbeat latency from indirect signals: OSD apply and commit latency, cluster-network congestion, OSD host CPU and memory pressure, and the OSD_SLOW_PING_TIME health checks.

This article covers how the heartbeat timeout mechanism works, why the effective grace period is often longer than 20 seconds, where the failure pattern shows up in production, and which correlated signals reveal the precursor before flapping begins.

What it is and why it matters

Two parameters govern the peer-to-peer heartbeat that determines whether an OSD is considered alive:

• osd_heartbeat_interval (default 6 seconds): OSDs ping their peers at random intervals less than this value. The randomization prevents synchronized ping storms.
• osd_heartbeat_grace (default 20 seconds): if a peer does not respond within this window, the OSD reports it as down to the monitors.

The monitor does not act on a single report. mon_osd_min_down_reporters (default 2) requires two reporting OSDs, and mon_osd_reporter_subtree_level (default host) requires them to come from different failure domains. This prevents a single noisy peer from marking a healthy OSD down. Once the threshold is met, the monitor marks the OSD down and issues a new OSD map epoch.

What matters here is the window between “heartbeat latency is rising” and “heartbeat latency has exceeded grace.” That window is your detection opportunity. Once grace is exceeded, the OSD is down, PGs re-peer, and if the cause is transient, the OSD comes back up only to flap again.

Do not confuse this with the suicide timeout

Ceph has a separate internal mechanism enforced by the heartbeat_map. If an OSD’s internal operation thread pool (the osd_op_tp thread) stops responding, the OSD kills itself. This appears in OSD logs as is_healthy 'OSD::osd_op_tp thread …' had timed out after 15 for the first timeout (osd_op_thread_timeout, default 15s), and as had suicide timed out after 150 when osd_op_thread_suicide_timeout (default 150s) is exceeded, which aborts the OSD. The 150s figure in Red Hat references is the suicide timeout.

This is not the same as osd_heartbeat_grace. The suicide timeout means the OSD’s own internal threads are stalled, typically due to slow disk I/O, RocksDB compaction, or OMAP bloat. The OSD crashes itself rather than serving stale state. Peer heartbeat timeouts, by contrast, mean the OSD may still be processing internally but is too slow to respond to inter-OSD pings within the grace window. The log messages look similar but point to different layers of the stack.

Do not confuse this with mon_osd_report_timeout

mon_osd_report_timeout (default 900 seconds) is the monitor-level backstop: if an OSD has not reported to the monitor directly within this window, the monitor marks it down independently of peer reports. In practice, peer heartbeat timeouts (20 seconds) fire long before report timeouts become relevant. The report timeout exists to catch OSDs that maintain a monitor connection but have lost all peer connectivity.

How it works

flowchart TD
    A["Host, disk, or network
pressure stalls OSD"] --> B["Heartbeat ping
latency rises"]
    B --> C["Ping exceeds 1s:
OSD_SLOW_PING_TIME warning"]
    B --> D{"Ping exceeds
osd_heartbeat_grace
default 20s?"}
    D -- Yes --> E["Peers report OSD down
2 reporters from different subtrees"]
    E --> F["MON marks OSD down
OSD map epoch bumps"]
    F --> G["PGs re-peer
recovery traffic starts"]
    G --> H["Stall clears
OSD returns to up"]
    H --> B

The cascade starts with anything that delays the OSD’s ability to respond to heartbeat pings. The ping is a lightweight message, but it competes for the same OSD resources as client I/O, recovery, scrub, and RocksDB compaction. When the op queue stalls, heartbeat responses stall with it.

The effective grace period is often not 20 seconds

mon_osd_adjust_heartbeat_grace defaults to true. When enabled, the monitor dynamically extends the grace period for individual OSDs based on laggy estimates derived from recent heartbeat behavior. The relevant tuning parameters are mon_osd_laggy_halflife, mon_osd_laggy_weight, and mon_osd_laggy_max_interval (default 5 minutes).

This feature was introduced to reduce false positives during brief network blips. In practice, it can mask genuine problems. In a reported Octopus-era rack failure test, the dynamic grace period ballooned far beyond the 20-second base (the laggy estimate is bounded by mon_osd_laggy_max_interval, default 5 minutes), so OSDs that should have been marked down within seconds took far longer to register as down. Setting mon_osd_adjust_heartbeat_grace to false resolved it immediately. The options behind this extension are unchanged in Reef and Squid.

The implication for detection: if your cluster has this feature enabled (the default), the effective grace period for a laggy OSD may be much longer than 20 seconds. The OSD_SLOW_PING warning and peer reports may arrive later than you expect. If you are investigating a slow-to-detect-down scenario, check this setting first.

The OSD_SLOW_PING health checks

Ceph exposes two health checks that fire before the grace period is exceeded:

• OSD_SLOW_PING_TIME_FRONT: heartbeat ping times on the front (public) interface exceed the configured threshold.
• OSD_SLOW_PING_TIME_BACK: heartbeat ping times on the back (cluster) interface exceed the configured threshold.

The threshold is ratio-based: the MON warns when ping time exceeds mon_warn_on_slow_ping_ratio (default 0.05) of osd_heartbeat_grace, i.e. 1s with default settings (mon_warn_on_slow_ping_time is 0 by default; setting it overrides the ratio).

These are precursor warnings. A 1-second heartbeat ping is nowhere near the 20-second grace threshold, but it indicates that network or OSD responsiveness is degrading. The health message reports the longest ping time seen but does not identify which OSDs are involved. To find the specific OSDs, use ceph health detail or the dump_osd_network admin socket command.

The dump_osd_network command (sent to an individual OSD via its admin socket) exposes per-pair heartbeat ping latency with 1-minute, 5-minute, and 15-minute averages, plus min, max, and last values. It is the only source of structured per-pair heartbeat latency data. This data is not exported by the Prometheus MGR module, which means automated alerting on heartbeat latency requires custom tooling or log-based checks.

Where it shows up in production

Overloaded OSD host

When a host is CPU-saturated, memory-pressured, or running competing workloads, all OSDs on that host struggle to respond to heartbeats promptly. This is common on hosts with many OSDs sharing limited RAM. As BlueStore cache pressure increases and the kernel begins reclaiming memory, OSD threads stall, and heartbeat responses queue behind stalled operations. If the delay exceeds grace, peers report the OSDs down. Because the host-level problem affects all OSDs on the host, multiple OSDs may flap simultaneously.

Recovery traffic on a shared NIC

Recovery and backfill traffic competes with heartbeat pings on the cluster network. If the cluster network is not separated from the public network, or if a single NIC handles all traffic, recovery bursts can saturate the link. During heavy recovery, heartbeat pings queue behind replication and backfill packets. On a 300-OSD NVMe cluster running FIO warm-up writes, operators observed heartbeat ping times of 1202ms (back interface) and 1535ms (front interface), triggering OSD_SLOW_PING_TIME warnings and collapsing throughput.

If the cluster uses separate public and cluster networks, a degradation or failure of the cluster network while the public network stays healthy creates a specific flapping pattern: OSDs report each other down via the public network, then immediately mark themselves back up when the public network heartbeat succeeds. This rapid cycle is classic flapping driven by asymmetric network failure.

Dying disk stalling the op queue

A disk with intermittent I/O errors, growing bad sectors, or firmware issues stalls the OSD’s op queue intermittently. The OSD may serve most requests normally but occasionally hang for several seconds during a specific sector read or a garbage-collection event in the firmware. These hangs delay heartbeat processing. If the hang is long enough to exceed grace, peers report the OSD down. When the hang clears, the OSD comes back up, and the cycle repeats.

This pattern is insidious because the disk may pass SMART checks between stalls, and the OSD’s average latency metrics may look normal between events. The signal is in the tail latency and the slow ops count, not the averages.

BlueStore DB spillover and RocksDB compaction stalls

When the BlueStore DB partition (RocksDB metadata) fills and spills to the slow data partition, RocksDB operations slow by one to two orders of magnitude. Compaction events that previously completed in milliseconds on SSD take seconds on HDD. During compaction, the OSD’s internal threads stall, and heartbeat responses are delayed. This creates periodic latency spikes that correlate with compaction activity. The bluefs slow_used_bytes counter in ceph daemon osd.<id> perf dump confirms whether spillover has occurred. Any non-zero value means the performance cliff has been reached.

cephadm upgrades and full sync

During orchestrated upgrades, a monitor performing a full sync on HDD-backed storage can saturate the host’s filesystem, causing all OSDs on that host to experience internal heartbeat_map timeouts. The OSDs reboot repeatedly while the orchestrator continues the upgrade without pausing. This is not a peer-heartbeat-grace failure in the strict sense, but it produces the same observable symptom: OSDs cycling up and down while the host is under I/O pressure. If you see flapping during a cephadm upgrade, check whether a monitor full sync is saturating a shared host.

The observability gap

The single most important thing to understand about this failure pattern is that the primary precursor signal is not in your standard metrics pipeline.

The Prometheus MGR module does not export per-OSD heartbeat latency. The ceph_osd_up metric tells you when an OSD has already been marked down, which is the outcome, not the precursor. There is no native OSD_FLAPPING health check; the pattern is established by watching OSD up/down churn (ceph osd stat) and OSD map epoch rate. Neither gives you the leading indicator.

To detect rising heartbeat latency before it becomes a flap, correlate indirect signals:

The correlation that matters most: a single OSD showing rising apply latency, a non-zero slow ops count, and an OSD_SLOW_PING_TIME warning at the same time is an OSD approaching heartbeat grace. If recovery or backfill is also active on the cluster network, the window before the OSD is reported down may be short.

How Netdata helps

Netdata’s per-second collection surfaces several correlated signals that help identify rising heartbeat latency before grace is exceeded:

• Per-OSD apply and commit latency (ceph_osd_apply_latency_ms, ceph_osd_commit_latency_ms) collected every second, with anomaly detection that flags individual OSDs deviating from their own baseline rather than relying on cluster averages that hide outliers.
• Slow ops count (ceph_healthcheck_slow_ops) tracked alongside OSD latency, so you can see the moment an OSD transitions from slow to stuck and correlate it with per-OSD latency spikes on the same timeline.
• Health detail signals including OSD_SLOW_PING_TIME_FRONT, OSD_SLOW_PING_TIME_BACK, and OSD_FLAPPING from ceph_health_detail, surfaced as individual time series rather than buried in a text status string.
• Host-level CPU, memory, disk I/O, and network metrics on every OSD host at per-second resolution, so OSD-level latency spikes can be correlated with host-level saturation in the same dashboard view.
• Cluster network bandwidth per interface, to distinguish client-driven from recovery-driven congestion and see when heartbeat pings are likely queuing behind bulk transfers.
• Cross-signal anomaly correlation across per-OSD latency, host load, and network utilization, so a heartbeat-precursor pattern (rising latency on one OSD during a recovery burst) appears as a correlated anomaly cluster rather than isolated noisy metrics.

Related guides

• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine
• Ceph nearfull: the 85% warning that decides whether the cluster can heal