Ceph HEALTH_WARN: which warnings are noise and which are structural

HEALTH_WARN is an umbrella status, not a single condition. It fires during normal recovery after an OSD restart, and it also fires when the cluster is one step from data loss. Both look identical if you only watch the top-level status. The common mistake is blanket-silencing WARN because it fires too often during recovery, then missing the structural warnings that signal real danger.

The fix is to stop treating HEALTH_WARN as one thing. The ceph_health_detail metric exposes each individual health check with a name label that identifies the specific condition. That name is how you split transient recovery warnings from structural problems.

This reference covers which health check codes are expected during normal operations, which require immediate investigation, and how to mute individual codes without blinding yourself to the structural ones.

What HEALTH_WARN actually means

The monitors compute cluster health from a set of health checks. It surfaces as ceph_health_status: 0 for HEALTH_OK, 1 for HEALTH_WARN, 2 for HEALTH_ERR. WARN means the cluster is still serving data but a condition requires attention. ERR means I/O may be blocked or data is at risk.

The important detail is that HEALTH_WARN is an aggregation. Behind it is a list of specific health check codes, each with its own severity, message, and affected components. The Prometheus MGR module exposes this as ceph_health_detail with labels name and severity only (the message text appears only in ceph health detail JSON output). The value is 1 when the check is active and 0 when inactive.

The name label is the key. It identifies which specific health check is firing. OSD_NEARFULL, MON_CLOCK_SKEW, PG_DEGRADED, SLOW_OPS, and OSD_SCRUB_ERRORS are all distinct names with distinct meanings. Some are transient and expected. Some are structural and dangerous. The top-level WARN status does not tell you which is which.

The split: transient vs structural

The operational question is simple: will this WARN clear on its own, or does it require human action? The table below maps common health check names to their category.

The dividing line is not the severity field. It is whether the condition self-resolves. Degraded PGs during active recovery are noise. Degraded PGs with zero recovery rate are structural. The same health check name can be either, depending on context.

flowchart TD
    A["HEALTH_WARN fires"] --> B["Read ceph_health_detail
by name label"]
    B --> C{"Check name and context"}
    C -->|"Degraded PGs +
recovery active"| D["Transient
Monitor recovery rate"]
    C -->|"Degraded PGs +
recovery stalled"| E["Structural
Recovery blocked"]
    C -->|"OSD_NEARFULL"| F["Structural
Capacity incident"]
    C -->|"OSD_SCRUB_ERRORS"| G["Structural
Data corruption"]
    C -->|"MON_CLOCK_SKEW
persistent"| H["Structural
Fix NTP or risk quorum"]
    C -->|"OSD_DOWN +
noout > 24h"| I["Structural
Noout trap"]
    C -->|"OSD flapping pattern"| J["Structural
Cascade risk"]

Transient warnings: monitor, do not mute

These conditions fire during normal cluster operations. They are expected after OSD restarts, during recovery, during rebalancing, and during planned maintenance. They should clear without intervention.

Degraded and recovering PGs during recovery. When an OSD fails or is added, PGs transition through degraded, recovering, and backfilling states. This is the cluster healing itself. The signal that matters is the recovery rate. If ceph_pool_recovering_bytes_per_sec is nonzero and the degraded count is decreasing, recovery is progressing normally.

Misplaced objects during rebalancing. ceph_num_objects_misplaced rises when CRUSH weights change or OSDs are added. Objects are on non-optimal OSDs but data is fully replicated. This clears as backfill completes.

Peering PGs after topology changes. PGs briefly enter peering after OSD state changes. This is normal. Peering that transitions to active+clean within minutes is expected.

OSD_DOWN with noout set for planned maintenance. If ceph_osd_flag_noout is 1 and the OSD down is intentional, the WARN is expected. The risk is forgetting to unset noout after maintenance. That moves it into the structural category.

Do not mute these codes globally. They are your signal that recovery is happening. The right approach is to alert on the combination of degraded PGs with stalled recovery, not on degraded PGs alone.

Structural warnings: investigate now

These conditions indicate genuine risk to data availability, integrity, or cluster stability. They do not self-resolve. Each one has a specific failure mode behind it.

OSD_NEARFULL. The cluster or individual OSDs have crossed the nearfull ratio (default 0.85). Ceph throttles backfill at this point. If another OSD fails, the cluster may not have enough spare capacity to recover, pushing past backfillfull (0.90) where recovery is blocked entirely. Check ceph osd df tree for per-OSD variance, not just the cluster average. The most-full OSD hits the wall first.

OSD_SCRUB_ERRORS and inconsistent PGs. Deep scrub found that replicas disagree on content, size, or checksums. This is silent data corruption from bit rot, firmware bugs, memory errors, or software bugs. The check ceph_health_detail{name="OSD_SCRUB_ERRORS"} is active. The per-PG inconsistency count is exposed per pool as ceph_pg_inconsistent and is visible via ceph pg dump | grep inconsistent. Do not blindly run ceph pg repair until you have identified which replica is corrupt. If the good replica fails before repair, the data is lost.

PG_FAILED_REPAIR. This is the escalation of inconsistent. Ceph tried to repair and could not. Manual intervention is required to identify the authoritative copy, force repair, or accept data loss. Work with support before forcing anything: ceph pg repair against the wrong replica propagates corruption.

MON_CLOCK_SKEW that persists. Clock drift between monitors beyond mon_clock_drift_allowed (default 0.05s) triggers this check. Transient skew after a MON reboot is normal and clears within minutes if NTP or chrony is healthy. Persistent skew is structural: it causes repeated Paxos elections, map distribution delays, and eventual quorum instability. If ceph_health_detail{name="MON_CLOCK_SKEW"} stays active for more than a few minutes, check NTP or chrony on the MON hosts.

SLOW_OPS. ceph_healthcheck_slow_ops counts operations stuck beyond osd_op_complaint_time (default 30s). These are not slow operations, they are stuck operations. Something is blocking I/O: disk failure, lock contention, peering delays, network partition, or resource exhaustion. Cluster-wide slow ops across many OSDs is far more concerning than isolated slow ops on one OSD.

OSD flapping (no native health check). An OSD cycling between up and down states. Each transition generates a new OSD map epoch that all OSDs must process, and triggers peering across every PG on that OSD. Flapping cascades: the peering overhead can slow healthy OSDs, causing them to miss heartbeats too. Set noout on the specific OSD to stop the peering storm, then investigate the root cause.

noout set for more than 24 hours. ceph_osd_flag_noout equal to 1 for an extended period is the single most common preventable Ceph outage. The operator set noout for maintenance, forgot to unset it, and the cluster stopped recovering from subsequent failures. Degraded PGs accumulate silently. The next failure may cause data loss.

LARGE_OMAP_OBJECTS. RGW bucket index shards have accumulated too many OMAP keys. This is the early stage of an OMAP storm. The OSD hosting the oversized index will degrade as RocksDB operations slow. Identify affected buckets and trigger resharding.

PG_NOT_DEEP_SCRUBBED sustained beyond the policy window. If deep scrubs are not completing within the configured interval, the cluster is accumulating verification debt. Corruption may be going undetected. Check whether noscrub or nodeep-scrub flags are set and forgotten.

POOL_NO_REDUNDANCY or TOO_FEW_OSDS. CRUSH rules cannot place replicas across enough failure domains. This is a configuration error, not a runtime fault, but it means data is not distributed as expected for failure tolerance.

How to read ceph_health_detail the right way

The name label on ceph_health_detail is how you split the two categories programmatically. Do not alert on the umbrella ceph_health_status alone. Alert on specific health check names, with context.

For inspecting active checks:

# Check which health checks are currently active
ceph health detail

# List active check names in machine-readable form
ceph health detail -f json-pretty | jq '.checks | keys'

The Prometheus metric ceph_health_detail{name="OSD_NEARFULL"} returns 1 when that check is active. Alert on specific names rather than on the aggregated status. For transient checks, add context. Alert on degraded PGs only when recovery rate is also near zero, which distinguishes stalled recovery from normal healing.

For muting, use per-code muting, never blanket suppression:

# Mute a specific check for a limited duration. Note: there is no
# `ceph health mute list` subcommand; muting always requires a code.
ceph health mute POOL_APP_NOT_ENABLED 1h

# Sticky mute persists for the full duration even if the condition clears
# (the --sticky flag is confirmed across Reef and Squid)
ceph health mute POOL_APP_NOT_ENABLED 24h --sticky

Muting is per-code. You cannot mute the entire HEALTH_WARN status, and you should not try to. The right pattern is to mute specific codes that are known-noise for your deployment, while leaving structural codes unmuted and alerting on them.

The checks that look like noise but are structural

Some health checks are easy to dismiss because they seem benign or because they fire during routine operations. These are the ones that cause outages when ignored.

noout flag. It looks like a maintenance artifact. It is the noout trap. A flag set for a one-hour maintenance window that was never cleared will silently prevent recovery for days or weeks. ceph_osd_flag_noout should alert if set for more than 24 hours without an accompanying maintenance ticket.

Clock skew. It looks like an NTP hiccup. Persistent MON_CLOCK_SKEW is a precursor to quorum loss. The default threshold is tight at 0.05s. VMs are particularly prone to drift. Do not disable the check. Fix NTP or chrony.

Scrub errors. They look like a one-time checksum mismatch. They are silent data corruption. If the consistent replica fails before repair, the corrupt copy becomes the only copy. Investigate promptly and identify which replica is wrong before running repair.

Nearfull. It looks like a capacity heads-up. It is the threshold past which recovery may be blocked. At 85% utilization, losing OSDs pushes the cluster toward backfillfull, where healing stops entirely. Treat nearfull as a capacity incident, not an advisory.

How Netdata helps

• Per-check health detail metrics. Netdata collects ceph_health_detail with the name label for each health check code. This lets you alert on specific structural checks like OSD_NEARFULL, OSD_SCRUB_ERRORS, and MON_CLOCK_SKEW without firing on transient recovery checks.
• Recovery rate correlation. Netdata surfaces recovery throughput alongside degraded PG counts on the same dashboards. Alerting on degraded PGs with near-zero recovery rate distinguishes stalled recovery from normal healing.
• Flag state monitoring. Cluster-wide flags like noout and norecover are collected as metrics. This catches the noout trap before it causes an outage.
• Per-OSD capacity and latency. Nearfull warnings on individual OSDs are visible alongside per-OSD commit and apply latency, so you can see whether a nearfull OSD is also a slow OSD.
• Clock skew via health checks. ceph_health_detail{name="MON_CLOCK_SKEW"} is collected directly, so you do not need a separate NTP monitoring path to catch persistent skew.

Related guides

• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert