Ceph MDS_DAMAGED: metadata journal or cache corruption

MDS_DAMAGED means the CephFS Metadata Server has found damaged metadata, either in its journal or in on-disk structures read from the metadata pool, and has deliberately refused to continue serving the rank. CephFS may be partially or fully unavailable, and recovery is not the usual automatic failover path. A standby that takes over would replay the same suspect journal, so the cluster parks the rank until a human intervenes.

Treat MDS_DAMAGED as a careful, manual recovery situation, not a restart-the-daemon situation. The wrong first move can convert recoverable journal corruption into permanent metadata loss. The playbook alert condition is ceph_health_detail{name="MDS_DAMAGE"} active for more than 120 seconds (the health check code is MDS_DAMAGE; MDS_DAMAGED is not a Ceph health check name).

This article covers how to read the damage, decide between journal recovery, scrub repair, and backup restore, and avoid the two failure modes that turn a damaged rank into a damaged filesystem: blindly resetting the journal, and running ceph pg repair on the metadata pool without identifying the authoritative copy.

What this means

The MDS keeps the CephFS namespace in an in-memory cache and writes a metadata journal to a RADOS pool. Each rank has a single journal replayed on failover. When the MDS encounters corrupt or missing metadata while reading from the metadata pool, it reports the MDS_DAMAGE health check. The monitor-generated cluster-level health message is “N mds daemon(s) damaged” (e.g. “1 mds daemon damaged”), with one detail line per damaged rank, and the daemon-reported check surfaces per-rank damage entries.

There are three distinct shapes of damage, each needing a different response:

flowchart TD
    A[MDS_DAMAGE active] --> B{damage ls shows what?}
    B -->|backtrace entry| C[Backtrace damage]
    B -->|journal read failure| D[Journal corruption]
    B -->|table object read failure| E[Bootstrap race or OSD op timeout]
    C --> F[Often benign: scrub with repair]
    D --> G[cephfs-journal-tool: inspect, export, recover_dentries]
    E --> H[Fix underlying cause, then ceph mds repaired]
    F --> I[ceph mds repaired]
    G --> I
    H --> I
    I --> J[Online MDS scrub: recursive, repair, force]
    J --> K[Rank returns to active]

Backtrace damage is a reverse-link inconsistency between an inode’s data objects and its path. It is frequently benign and repaired by an online MDS scrub. Journal corruption is more serious: the journal events themselves cannot be replayed cleanly, and you may lose metadata mutations that were journaled but not yet flushed to the backing store. Bootstrap-race damage happens when the MDS cannot read its table objects during startup because PGs in the metadata pool are not yet active, or because rados_osd_op_timeout is set aggressively enough to fail those reads.

MDS_DAMAGED is a refusal state, not a crash. The MDS could keep serving, but it has chosen not to, because continuing would risk spreading corruption to clients. Your job is to identify which kind of damage it is, repair or discard the affected metadata, then tell the monitor the rank is safe to start again with ceph mds repaired.

Common causes

Quick checks

Run these read-only commands before touching anything. They tell you which kind of damage you are dealing with and whether the underlying metadata pool is healthy.

# Confirm the health check and the rank
ceph health detail | grep -iE 'mds.*damage|MDS_DAMAGE'

# List active MDS daemons and rank state
ceph fs status
ceph mds stat

# List damage entries from a still-running MDS admin socket
ceph tell mds.<fs_name>:<rank> damage ls

# Check the metadata pool PG states for inconsistencies
# (replace <metadata_pool_id> with your pool's numeric ID)
ceph pg dump | awk '$1 ~ /^<metadata_pool_id>\./ && $NF !~ /active\+clean/'

# Look for OSDs hosting the metadata pool that are down or degraded
ceph osd tree | grep -E 'down|host'
ceph osd dump | grep -E 'flags|nearfull|full'

# Check for an aggressive rados_osd_op_timeout that can cause spurious damage
ceph config get mds rados_osd_op_timeout 2>/dev/null || echo "not set"

# Check for snaptrim activity that correlates with the damage timestamp
ceph osd dump | grep -E 'nosnaptrim|nodeep-scrub'

# Look for OOM kills or disk errors on the MDS host around the damage event
dmesg -T | grep -iE 'oom|ceph-mds|I/O error' | tail -n 50

Do not run ceph pg repair against the metadata pool yet. On a metadata pool, the primary copy is not necessarily authoritative, and pg repair can overwrite good metadata with bad. Identify the cause first.

How to diagnose it

1. Confirm which rank is damaged and the damage type. Start with ceph health detail and ceph fs status. The cluster-level message names the rank. If any MDS for that filesystem is still responsive, run ceph tell mds.<fs>:<rank> damage ls to enumerate per-rank damage entries. Each entry has a damage_type (backtrace, frag, dentry, inode, etc.), an ino, and an id you will need later for damage rm.

2. Distinguish backtrace damage from journal corruption. Backtrace entries with ino: 1 (the root inode) are a known pattern in older CephFS versions where the root backtrace was never written. Journal corruption shows up differently: the MDS log contains lines like failed to read journal or bad journal magic, and cephfs-journal-tool journal inspect reports the journal as unreadable or truncated.

3. Check whether the damage is real or a bootstrap artifact. If the MDS went damaged immediately after starting, look at the metadata pool PG states. If PGs were still peering when the MDS tried to read its table objects, the MDS may have failed reads and marked itself damaged. This matches tracker #51866 (mds daemon damaged after outage): a globally set rados_osd_op_timeout caused the MDS to fail table reads while the metadata pool PGs were still peering, and the damage resolved with ceph mds repaired <rank> once the PGs reached active+clean. The read-timeout code path is unchanged in current releases.

4. Export the journal before doing anything destructive. Before journal reset, event recover_dentries, or any damage rm, back up the journal: cephfs-journal-tool --rank=<fs>:<rank> journal export <backup.bin>. This is the insurance policy. If a later step makes things worse, you can re-import the original journal.

5. Check the underlying OSD and PG health of the metadata pool. If PGs in the metadata pool are inconsistent or down, fix that first. MDS damage on top of unhealthy RADOS is a much harder recovery.

Metrics and signals to monitor

Fixes

Recovery paths group by damage type. The wrong path for the wrong type makes things worse.

Backtrace damage (often benign)

If damage ls reports damage_type: "backtrace" and the inode is the root (ino: 1) or another inode whose data objects still exist, run an online MDS scrub with repair and force. The standard pattern from the disaster-recovery documentation:

# Bring the rank up enough to scrub. Mark the rank repaired so the
# monitor lets the MDS start, then scrub the affected subtree.
ceph mds repaired <role>
ceph tell mds.<fs_name>:0 scrub start / recursive,repair,force

# Once the scrub completes and reports the backtrace repaired,
# remove the damage entry so the health check clears.
ceph tell mds.<fs_name>:0 damage rm <id>

<role> is the filesystem role (for example 0 or <fs_name>:0). <id> is the damage entry ID from damage ls. Backtrace-only damage with intact data objects is the most recoverable MDS_DAMAGED scenario.

Journal corruption

When the journal itself is unreadable, you need cephfs-journal-tool. The workflow is: inspect, export, recover what you can, then reset.

# Assess journal health.
cephfs-journal-tool --rank=<fs>:<rank> journal inspect

# Back up the journal before any modification.
cephfs-journal-tool --rank=<fs>:<rank> journal export journal-backup.bin

# Recover dentries from the journal into the metadata pool.
# 'summary' shows what would be recovered; use 'apply' to write.
cephfs-journal-tool --rank=<fs>:<rank> event recover_dentries summary
cephfs-journal-tool --rank=<fs>:<rank> event recover_dentries apply

# WARNING: Only after recover_dentries. Reset is destructive: any
# journal events not recovered above are lost.
cephfs-journal-tool --rank=<fs>:<rank> journal reset --force

After journal reset, mark the rank repaired and start it.

Tradeoff: recover_dentries writes inodes and dentries from the journal into the backing store only if they are higher-versioned than what is already there. You will lose metadata mutations that were never journaled, and you may see orphaned inodes that need a later cephfs-data-scan pass to clean up. Run an online scrub after the rank is active to verify integrity.

Bootstrap race or rados_osd_op_timeout induced damage

If the damage appeared at MDS startup, with metadata pool PGs still peering, the journal is probably intact. The fix is to remove the cause, then clear the damaged flag:

# Remove the aggressive timeout if set.
ceph config rm mds rados_osd_op_timeout

# Wait until the metadata pool is active+clean.
ceph pg dump_stuck unclean

# Tell the monitor the rank is safe to start.
ceph mds repaired <role>

The same pattern applies when an MDS co-located with OSDs starts before the metadata pool is ready. The durable fix is startup ordering: delay MDS startup until PGs are active. There is no official systemd unit override for this; teams typically add an ordering dependency (unit After=/Wants=) or a wrapper that waits for the metadata pool before starting the MDS.

When to restore from backup instead

Prefer journal recovery when:

• The damage is backtrace-only.
• journal inspect reports the journal is mostly readable and only a small tail is corrupt.
• The filesystem has few in-flight mutations since the last scrub.

Prefer a backup restore of the metadata pool (or a full filesystem restore) when:

• journal inspect reports the journal as unreadable end-to-end.
• The metadata pool PGs are also inconsistent or down.
• recover_dentries recovers an implausibly small fraction of the expected inodes.
• You have a known-good metadata pool snapshot more recent than the corruption event.

The alternate-pool recovery procedure (build a fresh metadata pool and reconstruct metadata from the data pool using cephfs-data-scan) is documented but explicitly marked as not extensively tested in the Reef documentation. Treat it as a last resort behind journal recovery and backup restore.

Prevention

• Do not set rados_osd_op_timeout on MDS nodes. It can cause the MDS to fail reading its table objects during bootstrap and mark itself damaged even when the journal is fine. The default (no timeout) is correct for MDS.
• Run MDS scrubs regularly. Backtrace damage accumulates quietly and is trivially fixed by a scrub with repair. A weekly or monthly online scrub of the root subtree catches it early.
• Keep deep scrubs running on the metadata pool. noscrub and nodeep-scrub left set on the metadata pool is a common precursor to MDS damage going undetected until the MDS reads the bad object.
• Size the MDS cache to the working set. A chronically over-limit MDS spends time evicting under cap pressure, which slows journal flushes and widens the window where an interrupted commit can corrupt the journal.
• Pause snaptrim during heavy MDS load. Operators have reported journal corruption following snaptrim under load; ceph osd set nosnaptrim during known-heavy CephFS windows removes that variable.
• Disable MDS subtree rebalancing during incidents. Subtree export under load can cause slow requests that mask underlying journal corruption. Setting mds_bal_interval to 0 during recovery simplifies the picture.
• Back up the journal before any destructive action. This is the single highest-leverage habit. A journal export takes seconds and has saved recoveries that would otherwise have become restores.

How Netdata helps

• The Netdata Ceph collector surfaces ceph_health_detail per check, so MDS_DAMAGED appears as its own labeled time series alongside MDS_ALL_DOWN, FS_DEGRADED, and the umbrella ceph_health_status. You see the exact second the rank went damaged and which other checks fired at the same time.
• Per-second collection lets you correlate the MDS_DAMAGED transition with preceding signals: a spike in ceph_mds_mem_rss (OOM-kill precursor), an inconsistent PG appearing in the metadata pool, or an OSD going down on the host that holds the journal pool.
• ceph_mds_slow_reply and per-operation MDS latency histograms let you confirm whether the damage was preceded by cap pressure or subtree rebalancing slowdowns, which changes the recovery plan.
• Anomaly detection on MDS memory, request latency, and OSD commit latency can flag slow drift that often precedes an MDS crash-induced journal corruption, hours before the health check fires.
• The metadata pool PG state metrics are tracked per pool, so you can see whether inconsistent or down PGs preceded the MDS damage, which determines whether you are dealing with real metadata corruption or a bootstrap-race artifact.
• Annotations on the chart timeline let you mark the moment you ran cephfs-journal-tool journal export or ceph mds repaired, so the recovery is auditable after the fact.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural