Ceph MDS_ALL_DOWN: CephFS is completely unavailable

MDS_ALL_DOWN means no active Metadata Server (MDS) rank exists for a CephFS filesystem. Every CephFS client depending on that filesystem blocks on metadata operations. This is a hard outage for CephFS, not a degradation. The health check ceph_health_detail{name="MDS_ALL_DOWN"} goes active the moment the Monitor has no active rank to assign for that filesystem.

The metadata itself is journaled in a RADOS pool. Unless MDS_DAMAGED is also firing, the metadata is intact. The outage is availability, not loss. Recovery means restoring at least one active MDS, after which clients reconnect and resume.

What makes MDS_ALL_DOWN distinct from a normal failover: under normal conditions, an active MDS that stops sending beacons is replaced by a standby within roughly 10 to 30 seconds. If MDS_ALL_DOWN persists for more than a minute or two, the automatic failover itself failed. The usual reasons: there is no eligible standby, the standbys also crashed (often from the same OOM or workload), or the Monitor cannot assign the rank because of filesystem or journal state.

The playbook severity for MDS_ALL_DOWN is TICKET at more than 120 seconds, with an explicit caveat: if CephFS is your primary storage interface, escalate to PAGE. Treat MDS_ALL_DOWN as an outage in any deployment where CephFS serves production workloads.

What this means

Each CephFS filesystem has one or more active MDS ranks. The max_mds setting controls the active rank count. Each active rank should have a standby available to take over. The Monitor tracks MDS daemons via beacons: every MDS sends a beacon every mds_beacon_interval (default 4 seconds). If the Monitor does not hear from a daemon within mds_beacon_grace (default 15 seconds), it marks that daemon laggy and, if a standby is available, assigns the rank to the standby.

flowchart TD
    A[Active MDS rank stops beaconing] --> B{Standby available?}
    B -->|yes, eligible| C[Monitor assigns rank to standby]
    C --> D[Failover in 10 to 30s]
    B -->|no standby| E[MDS_ALL_DOWN]
    C -->|standby also crashed or OOM| E
    C -->|standby pinned to another fs| E
    C -->|MDS_DAMAGED blocks rejoin| F[MDS_DAMAGED]
    E --> G[CephFS clients hang on metadata ops]

MDS_ALL_DOWN fires when, for a given filesystem, no daemon holds any active rank and the Monitor cannot assign one. From the client side, this manifests as CephFS operations hanging indefinitely. Reads and writes to already-open files may continue briefly against client caches, but any metadata operation (stat, open, create, readdir) blocks. Eventually clients time out.

Two distinctions to make on first contact:

• MDS_ALL_DOWN vs FS_DEGRADED: FS_DEGRADED means the filesystem is operational but at reduced redundancy, typically because an active rank failed over and not enough standbys remain. CephFS is still serving I/O. MDS_ALL_DOWN is a full outage.
• MDS_ALL_DOWN vs MDS_DAMAGED: MDS_DAMAGED means an MDS reported damaged state, indicating metadata corruption in its journal or cache. This is the scenario where you may need ceph-recover-journal and where data integrity, not just availability, is at stake. MDS_ALL_DOWN without MDS_DAMAGED is purely an availability problem.

Common causes

Quick checks

# Check the health check and which filesystem is affected
ceph health detail | grep -A3 MDS_ALL_DOWN

# See active ranks, standbys, and per-rank state
ceph fs status

# Full MDS map with rank assignments, standby replay, and daemon states
ceph fs dump

# Count daemons per state
ceph mds stat

# Check filesystem configuration
ceph fs get <fs_name>

# On each MDS host, confirm whether the daemon process is running
systemctl status ceph-mds@<daemon_name>

# Check for OOM kills on MDS hosts
dmesg -T | grep -i "out of memory\|oom" | grep ceph-mds

# Verify the MDS journal pool is reachable
ceph osd pool stats <metadata_pool_name>

How to diagnose it

1. Confirm which filesystem and ranks are down. Run ceph fs status and ceph fs dump. Note the filesystem name, the configured max_mds, and the state of every MDS daemon. Answer: how many active ranks are configured, how many are currently active, how many standbys exist.

2. Determine whether the MDS daemons are running at all. On each MDS host, check systemctl status ceph-mds@<name> and ps aux | grep ceph-mds. If the daemons are not running, check journalctl -u ceph-mds@<name> and /var/log/ceph/ceph-mds.*.log. The daemon may have crashed, been OOM-killed, or been stopped by an operator.

3. Check for OOM kills. This is the most common root cause. The MDS can rapidly allocate memory until the kernel OOM killer terminates it, especially when the cache is undersized for the working set or when clients are not returning caps. Run dmesg -T | grep ceph-mds and journalctl -k | grep -i oom. Repeated OOM kills usually mean the daemon is restarting, OOMing again on journal replay, and never reaching active state.

4. Check standby configuration. Even if the original active failed cleanly, MDS_ALL_DOWN only persists if no standby can take over. Look at:

   • standby_count_wanted for the filesystem: if it is 0, the insufficient-standby health check is disabled and you may simply have no standbys.
   • mds_join_fs on each standby: standbys with filesystem affinity will only be assigned to their named filesystem.
   • refuse_standby_for_another_fs (default false): if true, standbys from other filesystems will not be used even as a last resort.
   • Standby-replay coverage: each active rank may have at most one standby-replay daemon, and a standby-replay daemon will not be assigned to take over a failure for another rank or a different filesystem. With max_mds greater than 1, a single standby-replay only protects one rank.

5. Check for MDS_DAMAGED. Run ceph health detail. If MDS_DAMAGED is also active, the path is different: the damaged daemon refuses to rejoin to protect metadata integrity. You need the damaged-MDS recovery procedure, which may involve ceph-recover-journal. Do not force a damaged MDS back into the cluster.

6. Check Monitor health. If MONs are struggling (quorum issues, slow Paxos), they may fail to assign ranks even when daemons are healthy. Run ceph quorum_status and ceph mon stat. Since Squid (v19), ceph fs set <fs> max_mds <n> on a cluster with filesystem health warnings requires --yes-i-really-mean-it (PR #59420); older runbooks that omit the flag fail with EPERM here.

7. Check the metadata pool. The MDS reads and writes its journal to a RADOS pool. If that pool is unavailable (PGs down, incomplete, or stuck), the MDS cannot start. Run ceph osd pool stats <metadata_pool> and check the pool’s PG states. The metadata is usually safe even when MDS_ALL_DOWN is firing, but if the metadata pool itself has failed, that is a different and more serious incident.

8. Read the MDS logs. /var/log/ceph/ceph-mds.<daemon>.log will tell you exactly why a daemon refused to go active, crashed, or got stuck. Common patterns: journal replay failures, damaged state reports, cap recall storms, or assertion failures with backtraces.

Metrics and signals to monitor

Fixes

All MDS daemons OOM-killed

This is the most common MDS_ALL_DOWN pattern. The daemon restarts via systemd, tries to replay the journal, allocates memory to cache inodes, and immediately OOMs again. Clients running recursive scans (find, rsync, du on huge trees) or operating on a large working set are typical triggers.

1. Raise mds_cache_memory_limit if the host has spare RAM. The default is 4GB. Increasing it gives the MDS headroom, but only if the host actually has the memory. Live-tunable via ceph tell mds.<id> injectargs --mds_cache_memory_limit <bytes>, then persist in the config.
2. Identify and stop the workload flooding the MDS cache. Look for clients with high unreturned cap counts: ceph daemon mds.<id> session ls, find sessions with large recalled_caps.
3. If a specific client is pathologically holding caps, you may need to evict it manually so the MDS can trim its cache and reach active state: ceph tell mds.<id> client evict id=<client-id>. Eviction drops the client’s session; any un-flushed buffered I/O from that client is lost.
4. Confirm mds_health_cache_threshold (default 1.5x the cache limit). If the MDS hits this threshold it reports the MDS_CACHE_OVERSIZED health warning, and if it cannot trim fast enough it may be killed by the OS before Ceph’s own backoff engages.

Tradeoff: raising the cache limit treats the symptom. If the workload genuinely needs more cache than the host can provide, you need more MDS ranks (raise max_mds) on separate hosts, or workload changes such as fewer simultaneous recursive operations.

No eligible standby

If ceph fs dump shows zero standbys, the filesystem is one daemon failure away from MDS_ALL_DOWN and you have hit it.

1. Start additional MDS daemons. They will register as standbys and the Monitor should immediately assign the failed rank.
2. Set standby_count_wanted to a sensible value (at least 1, ideally 2 for production) so the insufficient-standby check catches this before it becomes MDS_ALL_DOWN.
3. If you run multiple CephFS filesystems, verify standby affinity. mds_join_fs pins a standby to a specific filesystem. If all standbys are pinned to other filesystems and refuse_standby_for_another_fs is true (default false), the Monitor will not borrow them. You can temporarily set refuse_standby_for_another_fs to false, but the cleaner fix is to provision dedicated standbys per filesystem.

Standby-replay covers the wrong rank

With max_mds greater than 1, each active rank may have at most one standby-replay daemon, and that standby-replay follows only that rank’s journal. If rank 1 fails and only rank 0 has a standby-replay, rank 1 has no hot standby. A cold standby (if one exists) can still take over, but failover is slower because it must replay the journal from RADOS rather than from in-memory state.

1. Confirm standby-replay coverage per rank with ceph fs dump.
2. For multi-rank filesystems, ensure every active rank has a standby-replay, or keep enough cold standbys to cover the gap.

MDS daemon crashed (not OOM)

If the MDS exited due to an assertion, segfault, or other fatal error:

1. Check /var/log/ceph/ceph-mds.<daemon>.log for the backtrace or assertion message.
2. Use coredumpctl on the MDS host to fetch the core if one was produced.
3. If the crash is reproducible and no standby exists, you may need to restart the daemon and accept brief instability while it comes up. If it crashes again on journal replay, suspect journal corruption and treat this as a potential MDS_DAMAGED case.
4. ceph mds fail <role_or_gid> (a rank such as 0 or mds.0, or a daemon GID) can be used to force a daemon that is running but stuck to restart. Warning: this is disruptive; if the daemon was active and a standby is available, the failed daemon returns as a standby and the standby takes over.

MDS_DAMAGED co-condition

If MDS_DAMAGED is active alongside MDS_ALL_DOWN, the path is different. The damaged daemon refuses to rejoin to prevent making things worse.

1. Read the damaged daemon’s log carefully. It will identify which journal entry or metadata object triggered the damage report.
2. Do not force the damaged daemon to rejoin. The correct recovery path may involve ceph-recover-journal against the metadata pool.
3. If a clean standby exists, the Monitor may assign the rank to it, but the standby will itself refuse if the damage is in the shared journal. Treat MDS_DAMAGED as a metadata integrity incident and follow the CephFS damaged recovery procedure rather than the generic MDS_ALL_DOWN path.

Prevention

• Size mds_cache_memory_limit to the working set. The default 4GB is a starting point, not a rule. Track ceph_mds_mem_rss against the cache limit and against the host’s actual RAM budget. MDS memory is a stability requirement, not a performance optimization.
• Run enough standbys. Set standby_count_wanted to at least 1, ideally 2. For multi-rank filesystems, ensure every active rank has a standby-replay daemon. Standby-replay is rank-specific and will not fail over to a different rank.
• Watch the cap recall path. ceph_health_detail{name="MDS_CLIENT_RECALL"} and the ceph_mds_server_cap_revoke_eviction counter are leading indicators. Clients that hold caps and fail to return them drive cache pressure that ends in OOM.
• Monitor ceph_mds_mem_rss trends. A sustained upward trend is the precursor to the OOM loop that causes most MDS_ALL_DOWN incidents.
• Track MDS latency before it becomes an outage. ceph_mds_slow_reply and ceph_mds_reply_latency_sum / _count reveal MDS stress before the daemon crashes.
• Confirm MON health. MDS rank assignment depends on healthy MONs. Slow Paxos or quorum flapping can prevent the Monitor from assigning a rank even when daemons are available.
• Keep the metadata pool healthy. The MDS cannot start without its journal pool. Treat metadata pool PG degradation as seriously as MDS health.

How Netdata helps

• Per-second ceph_health_detail polling surfaces MDS_ALL_DOWN, FS_DEGRADED, and MDS_DAMAGED as distinct labeled signals, so you can tell a full outage from a degraded state or a corruption event without parsing CLI output.
• ceph_mds_mem_rss trending shows the memory ramp that precedes the OOM loop, the single most common MDS_ALL_DOWN root cause.
• Cap pressure signals (ceph_mds_caps, ceph_mds_inodes_with_caps, ceph_mds_server_cap_revoke_eviction, ceph_mds_server_session_recall_throttle, ceph_mds_server_global_recall_throttle) correlate with cache pressure, so you can identify the workload or client that pushed the MDS over.
• MDS latency counters (ceph_mds_reply_latency_sum / _count, ceph_mds_slow_reply) reveal rising metadata operation latency before it becomes a hang, giving you a window to act before failover is needed.
• Correlated MDS, metadata pool, and MON views in one timeline let you distinguish an MDS-local crash from a metadata-pool failure or a MON quorum problem.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators