Ceph MDS_CACHE_OVERSIZED: cache above limit and the cap recall that follows

Health-check names verified: ceph health detail reports these as MDS_CACHE_OVERSIZED and MDS_CLIENT_RECALL (with MDS_CLIENT_RECALL_MANY when many clients are affected); the MDS_HEALTH_* forms are the internal enum names in the Ceph source (mds_metric_name), not the displayed codes.

MDS_CACHE_OVERSIZED fires when an active Metadata Server daemon’s in-memory inode and capability cache crosses mds_cache_memory_limit * mds_health_cache_threshold. It is almost always workload-driven: a client has touched enough files in a short enough window that the MDS is caching more metadata than its budget allows.

The warning itself is benign for a few seconds. What matters is the cascade. To shrink its cache, the MDS revokes client capabilities (caps), one batch per session per second, capped by mds_recall_max_caps (default 30,000). If clients keep acquiring caps on new inodes faster than the MDS can recall them, you see a second warning (MDS_CLIENT_RECALL), the recall throttle counters climb, ceph_mds_mem_rss keeps rising, and at the end of the path sits client eviction.

The terminal state is not the cache warning. It is one or more CephFS clients blacklisted because they failed to return enough caps within session_timeout. From the client side that looks like I/O suddenly failing with EBLACKLISTED. From the operator side it looks like a “slow client” eviction log entry, hours after the original MDS_CACHE_OVERSIZED was dismissed as noise.

What this means

The MDS keeps the hot portion of the CephFS namespace in RAM: inodes, dentries, directory fragments, open file metadata, and the per-client capabilities that allow clients to cache reads and writes safely. Every inode cached on the MDS side typically has matching state on one or more clients, gated by caps. The cache is not an MDS-local structure; it is a distributed cache coordinated through cap messages.

mds_cache_memory_limit (default 4 GiB as of Reef/Squid) is a soft budget. The MDS does not hard-stop at it; it starts trimming. The actual recall trigger is mds_cache_reservation (default 0.05, meaning recall begins at 95% of the limit). When the cache crosses 95%, the MDS starts asking clients to return caps so it can evict the corresponding inodes. mds_health_cache_threshold (default 1.5) is only the warning threshold: if the cache still reaches 150% of the limit, the MDS_CACHE_OVERSIZED health check fires.

That gap, between 95% and 150%, is where most incidents actually live. By the time the warning fires, the recall machinery has already been running flat-out and losing. Two more signals usually accompany it:

• MDS_CLIENT_RECALL active, meaning a client is failing to release caps at the rate the MDS wants.
• ceph_mds_server_session_recall_throttle and ceph_mds_server_global_recall_throttle counters rising, meaning the MDS is bumping into its own recall throttles (mds_recall_max_caps per session, decay-limited globally).

If the gap is not closed, ceph_mds_server_cap_revoke_eviction ticks upward as the MDS evicts clients that exceed mds_max_caps_per_client (default 1M caps) or miss session_timeout (default 60s).

flowchart TD
    A["Client scans tree
find, rsync, du"] --> B["Cache crosses
95% of limit"]
    B --> C["MDS recalls caps
mds_recall_max_caps per session"]
    C --> D{"Recall rate beats
acquisition rate?"}
    D -- yes --> E["Cache shrinks
warning clears"]
    D -- no --> F["MDS_CACHE_OVERSIZED fires
at 150% of limit"]
    F --> G["MDS_CLIENT_RECALL fires
recall_throttle counters rise"]
    G --> H["Client crosses
mds_max_caps_per_client or
misses session_timeout"]
    H --> I["Client evicted
EBLACKLISTED on client"]

The fork at “Recall rate beats acquisition rate?” is the decision point. The rest of the article is about determining which side you are on and what to do about it.

Common causes

Quick checks

Run these read-only on the cluster and on the active MDS host. None of them modify state.

Both forms are valid on current releases: ceph tell mds.<id> ... is routed through the monitors and works from any host (the daemon must be up); ceph daemon mds.<id> ... connects to the admin socket directly and therefore only works on the MDS host while the daemon is running. The mix below uses whichever form suits the context.

# Top-level: is the warning still active, and which daemon is firing it?
ceph health detail | grep -E 'MDS_CACHE_OVERSIZED|MDS_CLIENT_RECALL'

# Cache size and inode pressure for the MDS
ceph tell mds.<id> cache status
ceph tell mds.<id> perf dump | jq '.mds_mem'

# Live sessions, sorted by caps held (look for the dominant client)
ceph tell mds.<id> session ls | jq 'sort_by(-.num_caps) | .[] | {id, num_caps, recalled_caps}'

# Effective config on this daemon (compare against what you think you set)
ceph config get mds mds_cache_memory_limit
ceph config get mds mds_cache_reservation
ceph config get mds mds_recall_max_caps
ceph config get mds mds_max_caps_per_client

# Confirm the daemon rank: is this the active MDS, or standby-replay?
ceph fs status

# Process memory for ground truth (rss in KB)
ps -eo pid,rss,args | grep ceph-mds

If the warning daemon is standby-replay and ceph health detail reports zero inodes in use, skip to the “Standby-replay false positive” subsection. This is a documented false positive on Quincy (17.2.x) and earlier, tracked in Red Hat Bugzilla #1951348 and #1944148 (both open and describe exactly this standby-replay oversized-cache warning).

How to diagnose it

1. Confirm the cache is actually oversized on an active MDS. Cross-check ceph tell mds.<id> cache status against mds_cache_memory_limit. If the daemon is standby-replay, treat it as the false-positive case regardless of the cache size value.
2. Find the dominant session. Sort session ls by num_caps. In the scan-driven case, one or two client addresses account for the bulk of the caps. Note those IPs.
3. Check whether recall is making progress. Pull ceph_mds_caps, ceph_mds_inodes_with_caps, ceph_mds_server_session_recall_throttle, and ceph_mds_server_global_recall_throttle over a few minutes. If ceph_mds_caps is flat or rising while the throttle counters rise, recall is losing.
4. Look for the eviction precursor. Check increase(ceph_mds_server_cap_revoke_eviction[5m]). Any non-zero value means the MDS has already evicted at least one client; expect EBLACKLISTED errors on the client side and entries in ceph osd blocklist ls.
5. Identify the workload. The dominant client IP from step 2 should map to a host you can inspect. The usual suspects are find /, recursive rsync, backup jobs, indexing services, and CI artifact pipelines. A short perf dump baseline before and after the job starts shows the cap acquisition rate.
6. Rule out the misconfiguration cases. Verify mds_cache_memory_limit is what you expect, especially under Rook, where a pod memory request without a limit does not propagate to the MDS. The behavior is tracked as rook/rook#8143 (“MDS configuration mds_cache_memory_limit is not set as expected”).

Metrics and signals to monitor

The signal names below are Netdata collector metric names (out of scope for this review); the underlying Ceph mds_server perf counters they map to were verified in the MDS source: cap_revoke_eviction is a named counter, and per-session and global recall throttles exist in Server::recall_client_state.

Fixes

The fix is one of: raise the limit because the working set genuinely needs it, change the workload so it does not pin the whole namespace, or address the specific edge case (slow client, standby-replay false positive, Rook misconfiguration). Restarting the MDS is almost never the right first move. Note that in Squid (19.2.0+) ceph mds fail and ceph fs fail require the --yes-i-really-mean-it confirmation flag when the MDS is active with a MDS_TRIM or MDS_CACHE_OVERSIZED health warning.

Raise mds_cache_memory_limit for a genuine working set

If ceph_mds_inodes_with_caps is consistently near the limit outside of any scan, and MDS_CLIENT_RECALL is steady background noise rather than an escalation, the working set is larger than the budget. Cache-related options are runtime-updatable; no MDS restart is required.

# Inspect current value
ceph config get mds mds_cache_memory_limit

# Raise it (example: 8 GiB). Apply live.
ceph config set mds mds_cache_memory_limit 8589934592

Two cautions. First, the MDS host must have the RAM. ceph_mds_mem_rss should track the new limit within a few minutes; if it does not, the config did not apply (re-check with ceph config get mds ... on the host). Second, under Rook, set the value through the CephFilesystem CRD’s metadataServer.resources.limits.memory, not only via ceph config set, otherwise the next reconcile may reset it.

Throttle the scan, do not throttle the MDS

If a find, rsync, or backup job is the trigger, the right fix is on the workload side. The variable that matters is the metadata-op rate, not CPU or data-IO priority, so nice/ionice alone usually do not help; you need to slow the directory-walk rate itself.

• Add a small sleep between directory descents in scan scripts (find ... -exec sh -c '...' \; with a delay, or a rate-limited walker). This is the only reliable way to drop the cap acquisition rate below the recall rate.
• Replace rsync -a /src/ /dst/ over the whole tree with targeted syncs, or use --update plus a manifest to avoid re-scanning unchanged subtrees.
• For backup systems that walk CephFS, prefer CephFS snapshots (ceph fs snap) as the backup source instead of a live walk. Snapshots do not pin caps the same way.
• Schedule scans during low client-count windows so per-client cap pressure is bounded by mds_max_caps_per_client.

If you must tune the MDS to absorb a known one-time scan, raise mds_recall_max_caps (per-session recall batch size, default 30,000). This lets the MDS recall more caps per session per second, which can close the gap on scan-driven incidents. The tradeoff is more recall traffic on the MDS-to-client link.

# Increase per-session recall batch (example: 50,000)
ceph config set mds mds_recall_max_caps 50000

Tune in small steps and watch ceph_mds_server_session_recall_throttle fall while ceph_mds_caps falls.

There is no definitive upstream guidance on tuning mds_recall_max_decay_threshold (default 128K) for very large directory trees; it is a decay threshold on the per-session recall throttle that operator mailing lists do tune, so treat it as an experimental lever and validate on a test workload.

Address a slow or stuck client

If session ls shows one or two clients with large recalled_caps counts and old recall timestamps, the MDS is waiting on them. Investigate the client host:

• Kernel CephFS client: check dmesg for cap-related stalls, look at client memory pressure, verify the client is not itself under OOM.
• FUSE client: check the client process CPU and memory.
• Network: cap recall messages share the same path as data. Packet loss or saturation between the client and the MDS will manifest as slow recall.

If the client is genuinely wedged, manual eviction is a last resort. The client will see EBLACKLISTED; clear the blocklist entry once the client has remounted and the workload is no longer stuck.

Verified syntax: ceph tell mds.<id> session evict <client-id> (the session evict admin-socket command takes a client identifier).

Standby-replay false positive

If ceph health detail shows MDS_CACHE_OVERSIZED on a standby-replay daemon with “0 inodes in use by clients, 0 stray files,” this is the documented false positive. The standby-replay daemon replays the active MDS’s journal and accumulates metadata without having any clients to recall caps from, so it cannot trim. Restarting the standby-replay MDS clears the warning with no client impact:

Verified unit names: under classic deployment this is `ceph-mds@<daemon-id>.service`; under cephadm it is `ceph-<fsid>@mds.<host>.<name>`. Adjust to your deployment.

# Safe: only the standby-replay daemon is affected. Active MDS continues serving.
systemctl restart ceph-mds@<standby-replay-daemon-id>

If this fires repeatedly, the underlying issue is tracked at tracker.ceph.com/issues/48673 (“High memory usage on standby replay MDS”); consider reducing journal size or filing a bug against your Ceph version.

Prevention

• Size mds_cache_memory_limit for the working set, not the average. Measure ceph_mds_inodes_with_caps during peak load and set the limit with at least 30% headroom. The 4 GiB default assumes a modest namespace; large CephFS deployments routinely run at 16 to 32 GiB.
• Alert on ceph_mds_mem_rss trending toward host RAM, not just on MDS_CACHE_OVERSIZED. The warning fires at 150% of the configured cache limit; OOM fires at host RAM. The gap between those two is your runway.
• Track ceph_mds_server_cap_revoke_eviction as a counter and alert on any increase. Evictions are user-visible incidents; they should never be a surprise.
• Gate backup and indexing jobs that walk CephFS. Treat any process that does find / or rsync / against CephFS the way you would treat a full table scan in a database: require an approval, schedule it, and rate-limit the walk.
• If you run multi-active MDS (max_mds > 1), monitor per-rank cache pressure separately. Subtree partitioning does not guarantee balanced cap distribution; one rank can hit MDS_CACHE_OVERSIZED while others are idle.
• Under Rook, set both requests and limits for the MDS pod. Verify the effective mds_cache_memory_limit after deploy, not just the CRD value.
• Review recall throttle settings when the cluster’s client count or namespace grows. Defaults tuned for a small cluster will starve recall on a large one.

How Netdata helps

• The ceph_health_detail metric with the name label lets you alert on MDS_HEALTH_CACHE_OVERSIZED and MDS_CLIENT_RECALL independently. Both firing on the same MDS rank is the strongest signal that the cascade has started.
• ceph_mds_caps and ceph_mds_inodes_with_caps per MDS daemon show the distributed cache size directly. A rising curve against a flat mds_cache_memory_limit is the leading indicator, before the health check fires.
• ceph_mds_server_session_recall_throttle and ceph_mds_server_global_recall_throttle counters, viewed as rates, show whether the recall machinery is saturated. Flat ceph_mds_caps with rising throttle counters is the “recall is losing” signature.
• ceph_mds_server_cap_revoke_eviction viewed as increase(...[5m]) is the eviction alarm. Any non-zero value means clients are already seeing EBLACKLISTED.
• Correlating ceph_mds_reply_latency and ceph_mds_slow_reply against the cache-pressure metrics separates an MDS that is merely hot from one that is stuck behind its own recall throttle.
• ceph_mds_mem_rss next to the host’s available memory gives the OOM runway directly. Per-second resolution catches the sharp ramp that precedes an OOM kill, which the 150%-of-limit health check does not see.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural