Cassandra disk space exhaustion: emergency recovery when the data volume fills

A Cassandra node that runs out of disk space does not degrade gracefully. Compaction halts because it cannot allocate temporary space to merge SSTables. Old SSTables are never deleted. Writes append to the commitlog until segment allocation blocks. At that point the node rejects mutations and CommitLog.WaitingOnSegmentAllocation climbs. You may see No space left on device errors while the data volume still reports a few percent free, because Cassandra’s internal headroom requirements are stricter than the filesystem.

If you are responding to a PAGE for disk exhaustion, the node likely has less than 10% free space and write-path impact is confirmed. This runbook identifies what is consuming space, reclaims it safely without restarting the node, and gets compaction moving again.

What this means

Cassandra appends every write to the commitlog and inserts it into a memtable. When memtables flush, they become immutable SSTables. Compaction merges SSTables in the background to purge tombstones and reclaim space. That merge requires temporary disk space roughly equal to the size of the input SSTables.

When the filesystem fills, compaction cannot reserve temporary space. Without compaction, old SSTables remain on disk. New flushes and commitlog segments continue to consume space. Eventually the commitlog cannot allocate new segments. The node blocks writes, increments WaitingOnSegmentAllocation, and returns failures to clients.

flowchart TD
    A[Disk usage exceeds compaction headroom] --> B[Compaction cannot reserve temp space]
    B --> C[Compaction stalls]
    C --> D[Old SSTables are never deleted]
    D --> E[Disk usage rises from flushes and commitlog]
    E --> F[Commitlog segment allocation blocks]
    F --> G[Writes are rejected]
    G --> H[Hints accumulate on coordinators]

Common causes

Quick checks

Run these read-only checks to assess scope before making changes.

# Compare Cassandra data load to filesystem free space
nodetool info | grep "Load"
df -h /var/lib/cassandra/data

# Check commitlog pressure
nodetool info | grep -i "Commit Log"

# List snapshots to identify hard-linked space consumers
nodetool listsnapshots

# Check hints size on this node
du -sh /var/lib/cassandra/hints/

# Find the largest keyspaces by disk usage
du -sh /var/lib/cassandra/data/*/

# Check compaction status and pending tasks
nodetool compactionstats

# Search logs for disk-full errors
grep -i "no space left\|insufficient disk space" /var/log/cassandra/system.log

How to diagnose it

1. Confirm filesystem utilization. Use df -h on the data directory, commitlog directory, and hints directory. If any mount is above 90% full, treat it as critical.
2. Compare Cassandra’s Load metric to filesystem usage. Run nodetool info | grep Load. If df shows far more consumption than Load, the difference is likely snapshots, hints, commitlog segments, or compaction temporary files.
3. Identify snapshot bloat. Run nodetool listsnapshots. Snapshots share hard links with live SSTables, but they prevent old SSTables from being deleted after compaction. The True size field shows actual unique data; Size on disk shows the hard-link footprint. Even if df looks manageable, snapshots can tip a node over during a major compaction.
4. Check the commitlog allocation state. Look for WaitingOnSegmentAllocation > 0 via JMX or your monitoring agent. This confirms the write path is blocked by disk space, not by GC pressure or network partition.
5. Review compaction status. Run nodetool compactionstats. If pending tasks are high but active compactions are zero, the compaction executor is stalled waiting for temp space. This is the signature of disk exhaustion with STCS.
6. Inspect hints. Run du -sh /var/lib/cassandra/hints/. If hints are large, a recent node outage caused coordinators to buffer writes locally. Hints replay will add more write pressure once the target returns.
7. Check for write rejections. Look in system.log for No space left on device. Correlate with client write timeout and unavailable metrics to confirm user impact.

Metrics and signals to monitor

Fixes

Clear snapshots

Snapshots are the fastest way to reclaim space because they use hard links. Removing them deletes the hard links but does not touch live data. Only clear snapshots after verifying that backups are complete and retained elsewhere.

# Remove all snapshots. Destructive if you rely on these for backups.
nodetool clearsnapshot --all

Tradeoff: If snapshots are your only backup, this is irreversible. Ensure backups are stored off-node first.

Clear saved caches

Saved caches are non-critical and rebuilt automatically on restart.

# Safe to delete; Cassandra rebuilds these on restart.
rm -rf /var/lib/cassandra/saved_caches/*

Address hint accumulation

Hints consume disk space on coordinators. If the target node is back online, hints replay automatically and free space as they complete. If the target is permanently decommissioned, remove its hint files after confirming the node will not rejoin. Do not delete active hint files for a downed target; if you do, run a full repair once it returns to restore consistency.

Add storage

If the volume can be expanded through LVM, cloud block storage, or a new mount, expand the filesystem. After space is added, compaction should resume automatically. Monitor nodetool compactionstats to confirm active compactions restart.

Reduce compaction pressure temporarily

If you are on STCS and cannot add storage immediately, do not force a major compaction because it requires temporary space equal to the current data size. Focus on removing snapshots and saved caches first. If you must change compaction strategy to reduce space amplification later, schedule that for a maintenance window after the crisis is resolved.

Prevention

• Monitor filesystem free space directly with df, not only the Load metric. Load excludes commitlog, hints, snapshots, and compaction temp files.
• Automate snapshot cleanup. Verify backups, then schedule nodetool clearsnapshot or automate via your backup tool.
• Monitor the hints directory size on all nodes. Hints should be near zero when all nodes are healthy.
• Separate commitlog and data directories onto different mounts. Sharing a device accelerates exhaustion and adds I/O contention.
• Size disks according to compaction strategy. Maintain greater than 50% free for STCS, greater than 30% for LCS or TWCS, plus a buffer for snapshots and hints.
• Track PendingCompactions and SSTable count as leading indicators. A rising trend predicts disk exhaustion days before it happens.

How Netdata helps

• Correlates disk space usage on the data mount with CommitLog.WaitingOnSegmentAllocation and Compaction.PendingTasks to surface write-path blockage before the disk hits 100%.
• Tracks per-table SSTable counts and compaction backlog alongside filesystem metrics, distinguishing a compaction stall from simple data growth.
• Alerts on commitlog pending tasks and storage exceptions using composite patterns, reducing noise from transient spikes.
• Disk I/O utilization per device confirms whether commitlog and data share a saturated disk.

Related guides

• Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS
• Cassandra compaction death spiral: when writes outrun compaction throughput
• Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM
• Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones
• Cassandra GC death spiral: long pauses, gossip flapping, and recovery
• Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses
• Cassandra heap pressure: sizing the JVM heap and tuning G1GC
• Cassandra monitoring checklist: the signals every production cluster needs
• Cassandra monitoring maturity model: from survival to expert
• Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery
• Cassandra pending compactions growing: the compaction backlog runbook
• Cassandra Scanned over N tombstones warning: finding the offending query