CockroachDB disk space running out: capacity_available trends and the 20% rule

When a CockroachDB store runs out of disk space, the node cannot accept writes, cannot compact its LSM tree (the operation that would reclaim space), and enters a downward spiral that typically requires operator intervention. The capacity_available metric tracks remaining free space per store, but raw free space alone does not tell the full story. MVCC garbage, protected timestamps, compaction space amplification, and replica rebalancing all consume space faster than headline data growth suggests.

The 20% rule is the operational floor: keep at least 20% of each store’s capacity free at all times. CockroachDB documentation recommends this minimum, with 30% preferred to absorb compaction amplification and Raft snapshot staging. Below 15% free space, compaction may be unable to run, creating an unrecoverable loop where the system needs free space to free space.

This article covers how to read capacity_available trends, distinguish genuine data growth from MVCC garbage accumulation, estimate runway, and respond when a node approaches the threshold.

What this means

Disk space exhaustion in CockroachDB is an availability problem, not just a capacity problem. When a store fills past the point where compaction can function:

• Writes stop. Raft log entries cannot be persisted. The node loses leadership of its ranges.
• The cluster compensates by up-replicating. Surviving nodes receive new replicas, each requiring disk space. If those nodes are also low on space, one full node can cascade into multiple full nodes.
• Compaction stalls. Pebble needs temporary free space to merge SSTables across LSM levels. A worst-case compaction of the largest level can require free space equal to that level’s size. Without headroom, the LSM tree cannot be optimized, read amplification climbs, and the node degrades further.
• The node may not restart. If the WAL cannot be replayed due to insufficient space, the process cannot complete startup.

flowchart TD
    A[capacity_available declining] --> B{Free > 15%?}
    B -- No --> C[Compaction stalls]
    C --> D[L0 sublevels climb]
    D --> E[Write stalls]
    E --> F[Node loses Raft leases]
    F --> G[Cluster up-replicates ranges]
    G --> H[Remaining nodes lose free space]
    H --> I[Multi-node disk-full cascade]
    B -- Yes --> J[System stressed but stable]

The cascade risk is why disk space thresholds are set conservatively. A single node hitting 100% utilization is not an isolated incident. The cluster’s self-healing mechanism (up-replication) actively makes the problem worse by consuming space on other nodes to restore replica counts.

Common causes

Quick checks

All of these are read-only and safe to run at any time.

# Check capacity metrics per store
curl -s http://localhost:8080/_status/vars | grep -E '^capacity'

# OS-level disk usage for the store directory
df -h /path/to/cockroach-data

# Compare live data to total bytes to estimate MVCC garbage ratio
curl -s http://localhost:8080/_status/vars | grep -E 'live_bytes|key_bytes|value_bytes'

# Check for protected timestamp records blocking GC
curl -s http://localhost:8080/_status/vars | grep -i protected

# Check compaction health
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

# Check for under-replicated ranges (indicates recovery in progress)
curl -s http://localhost:8080/_status/vars | grep ranges_underreplicated

How to diagnose it

1. Identify which stores are approaching the threshold. Check capacity_available per store. The metric is per-store, not per-node. A node with multiple stores can have one store near capacity while another has ample headroom.

2. Determine whether growth is data or garbage. Compare live_bytes to capacity_used. If live_bytes is stable but capacity_used is climbing, the growth is MVCC garbage: old versions and tombstones that have not yet been compacted away. This distinction determines your response. Garbage accumulation is reversible by fixing the GC blocker. Genuine data growth requires adding capacity.

3. Check for protected timestamp records. Protected timestamps from CDC changefeeds, backups, and other features prevent MVCC garbage collection below their timestamp. A stalled changefeed or hung backup creates a protected timestamp that silently blocks GC. Disk usage grows unboundedly while live_bytes stays flat. This is one of the most insidious failure modes in CockroachDB because it is silent until the disk is full. Check spanconfig_kvsubscriber_protected_record_count and jobs_changefeed_protected_age_sec.

4. Check compaction health. If storage_l0_sublevels is elevated, compaction is falling behind. Disk I/O is saturated or disk space is too low for compaction to proceed. L0 growth and disk space exhaustion compound each other: high L0 consumes more disk, and low disk prevents compaction from reducing L0.

5. Estimate runway. Use linear extrapolation of capacity_available decline rate, but account for MVCC garbage separately. If GC is keeping up, the decline rate reflects true data growth. If GC is blocked (protected timestamps, GC TTL too long), the decline rate is inflated by garbage accumulation and will stabilize once the blocker is removed. Time to 80% utilization at the current rate gives your operational window. Anything under 30 days warrants a capacity plan.

6. Check for cascade conditions. If ranges_underreplicated is nonzero and the cluster is actively up-replicating, surviving nodes are receiving replica snapshots that consume disk space. Verify that target nodes have adequate free space before the cluster fills them trying to heal.

Metrics and signals to monitor

Fixes

Emergency: node has already filled up

If a node has shut down due to disk exhaustion, look for an emergency ballast file in the node’s storage directory. If one was pre-created (either manually via cockroach debug ballast or automatically during a prior startup), deleting it frees reserved space that may allow the node to start.

Warning: Only delete the ballast file on a stopped node, and only when the disk is genuinely full. Deleting it on a running node removes your emergency reserve.

Once the node is back, add permanent storage or reduce data immediately. The node will recreate the ballast file on the next successful startup, consuming the space you just freed.

MVCC garbage is the culprit

If live_bytes is flat but capacity_used is climbing, the growth is garbage. Steps:

1. Check for stalled or failed changefeed jobs and backups. Cancel or resume them to release protected timestamp records that are blocking GC.
2. Verify GC TTL (gc.ttlseconds) is not set excessively high. Longer TTL means more historical versions retained on disk.
3. Wait for GC to run. It is asynchronous. Monitor live_bytes to capacity_used convergence over the following hours.
4. After GC releases old versions, compaction must still run to reclaim physical space. Monitor storage_l0_sublevels to confirm compaction is progressing and not blocked by remaining space constraints.

Compaction blocked by low space

If the store is so full that compaction cannot proceed, the system is in a feedback loop. Options:

1. Free space by any means available: delete the ballast file, clean up temporary files, rotate logs.
2. Reduce write rate to slow L0 growth while compaction catches up. Pause bulk operations, tighten admission control.
3. If the node is part of a healthy cluster, consider decommissioning it to move data to nodes with more capacity. Verify that target nodes have adequate free space first; the allocator avoids stores that are effectively full, but it does not reserve headroom for future growth.

Genuine capacity exhaustion

If live_bytes is growing proportionally to capacity_used, the data is genuinely outgrowing the provisioned storage:

1. Expand the volume if the storage backend supports online expansion.
2. Add nodes to the cluster. New nodes receive rebalanced replicas, spreading data across more storage.
3. Review data retention. If old data can be archived or dropped, do so. Note: DROP TABLE and large DELETE operations do not immediately free space. MVCC tombstones must propagate through compaction before physical space is reclaimed. A large DELETE can temporarily increase disk usage before it decreases.

Prevention

• Alert on capacity_available per store at two thresholds. Ticket at 20% free with a downward trend. Page at 10% free and declining. These thresholds give hours to days of runway before cascade risk becomes acute.
• Track the live_bytes to capacity_used ratio over time. A diverging ratio means garbage is accumulating faster than compaction can reclaim it. Catch this before it fills the disk.
• Monitor protected timestamp records. A stalled changefeed or hung backup can silently block GC indefinitely. Alert on protected timestamp age above 24 hours when no active job explains it.
• Monitor L0 sublevels alongside disk usage. Elevated L0 means compaction is falling behind, which increases disk space pressure and requires more free space to resolve.
• Plan capacity with linear extrapolation. Project capacity_available decline using the current growth rate. Account for MVCC garbage by tracking live_bytes growth as the true minimum floor. If the garbage-adjusted runway is under 30 days, plan storage expansion.
• Verify free space before decommissioning nodes. The cluster moves replicas to surviving nodes during decommission. Ensure each surviving node has enough capacity to absorb the additional data plus compaction headroom.

How Netdata helps

• Per-second capacity_available tracking per store lets you compute depletion rates with enough resolution to catch sudden growth spikes that 15-minute snapshots miss.
• Correlate capacity with L0 sublevels, write stalls, and MVCC garbage metrics on the same timeline. When disk usage spikes, seeing whether L0 is also climbing tells you immediately whether compaction is the bottleneck or data growth is the driver.
• Protected timestamp age and changefeed lag displayed alongside disk usage trends surface the silent GC-stall pattern before it fills the disk.
• ML anomaly detection on capacity_used growth rate flags acceleration that static thresholds miss. A store growing 1 GB per day that suddenly starts growing 5 GB per day is an early warning, even if absolute free space is still above 20%.
• Per-store granularity ensures you see the one store approaching capacity even when the cluster aggregate looks healthy.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes