CockroachDB store is full: emergency disk recovery and why deletes don't free space

A CockroachDB node’s store directory is full or nearly full, and the node may have already shut itself down to protect data integrity.

Deleting data will not help immediately. CockroachDB uses MVCC: a SQL DELETE writes tombstones at new timestamps rather than removing old data. Until those tombstones propagate through Pebble compaction, the old versions stay on disk. A large-scale delete can temporarily increase disk usage because the tombstones are written before old data is reclaimed.

The underlying problem is usually silent accumulation: MVCC garbage that GC never reclaimed, protected timestamps from stalled changefeeds or hung backups that blocked GC entirely, or capacity exhaustion that nobody was watching.

What this means

CockroachDB needs free disk space for three concurrent activities: Pebble compaction (which merges SSTables and reclaims obsolete data), Raft snapshot staging (up to 512 MiB per range during rebalancing), and WAL writes. When available space drops below approximately 15%, compaction loses the working room it needs to merge files. Below 10%, the system enters a death spiral it may not escape.

CockroachDB has built-in store fullness thresholds. At 92.5% utilization, the allocator stops rebalancing new replicas to that store. At 95%, it begins actively moving replicas away with high priority. When the storage engine can no longer write — a filesystem disk-full error, or available space below half the ballast size (512 MiB with the default 1 GiB ballast) — the node shuts down to protect data integrity.

Once a node shuts down due to disk pressure, you cannot simply restart it. WAL replay and compaction startup need free space that does not exist.

flowchart TD
    A[Store fills past 85%] --> B[Compaction space amplification spikes]
    B --> C[Compaction throughput drops]
    C --> D[L0 sublevels accumulate]
    D --> E[Reads slow, compaction slows further]
    E --> C
    D --> F[Free space drops below 15%]
    F --> G[Compaction cannot obtain working space]
    G --> H[Write stalls begin]
    H --> I[Node cannot commit Raft entries]
    I --> J[Liveness lost, leases transferred away]

The feedback loop is the core problem. Compaction reclaims space, but compaction itself needs free space to write merged output. When space is exhausted, the fix mechanism is also blocked.

Common causes

Quick checks

Run these read-only checks on a node that is still responding:

# Check capacity per store from the metrics endpoint
curl -s http://localhost:8080/_status/vars | grep -E 'capacity'

# OS-level disk space on the store directory
df -h /path/to/cockroach-data

# Check L0 sublevels (compaction health)
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

# Check write stalls (most severe storage signal)
curl -s http://localhost:8080/_status/vars | grep storage_write_stalls

# Check read amplification (compaction debt indicator)
curl -s http://localhost:8080/_status/vars | grep rocksdb_read_amplification

# Check protected timestamp records (GC blockers)
curl -s http://localhost:8080/_status/vars | grep -E 'protected_record_count|protected_age_sec'

# Check if the node is still running
curl -s http://localhost:8080/_status/vars | grep sys_uptime

If the node has already shut down, the metrics endpoint will not respond. Fall back to df -h on the store directory from the host OS.

How to diagnose it

1. Determine which stores are critical. Check capacity_available per store. A node with multiple stores can have one at 98% and another at 60%. Focus recovery on the critical store.

2. Separate data growth from garbage. If capacity_used is growing but live data is stable, the problem is MVCC garbage or protected timestamps, not data volume. Check the ratio of live bytes to total bytes per range to see how much on-disk data is garbage awaiting collection.

3. Check for protected timestamps. This is the most insidious cause. A stalled changefeed or hung backup creates a protected timestamp record that prevents MVCC GC below that timestamp. Disk grows silently. Check spanconfig_kvsubscriber_protected_record_count and jobs_changefeed_protected_age_sec. If protected age exceeds 24 hours, investigate the owning job immediately.

4. Assess compaction health. Check storage_l0_sublevels. Above 20 sustained means the store is already in or approaching a write stall. Check storage_write_stalls for active stall events. If L0 is high but decreasing, compaction is catching up. If it is high and rising, the death spiral is active and you need to reduce write pressure.

5. Identify space consumers. If the node is still running, check which tables or ranges consume the most space. Large DELETE or UPDATE operations create tombstone pressure that temporarily inflates on-disk size before compaction reclaims it.

Metrics and signals to monitor

Fixes

Delete the emergency ballast file

CockroachDB creates an emergency ballast file in each node’s store directory as a space reservation. The file exists specifically so that deleting it frees enough space to recover a node that shut down due to a full disk.

# Find the ballast file in the store directory
find /path/to/cockroach-data -name '*ballast*' -ls

# Delete it to free emergency space
# Non-destructive: this is a pre-allocated placeholder, not database content
rm /path/to/cockroach-data/<ballast_file_name>

This is non-destructive to data. The ballast file is a pre-allocated placeholder, not database content. After deleting it, the node should have enough free space to start and run compaction.

Once the node recovers, CockroachDB recreates the ballast automatically during periodic capacity calculations when there is sufficient free space (at least 4x the ballast size, or when more than 10 GiB would remain free). The node is not fully recovered until the ballast is reestablished.

Lower gc.ttlseconds and force garbage collection

If MVCC garbage is the problem, reducing the GC TTL accelerates reclamation. The gc.ttlseconds zone configuration parameter determines how long old MVCC versions are retained before becoming eligible for GC.

-- Lower GC TTL to accelerate garbage reclamation
-- WARNING: This reduces your ability to run AS OF SYSTEM TIME queries
ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 600;

After changing the setting, the MVCC GC queue processes eligible ranges automatically, but pace depends on GC queue throughput.

For emergency recovery after dropping a large table, the CockroachDB operational FAQ documents a manual procedure:

1. Lower gc.ttlseconds to 600 on the relevant zone.
2. Capture range IDs before dropping: SHOW RANGES FROM TABLE your_table;
3. Drop the table.
4. Manually enqueue each range in the MVCC GC queue via the DB Console Advanced Debug page with SkipShouldQueue checked.

This is aggressive and tedious for large tables. Use it only when standard GC cannot keep pace with the emergency.

Cancel stalled changefeeds and hung backups

If protected timestamps are blocking GC, address the owning job. A stalled changefeed keeps a protected timestamp that prevents all GC below its timestamp. Disk grows unboundedly while every other cluster metric looks healthy.

-- Check changefeed job health
SELECT job_id, status, running_status, created
FROM crdb_internal.jobs
WHERE job_type = 'CHANGEFEED'
  AND status NOT IN ('succeeded', 'canceled')
ORDER BY created DESC;

-- Cancel a stalled changefeed
-- WARNING: Disruptive to downstream consumers
CANCEL JOB <job_id>;

After canceling, the protected timestamp is released and MVCC GC resumes. Monitor spanconfig_kvsubscriber_protected_record_count to confirm the record is removed and watch disk usage decline as compaction reclaims the freed garbage.

Add storage capacity

The cleanest long-term fix. Add a new store to the node (if the deployment supports it) or add new nodes to the cluster. New nodes receive replicas via rebalancing, redistributing data and relieving pressure on the full store.

Adding capacity takes time. The cluster must generate and transfer Raft snapshots (up to 512 MiB per range) to the new storage. During this process, the recovering store still needs free space to operate. Start this process early, not when the store is already at 95%.

Run offline compaction

The cockroach debug compact command performs manual compaction on a stopped node. It can reclaim space from obsolete SSTables that online compaction has not yet processed.

# WARNING: Offline tool. The node must be stopped first.
# Stopping the node means its replicas lose a member and the cluster
# begins rebalancing. Use only when the node cannot start due to disk
# pressure and ballast file deletion was insufficient.
cockroach debug compact /path/to/cockroach-data

The tool has minimal logging output. Monitor the store directory size from the OS during the operation to confirm progress.

Prevention

• Alert on capacity_available below 20% with an upward trend. Below 15%, compaction loses working space and the death spiral begins. Do not wait for the paging alert at 10%.

• Monitor MVCC garbage as a ratio. Track total bytes minus live bytes relative to total bytes. If garbage exceeds 20-30% of total store size, GC is falling behind and accumulating a disk debt.

• Monitor protected timestamp records. Check spanconfig_kvsubscriber_protected_record_count and jobs_changefeed_protected_age_sec. Protected age above 24 hours without an active operation is a silent disk growth emergency that will not trigger any other alarm.

• Size for 30% free space minimum. CockroachDB documentation recommends at least 20% free at all times. 30% provides headroom for compaction space amplification and Raft snapshot staging during rebalancing.

• Watch L0 sublevels proactively. Sustained values above 5-10 mean compaction is lagging. This is the earliest warning that the LSM tree is accumulating debt.

• Audit changefeed health regularly. A stalled changefeed is the single most insidious cause of unbounded disk growth because it silently blocks GC while the cluster appears healthy from every other angle.

How Netdata helps

• Per-second capacity tracking on capacity_available, capacity_used, and total capacity per store reveals the exact inflection point where disk growth accelerates, not a sampled snapshot every 15 or 30 seconds.

• L0 sublevel correlation with disk space exposes the feedback loop between compaction debt and space exhaustion. If storage_l0_sublevels is climbing while capacity_available drops, you are watching the death spiral form in real time.

• Protected timestamp and changefeed age metrics surface the silent GC-blocker pattern. Correlating spanconfig_kvsubscriber_protected_record_count with MVCC garbage growth and disk usage trends identifies the root cause before the disk is full.

• Write stall detection at per-second resolution catches sub-minute stall events that standard scrape intervals miss.

• ML anomaly detection on disk usage growth rates catches gradual capacity exhaustion days or weeks before the critical threshold fires.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert