CockroachDB MVCC garbage growing: tombstones, GC lag, and silent disk growth

MVCC garbage accumulation produces no errors, no latency spikes, and no user-visible symptoms until the disk fills. By then, you may be hours away from a compaction death spiral that takes the node or cluster offline.

CockroachDB retains old MVCC versions of every key until garbage collection removes them after the GC TTL window (default 25 hours through v22.2; 4 hours for new clusters since v23.1). Deletes and updates create tombstones: markers that a key was removed at a specific timestamp. These tombstones persist in the LSM tree until MVCC GC runs and Pebble compaction physically removes them downstream. If anything blocks that cleanup, dead data accumulates silently.

The most common blocker is a protected timestamp record. CDC changefeeds, backup jobs, and certain long-running operations create these records to prevent GC from reclaiming versions they still need. If the owning job stalls, fails, or is paused without releasing the record, MVCC GC cannot progress past the protected timestamp. Garbage accumulates without bound.

This article covers how to detect MVCC garbage growth early, identify the blocker, and reclaim space before disk pressure cascades into a storage failure.

What this means

MVCC garbage is dead data: old key versions and tombstones that have aged past the GC TTL but have not yet been physically removed. The on-disk footprint of a range includes both live bytes (the current, readable versions) and total bytes (live plus garbage). When garbage grows, the gap between live bytes and total bytes widens.

The MVCC GC queue processes ranges periodically. For each range, it computes a GC threshold: the current time minus the GC TTL. It then checks for protected timestamp records covering that range. If a protected timestamp exists, the effective GC threshold is pushed forward to the protected timestamp, meaning GC cannot reclaim anything older than that point. The garbage below the protected timestamp stays on disk.

This is by design for active backups and changefeeds: they need historical versions to produce correct incremental output. The problem arises when the owning job is no longer making progress but has not released its protected timestamp. Common scenarios include a paused changefeed, a backup stuck retrying against an unreachable destination, or a schema change job in a reverting state.

The failure cascade is predictable: garbage grows, disk space shrinks, compaction eventually cannot find enough free space to run, and the LSM enters a death spiral where write stalls compound with read amplification.

flowchart TD
    A["Deletes / updates create tombstones"] --> B["MVCC garbage accumulates past GC TTL"]
    B --> C{"Protected timestamp blocking GC?"}
    C -->|No| D["MVCC GC reclaims old versions"]
    D --> E["Pebble compaction removes tombstones"]
    E --> F["Disk space stabilizes"]
    C -->|Yes| G["GC threshold pushed forward"]
    G --> H["Garbage bytes grow without bound"]
    H --> I["Disk fills silently"]
    I --> J["Compaction starved for space"]
    J --> K["L0 sublevels climb, write stalls"]

Common causes

Quick checks

These commands are safe, read-only, and can be run during production incidents.

# Check disk space per store from Prometheus metrics
curl -s http://localhost:8080/_status/vars | grep -E 'capacity_(used|available)'

# Check protected timestamp record count and age
curl -s http://localhost:8080/_status/vars | grep -E 'protected_record_count|protected_age_sec'

# Check changefeed lag, error retries, and running changefeed count
curl -s http://localhost:8080/_status/vars | grep -E 'changefeed_max_behind|changefeed_error_retries|changefeed_running'

# Check L0 sublevels (downstream storage health)
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

# Check read amplification (tombstones inflate this; the metric is still exported as rocksdb_read_amplification)
curl -s http://localhost:8080/_status/vars | grep -E 'rocksdb_read_amplification|storage_read_amplification'

For protected timestamp and job diagnostics via SQL (admin-only, use for diagnosis not continuous monitoring):

-- Find changefeed and backup jobs that may be holding protected timestamps
SELECT job_id, job_type, status, running_status, created
FROM crdb_internal.jobs
WHERE job_type IN ('CHANGEFEED', 'BACKUP', 'RESTORE')
  AND status NOT IN ('succeeded', 'canceled')
ORDER BY created DESC;

-- Check MVCC garbage vs live bytes per range (expensive, use sparingly)
-- Garbage = (key_bytes + val_bytes) - live_bytes
SHOW RANGES FROM TABLE <your_table> WITH DETAILS;

How to diagnose it

1. Confirm garbage is the problem, not live data growth. Pull capacity_used and capacity_available per store. If disk is growing but you know data volume is stable, garbage is the likely cause. If you have access to SHOW RANGES WITH DETAILS, compare live_bytes against total bytes (key_bytes + val_bytes). A large gap means significant garbage.

2. Check for protected timestamp records. Monitor spanconfig_kvsubscriber_protected_record_count. If the count is nonzero or growing, check jobs_changefeed_protected_age_sec for age. Protected age above 24 hours with no active backup or CDC operation completing is a red flag.

3. Identify the owning job. Query crdb_internal.jobs for any job in running, pause-requested, or reverting state that matches the protected timestamp age. Changefeeds are the most common culprit. A changefeed that fell behind the GC TTL window will hold a protected timestamp to avoid losing data it has not yet emitted.

4. Rule out GC TTL mismatch. If no protected timestamps exist, check whether the default TTL (25 hours through v22.2, 4 hours since v23.1) is too long for your churn rate. High-volume delete workloads with a long TTL will naturally accumulate more garbage before GC runs. This is expected behavior, not a bug, but it can consume surprising amounts of disk.

5. Check for version-specific GC bugs. On v23.2.x, there is a known issue (#136101) where MVCC GC does not run after certain schema changes because the GC hint is not set. The GC queue may estimate next GC in thousands of days. If garbage is growing across all ranges and no protected timestamps are present, check whether your version is affected.

6. Verify the MVCC GC queue is not being starved by admission control. As of v23.2, MVCC GC is subject to IO admission control. If the store-write admission queue is overloaded, GC work may be deprioritized behind foreground writes. Check admission.io.overload and admission wait durations.

Metrics and signals to monitor

Fixes

Stalled or paused changefeed holding a protected timestamp

This is the most common cause. If a changefeed has fallen behind or is paused, its protected timestamp prevents GC.

• If the changefeed is paused and you no longer need it, cancel it with CANCEL JOB <job_id>. This releases the protected timestamp.
• If the changefeed is running but lagging, investigate the sink (Kafka, cloud storage, webhook). Network issues, destination throttling, or schema mismatches cause backpressure.
• CockroachDB changefeeds support a gc_protect_expires_after option that automatically expires protected timestamps older than the specified duration and cancels the changefeed. There is no default for the option (0 disables it). As a backstop since v23.1, the changefeed.protect_timestamp.max_age cluster setting (default 4 days) fails a changefeed whose protected timestamps age past it. If the changefeed is truly stuck, these are the automatic release mechanisms, but they may take longer than your disk headroom allows.

Hung backup job

A backup stuck retrying against an unreachable destination holds a protected timestamp for the duration of the backup.

• Check the backup destination connectivity.
• If the job is in reverting state, it is rolling back and will release the protected timestamp when revert completes. Monitor progress.
• If the job is stuck in running with no progress, consider canceling it if you have a recent successful backup within your RPO window.

GC TTL too long for the workload

If garbage is accumulating because the default TTL (25 hours through v22.2, 4 hours since v23.1) is too conservative for a high-churn workload, you can lower gc.ttlseconds at the zone level for specific tables or databases.

-- Lower GC TTL for a high-churn table (use with caution)
ALTER TABLE <table> CONFIGURE ZONE USING gc.ttlseconds = 3600;

Warning: Lowering GC TTL reduces the window for point-in-time reads, incremental backups, and changefeed recovery. If you run incremental backups or CDC on the table, the TTL must be longer than the backup interval or changefeed checkpoint interval. Otherwise, incremental backups will fail because the required MVCC history has been garbage collected.

Emergency: manual GC enqueue

For cases where GC is not running on specific ranges (known bug or stuck queue), the operational FAQ documents an emergency procedure:

1. Temporarily lower gc.ttlseconds for the affected table (e.g., ALTER TABLE <table> CONFIGURE ZONE USING gc.ttlseconds = 600;).
2. Find the range IDs with SHOW RANGES FROM TABLE <table>;, then in the DB Console open Manually enqueue range in a replica queue, select queue mvccGC for each range ID, and check SkipShouldQueue to force processing.

This is a disruptive operation. It should only be used when disk pressure is critical and you understand the consequences.

Prevention

• Monitor protected timestamp age alongside changefeed lag. A growing jobs_changefeed_protected_age_sec with a lagging changefeed is the earliest warning. Alert on protected age above 24 hours.
• Track the ratio of live bytes to total bytes per store. If you have access to SHOW RANGES WITH DETAILS, garbage is (key_bytes + val_bytes) - live_bytes. A growing gap means GC is falling behind.
• Set changefeed checkpoint intervals shorter than the GC TTL. If a changefeed checkpoints less frequently than the GC window, it will always hold a protected timestamp close to the TTL boundary, leaving little GC headroom.
• Review backup frequency against GC TTL. Incremental backups require MVCC history from the last backup. If the interval between incremental backups exceeds the GC TTL, either lower the backup interval or raise the TTL with a protected timestamp backup.
• Upgrade to benefit from tombstone improvements. Size-carrying point tombstones (v23.2, PR #104539) help Pebble reclaim disk space more efficiently after MVCC GC by embedding the deleted KV pair size in the tombstone. This improves compaction space estimates. MVCC range tombstones (enabled by default in v23.1) allow bulk deletions to write a single range tombstone instead of billions of point tombstones.
• Watch for schema-change-related GC stalls on v23.2.x. If GC does not progress after a DROP or schema change, check whether the GC hint is empty and consider manual enqueue.

How Netdata helps

Netdata surfaces several signals that, when correlated, shorten the diagnosis of MVCC garbage growth:

• Per-second disk capacity metrics (capacity_used, capacity_available) let you spot the divergence between live data growth and total disk consumption before the 20% threshold is breached.
• Protected timestamp record count and age are monitored continuously, so a stalled changefeed or hung backup is visible as a growing protected age, not discovered when the disk is full.
• Changefeed lag and error metrics correlate directly with protected timestamp growth. A changefeed with growing changefeed_max_behind_nanos and rising jobs_changefeed_protected_age_sec is the textbook precursor to silent disk growth.
• L0 sublevel count and read amplification show the downstream storage impact. If garbage is inflating tombstone counts, these metrics degrade before write stalls hit.
• ML-based anomaly detection flags unusual patterns in disk growth rate, protected timestamp age, or changefeed lag that static thresholds miss. This is valuable for this failure mode because it is slow and silent by nature.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs