CockroachDB protected timestamp GC stall: stalled changefeeds that silently fill the disk

Disk space is growing on your CockroachDB cluster and nothing explains it. capacity_available trends downward hour after hour, but live data size has not changed. No bulk imports, no schema changes, no write spike. The SQL layer looks healthy: latency is fine, throughput is normal. But the disk is filling.

A CDC changefeed, a backup job, or another internal operation has created a protected timestamp (PTS) record that prevents MVCC garbage collection from reclaiming old data versions. The job has stalled, paused, or entered a retry loop, but the PTS record persists. Dead MVCC data accumulates underneath it. Disk usage grows linearly until free space drops below what compaction needs, at which point L0 sublevels spike, write stalls begin, and nodes lose liveness.

What this means

CockroachDB uses MVCC to manage data. Every write creates a new row version at a specific timestamp. Old versions become garbage after they fall outside the GC TTL window (gc.ttlseconds), and background garbage collection reclaims that space.

Protected timestamps override this mechanism. CDC changefeeds and backups create PTS records that tell the storage engine: do not garbage-collect any data below this timestamp. A changefeed needs historical data to emit changes it has not yet sent downstream. A backup needs a consistent snapshot at backup time.

The problem arises when the owning job stalls. If a changefeed’s downstream sink goes down, the changefeed stops making forward progress. Its checkpoint timestamp stops advancing. The PTS record stays pinned at that old timestamp. Every write to the protected ranges creates new MVCC versions that GC cannot clean up. Disk usage grows at the rate of write churn on those ranges.

flowchart TD
    A["Changefeed/backup stalls"] --> B["PTS record pinned at old timestamp"]
    B --> C["MVCC GC blocked"]
    C --> D["Dead MVCC versions accumulate"]
    D --> E["Disk grows silently"]
    E --> F["Compaction space exhausted"]
    F --> G["L0 sublevels spike"]
    G --> H["Write stalls begin"]
    D -.->|Cancel or resume job| I["PTS record released"]
    I --> J["GC resumes, compaction reclaims garbage"]
    J --> K["Disk space stabilizes"]

Before CockroachDB v23.2, a stalled changefeed could hold a PTS record indefinitely. Starting in v23.2, changefeed.protect_timestamp.max_age (default 4 days) automatically expires PTS records for changefeeds that make no forward progress, causing the changefeed to fail. This is a safety net, not a monitoring strategy. Four days of accumulated garbage on a write-heavy table can consume substantial disk before the protection expires.

Common causes

Quick checks

# Check protected timestamp record count and age
curl -s http://localhost:8080/_status/vars | grep -E 'protected_record_count|protected_age_sec'

# Check changefeed lag, error retries, and running/paused counts
curl -s http://localhost:8080/_status/vars | grep -E 'changefeed_max_behind|changefeed_error_retries|currently_running|currently_paused'

# Check disk space per store
curl -s http://localhost:8080/_status/vars | grep -E 'capacity_(used|available)'

# Check L0 sublevels to see if compaction is already in trouble
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

-- Find changefeed jobs that are not succeeded or canceled
SELECT job_id, job_type, status, running_status, fraction_completed, created
FROM crdb_internal.jobs
WHERE job_type = 'CHANGEFEED' AND status NOT IN ('succeeded', 'canceled')
ORDER BY created DESC;

-- Find all non-succeeded jobs (backups, schema changes, imports)
SELECT job_id, job_type, status, running_status, fraction_completed, created
FROM crdb_internal.jobs
WHERE status NOT IN ('succeeded', 'canceled')
ORDER BY created DESC;

How to diagnose it

1. Confirm the PTS record exists and is stale. Check spanconfig.kvsubscriber.protected_record_count and jobs.changefeed.protected_age_sec. If protected age exceeds your configured gc.ttlseconds, garbage is accumulating.

2. Identify the owning job. Query crdb_internal.jobs for non-succeeded jobs. Cross-reference running_status and fraction_completed to determine if the job is making progress or stuck.

3. Correlate with changefeed lag. If changefeed_max_behind_nanos is growing and changefeed_error_retries is incrementing, the changefeed is actively failing to emit changes. The downstream sink may be unreachable, authentication may have expired, or the sink may be rejecting writes.

4. Check disk space trajectory. Compare the capacity_available trend against the rate of MVCC garbage accumulation. Calculate how long until free space drops below 20%. This determines urgency.

5. Verify L0 sublevel health. If storage_l0_sublevels is above 10, compaction is under pressure. If above 20, write stalls are imminent.

6. Distinguish PTS-driven disk growth from data growth. If live data size is stable but total disk usage is growing, the growth is MVCC garbage blocked by PTS records. If live data is also growing, you have a compounding problem.

Metrics and signals to monitor

Fixes

Resume or cancel the stalled changefeed

The immediate fix is to address the owning job. If a changefeed is paused, resume it:

RESUME JOB <job_id>;

If the changefeed is in an error state or retry loop and the downstream issue cannot be quickly resolved, cancel it:

CANCEL JOB <job_id>;

Canceling releases the PTS record. GC resumes on the next cycle, and MVCC garbage below the former PTS timestamp becomes eligible for collection.

Space is not reclaimed instantly. Tombstones propagate through LSM compaction, which requires disk I/O and working space. On a nearly full disk, compaction needs space to run but space only frees up through compaction.

Tradeoff: Canceling a changefeed means downstream consumers lose their change stream and need a full backfill. If the sink can be repaired quickly, resuming is preferable. If the changefeed has been stalled for days, the accumulated catchup scan may be as expensive as a fresh backfill.

Release PTS records from a hung backup

If a backup job is the culprit, cancel it:

CANCEL JOB <job_id>;

The backup’s PTS record is released when the job completes its revert phase and enters canceled state. The revert may take time proportional to how far the backup progressed.

Emergency disk space recovery

If the disk is critically full and compaction cannot proceed, CockroachDB maintains an emergency ballast file in each node’s store directory as a pre-allocated space reserve.

# Emergency only: locate the ballast file
find /path/to/cockroach-data -name '*ballast*' -ls

# Destructive: remove the ballast file to free working space for compaction.
# Do this only after you have canceled or resolved the stalled job.
rm /path/to/cockroach-data/<ballast_filename>

Removing the ballast file buys time but does not fix the underlying problem. Use it only to get enough headroom for compaction to begin reclaiming garbage after you have released the PTS record. Without the underlying fix, the disk will fill again.

Configure automatic PTS expiration

Starting in v23.2, set changefeed.protect_timestamp.max_age to cap how long a stalled changefeed can hold a PTS record:

SET CLUSTER SETTING changefeed.protect_timestamp.max_age = '48h';

The default is 4 days. Lower it based on your disk capacity and write churn rate. When this limit is reached, the changefeed’s PTS record expires and the changefeed fails.

For per-changefeed control, use the gc_protect_expires_after option when creating the changefeed:

CREATE CHANGEFEED FOR TABLE my_table
INTO 'kafka://broker:9092'
WITH gc_protect_expires_after = '24h';

This automatically expires the PTS record and cancels the changefeed if it stalls for longer than the specified duration. This converts an unbounded silent disk fill into a bounded, noisy job failure that your existing job monitoring catches.

Prevention

• Alert on protected timestamp age. Alert when jobs.changefeed.protected_age_sec exceeds your GC TTL window. This is the earliest reliable signal that garbage is accumulating. Page when it exceeds 48 hours.
• Track changefeed health. Monitor changefeed_max_behind_nanos and changefeed_error_retries. Growing lag or incrementing retries precede PTS stalls.
• Set gc_protect_expires_after on production changefeeds. Without it, a stalled changefeed holds a PTS record indefinitely (pre-v23.2) or for up to 4 days (v23.2+ default). This is the single most effective preventive measure.
• Audit paused changefeeds. A paused changefeed holds a PTS record. Any paused changefeed should correspond to a planned maintenance window.
• Track live bytes to total bytes ratio. If total on-disk size diverges from live data size, MVCC garbage is accumulating. This catches PTS stalls, GC configuration issues, and tombstone accumulation from large deletes.
• Size for compaction headroom. Maintain at least 20% free disk space for compaction working space. With heavy MVCC churn from active changefeeds, consider 30%.

How Netdata helps

The signals for this failure mode change slowly over hours, then cascade quickly. Per-second collection on protected_age_sec and capacity_available catches the slow linear decline before it becomes a cascade. Correlating changefeed_max_behind_nanos, protected_age_sec, and capacity_available in a single view makes the causal chain visible: changefeed stalls, PTS age climbs, disk declines.

Composite alerts on storage_l0_sublevels alongside disk space metrics flag the transition from PTS stall to compaction failure. Per-store metrics are essential here: a single store filling up is the first symptom, and aggregate disk metrics mask it.

See database monitoring with Netdata for integration details.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit