CockroachDB changefeed lag: changefeed_max_behind_nanos and the GC time bomb

Changefeed lag in CockroachDB starts as a consumer latency problem and ends as a disk space emergency. The changefeed_max_behind_nanos gauge measures how far behind your CDC feeds have fallen, but the real danger is what follows: a stalled changefeed holds a protected timestamp that prevents MVCC garbage collection, and dead data accumulates silently until the disk fills. The cluster appears healthy from the SQL side. Queries are fast, nodes are up, ranges are available. But every deleted row and every overwritten value stays on disk because GC cannot reclaim the space.

What this means

changefeed_max_behind_nanos is a gauge reporting the maximum lag across all changefeeds on the cluster, in nanoseconds between the most recent resolved timestamp emitted and the current time. A growing value means at least one changefeed cannot keep pace with the write rate.

For near-real-time CDC pipelines, lag above 30 seconds is concerning and above 5 minutes is actionable. But lag is only the visible symptom. The critical concern is what a stalled or failed changefeed does to garbage collection.

CockroachDB uses MVCC with garbage collection to reclaim space from old versions and deleted data. GC cleans up versions older than the GC TTL threshold (default gc.ttlseconds = 90000, or 25 hours). Protected timestamps created by changefeeds, backups, and other features mark a point below which GC cannot proceed. As long as a protected timestamp exists, all MVCC versions at or after that timestamp remain on disk.

A changefeed that stalls or is paused holds its protected timestamp in place. Every subsequent write, update, and delete creates new MVCC versions that GC cannot clean up. Disk usage grows monotonically, proportional to the write rate and the duration of the stall. Operators call this the GC time bomb: a completely silent disk fill with no visible error until capacity is exhausted.

flowchart TD
    A[Changefeed lags or stalls] --> B[Protected timestamp frozen]
    B --> C[MVCC GC blocked]
    C --> D[Garbage accumulates on disk]
    D --> E[Disk space shrinking]
    E --> F[Compaction starved for space]
    F --> G[LSM death spiral]
    A --> H[Feed resumed or fixed]
    H --> I[Protected timestamp advances]
    I --> J[GC resumes]
    J --> K[Garbage reclaimed]

Common causes

Quick checks

These are safe, read-only checks for initial triage. All hit the Prometheus endpoint on localhost:8080.

# Current changefeed lag (nanoseconds)
curl -s http://localhost:8080/_status/vars | grep changefeed_max_behind_nanos

# Changefeed error retry rate
curl -s http://localhost:8080/_status/vars | grep changefeed_error_retries

# Running changefeed count
curl -s http://localhost:8080/_status/vars | grep jobs_changefeed_currently_running

# Protected timestamp age (seconds)
curl -s http://localhost:8080/_status/vars | grep jobs_changefeed_protected_age_sec

# Protected timestamp record count
curl -s http://localhost:8080/_status/vars | grep spanconfig_kvsubscriber_protected_record_count

# Disk space per store
curl -s http://localhost:8080/_status/vars | grep -E 'capacity_(used|available)'

-- List changefeed jobs and their status
SELECT job_id, status, running_status, fraction_completed, created
FROM crdb_internal.jobs
WHERE job_type = 'CHANGEFEED'
  AND status NOT IN ('succeeded', 'canceled')
ORDER BY created DESC;

How to diagnose it

1. Check the lag trend. Look at changefeed_max_behind_nanos over time, not just the current value. A flat but elevated value (for example, steady at 2 minutes) may indicate a fixed latency floor. A monotonically increasing value means the feed is losing ground and will eventually trigger the protected timestamp cascade.

2. Identify which changefeed is behind. changefeed_max_behind_nanos is a cluster-level aggregate. It reports the maximum lag across all feeds but does not identify which feed is the source. Use crdb_internal.jobs to list all changefeed jobs and their status. A feed in running status that is not making progress on fraction_completed or resolved timestamp advancement is the likely culprit.

3. Check protected timestamp age. Compare jobs_changefeed_protected_age_sec against your GC TTL (gc.ttlseconds, default 90000 or 25 hours). If protected age exceeds the TTL window, MVCC versions that should have been collected are still on disk.

4. Correlate with disk growth. Check whether capacity_used is growing faster than your live data growth rate. If disk is growing but watched tables are not receiving proportionally more writes, the growth is unreclaimed MVCC garbage behind the protected timestamp.

5. Check the sink. If changefeed_error_retries is climbing, the changefeed is hitting errors emitting to its sink. Network issues, sink capacity limits, and credential problems all manifest here. Check connectivity from CockroachDB nodes to the sink endpoint.

6. Check coordinator health. Each changefeed runs on a coordinator node. If that node is CPU-saturated, GC-thrashing, or has lost liveness, the changefeed stalls. The job may show as running but make no progress. Check the coordinator node CPU, liveness, and error logs.

Metrics and signals to monitor

Fixes

Sink connectivity or throughput problems

If changefeed_error_retries is climbing, the changefeed is failing to emit records to its sink. Common causes: network partition to the Kafka broker, cloud storage credential expiration, webhook endpoint returning errors, or sink capacity insufficient for the write rate.

Fix the sink first. Verify connectivity from CockroachDB nodes to the sink endpoint. Check sink-side metrics such as Kafka broker lag or cloud storage rate limits. Once the sink is healthy, the changefeed should drain its backlog and lag should decrease.

If the sink fundamentally cannot keep up with the write rate, you need to scale the sink or reduce the scope of the changefeed (fewer watched tables, filtered columns).

Paused or failed changefeed

A changefeed in paused or failed status holds its protected timestamp. This is the GC time bomb scenario.

Resume the feed with RESUME JOB <job_id> if the underlying issue is resolved. The protected timestamp advances as the feed catches up, and GC resumes.

Cancel the feed with CANCEL JOB <job_id> if it is no longer needed. This releases the protected timestamp immediately, but with two costs: CANCEL JOB initiates a reversal that consumes time and I/O, which matters if disk is already tight, and you lose the CDC position entirely. Recreating the feed requires a fresh initial scan.

If the feed cannot be resumed (persistent sink failure, schema change conflict, version bug), cancellation may be the only option to stop disk growth.

Protected timestamp age approaching GC failure

Since v23.2, CockroachDB always maintains a protected timestamp for running changefeeds. The protect_data_from_gc_on_pause option is deprecated. The cluster setting changefeed.protect_timestamp.max_age (default 4 days) controls how long a protected timestamp can exist without forward progress before the changefeed fails with a permanent error.

If a protected timestamp has been stuck for days, the changefeed.protect_timestamp.max_age mechanism should eventually fail the changefeed and release the timestamp. If the mechanism is not firing, or the feed is making minimal progress that resets the timer without actually draining, manual intervention is required.

The per-changefeed option gc_protect_expires_after provides an alternative: it expires PTS records after a defined duration and cancels the job.

Disk space critical

If disk is below 20% free, prioritize:

1. Cancel or resume stalled changefeeds to release protected timestamps. Be aware that CANCEL JOB itself consumes I/O for the reversal. If the node is already struggling, prefer RESUME JOB where possible.
2. Wait for GC to reclaim MVCC garbage. This may take a full GC cycle after the timestamp is released.
3. Monitor capacity_available for recovery.

If disk fills completely before GC can run, the node enters a death spiral where compaction cannot proceed, leading to LSM compaction failure and potential node loss.

Prevention

• Alert on changefeed_max_behind_nanos trend, not just threshold. A steadily increasing value is more actionable than a fixed threshold. For near-real-time CDC, alert at above 30 seconds sustained and page at above 5 minutes.

• Alert on jobs_changefeed_protected_age_sec relative to gc.ttlseconds. This is the most direct signal for the GC time bomb. Alert when protected age exceeds the GC TTL window.

• Monitor spanconfig_kvsubscriber_protected_record_count for unexpected growth. Records should exist only during active CDC or backup operations and should not accumulate without forward progress.

• Track disk growth rate vs. live data growth rate. If disk is growing faster than live data, suspect GC blockage from protected timestamps.

• Verify changefeed job health after node restarts and failovers. A changefeed coordinator that moves to a new node may need time to resume. Check job status after any cluster event.

• Avoid min_checkpoint_frequency=0. Versions v25.4 through v26.1 reportedly have a bug where setting min_checkpoint_frequency to 0 prevents the resolved timestamp from advancing, causing the changefeed to lag indefinitely. Use the default or a value of at least 500ms.

How Netdata helps

• Per-second changefeed_max_behind_nanos collection lets you distinguish a flat latency floor from a monotonically increasing trend. This is the critical diagnostic distinction for whether the feed is losing ground.

• Correlating changefeed lag with protected timestamp age on the same dashboard makes the GC time bomb visible. When jobs_changefeed_protected_age_sec stops advancing while changefeed_max_behind_nanos grows, you can see exactly when the protected timestamp froze.

• Disk space metrics alongside protected timestamp metrics show the cause-and-effect chain. You can watch capacity_available declining in lockstep with rising protected timestamp age, confirming the growth is GC-blocked MVCC garbage rather than live data.

• Anomaly detection on changefeed_error_retries catches sink problems early. A spike in retry rate often precedes lag growth by minutes, giving you a window to fix the sink before the protected timestamp stalls.

• Per-node CPU and liveness correlation helps identify whether a changefeed stall is caused by coordinator node degradation rather than a sink problem.

Netdata’s CockroachDB monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit