CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats

GC pause time spiking, node liveness flapping, lease transfers churning. SQL latency oscillates between acceptable and terrible. The cluster is alive but unstable, and the oscillation pattern is the tell.

CockroachDB runs on the Go runtime, which performs stop-the-world garbage collection pauses. When those pauses grow long enough, they block the Raft heartbeat loop. The node stops responding to heartbeats for the duration of the pause. If the pause is long enough, the cluster declares the node dead and redistributes its leases. Then the pause ends, the node recovers, leases come back, the heap grows again, and the cycle repeats.

This is the Memory Pressure to GC Thrashing to Raft Liveness Failure cascade. It is self-perpetuating, and the oscillation makes it hard to distinguish from network issues or node crashes. The leading indicator is GC pause duration and frequency trending upward, well before liveness is actually lost.

What this means

Each node in a CockroachDB cluster renews a liveness record via a heartbeat. Raft heartbeats flow continuously between replicas for every range on the node. A node with 10,000 ranges runs 10,000 Raft state machines, each consuming CPU for ticking, proposing, and applying entries.

Go’s garbage collector performs stop-the-world pauses during certain phases. For a healthy CockroachDB process, these pauses are sub-millisecond and invisible. Under memory pressure, pauses grow to hundreds of milliseconds, then to seconds. The thresholds that matter:

• GC CPU below 5%: healthy
• 5-10%: elevated, worth investigating
• 10-15%: concerning, active risk of degradation
• Above 15%: active degradation
• Individual pauses above 500ms: warning
• Individual pauses above 1s: risk of liveness loss, especially if repeated

GC CPU is computed from the cumulative sys_gc_pause_ns counter: the delta between scrapes divided by the scrape interval gives the fraction of wall time spent in GC stops. CockroachDB also exposes sys_gc_pause_percent, the current GC pause percentage, natively.

When an individual GC pause approaches or exceeds the heartbeat interval, the node cannot renew its liveness record or process Raft ticks. The cluster interprets this silence as node failure.

The exact heartbeat intervals, expiry windows, and liveness model can vary across CockroachDB versions, with recent releases transitioning toward store-level liveness and lease-based failure detection. By default the liveness record is renewed every ~3 seconds and expires after 6 seconds; with store-level liveness, a store is considered dead after server.time_until_store_dead (default 5 minutes).

flowchart TD
    A["Go heap grows under\nmemory pressure"] --> B["GC pause duration\nincreases"]
    B --> C{"Pause exceeds\nheartbeat interval?"}
    C -->|No| D["Elevated latency\nbut node stays live"]
    C -->|Yes| E["Node misses\nliveness renewal"]
    E --> F["Cluster declares\nnode not-live"]
    F --> G["Leases redistributed\nto other nodes"]
    G --> H["GC pause ends\nnode recovers"]
    H --> I["Leases return\nheap grows again"]
    I --> B

Each cycle creates a burst of lease transfers, each with a brief unavailability window for the affected ranges. If the underlying memory pressure is not resolved, the oscillation continues indefinitely.

Common causes

Quick checks

All read-only and safe to run during an incident:

# Check GC pause and count metrics
curl -s http://localhost:8080/_status/vars | grep -E 'sys_gc_pause|sys_gc_count'

# Check process memory components
curl -s http://localhost:8080/_status/vars | grep -E 'sys_rss|sys_go_allocbytes|sys_cgo_allocbytes'

# Check SQL memory budget
curl -s http://localhost:8080/_status/vars | grep 'sql_mem_root_current'

# Check goroutine count
curl -s http://localhost:8080/_status/vars | grep 'sys_goroutines'

# Check lease transfer rate (elevated = liveness oscillation)
curl -s http://localhost:8080/_status/vars | grep 'leases_transfers_success'

# Check node liveness status (status: 0=UNKNOWN 1=DEAD 2=UNAVAILABLE 3=LIVE 4=DECOMMISSIONING 5=DECOMMISSIONED 6=DRAINING)
curl -s http://localhost:8080/_status/nodes | python3 -c "
import json, sys
data = json.load(sys.stdin)
statuses = {0: 'UNKNOWN', 1: 'DEAD', 2: 'UNAVAILABLE', 3: 'LIVE', 4: 'DECOMMISSIONING', 5: 'DECOMMISSIONED', 6: 'DRAINING'}
liveness = {int(k): v for k, v in data.get('liveness_by_node_id', {}).items()}
for n in data['nodes']:
    print(f'Node {n[\"desc\"][\"node_id\"]}: liveness={statuses.get(liveness.get(n[\"desc\"][\"node_id\"]), \"UNKNOWN\")}')"

# Check OS-level process memory
cat /proc/$(pgrep -x cockroach)/status | grep -E 'VmRSS|VmSize'

# Check for heartbeat-related log entries
# Log path varies by deployment; default is <data-dir>/logs/cockroach.log
grep -iE "slow heartbeat|failed.*heartbeat|context deadline exceeded" /path/to/cockroach-data/logs/cockroach.log | tail -20

How to diagnose it

1. Confirm GC pressure is the root cause. Compute GC CPU from sys_gc_pause_ns deltas between scrapes. If GC CPU is above 5%, you are in elevated territory. Above 15%, you have active degradation. A rising rate means the heap is growing faster than GC can keep up.

2. Check whether individual pauses reach the danger zone. The cumulative sys_gc_pause_ns tells you aggregate pressure. For individual pause duration, correlate spikes in the GC pause graph with liveness events in the node logs. “Context deadline exceeded” errors in heartbeat paths are a strong signal.

3. Identify the memory consumer. Check three signals:

   • sys_go_allocbytes: Go heap (SQL execution, metadata, goroutine stacks)
   • sys_cgo_allocbytes: CGo allocations, primarily Pebble block cache and memtables
   • sql_mem_root_current: SQL execution buffers (sorts, hash joins, result sets)

   The one growing fastest is the likely driver. CGo allocations are manually managed, not controlled by Go GC. If CGo memory grows while Go heap is stable, the leak is in the storage engine layer.

4. Check RSS against your memory limit. RSS should stay below 75-80% of available memory or cgroup limit. The remaining headroom is needed for OS page cache (critical for Pebble SSTable reads), Go GC overhead, and burst allocations.

5. Verify the oscillation pattern. If node liveness is flapping in sync with GC pause spikes, you have confirmed the cascade. The lease transfer rate will be elevated, and ranges_unavailable may briefly spike above zero during each oscillation. Each lease transfer creates a brief period of increased latency for affected ranges.

6. Distinguish from disk stalls. A disk stall produces similar liveness symptoms but shows elevated WAL fsync latency and storage_disk_stalled rather than GC pressure. Check both to avoid misdiagnosis. Both can coexist: disk I/O contention can slow compaction, which inflates memory usage, which triggers GC pressure.

Metrics and signals to monitor

Fixes

Kill memory-intensive queries

If sql_mem_root_current is the driver, a single large query may be consuming most of the SQL memory budget. Identify and cancel it:

CANCEL QUERY 'query_id';

This disrupts the active query. Check crdb_internal.node_statement_statistics for queries with high memory usage patterns. Look for sorts without indexes, hash joins on large tables, and queries that materialize large result sets before truncating with LIMIT.

The SQL memory budget is per-node, not per-query. A single large query can starve all others.

Reduce memory budgets

If --max-sql-memory or --cache are configured too aggressively relative to available memory, the Go runtime is forced into tight GC cycles. Both default to 25% of system memory:

• --max-sql-memory: SQL execution memory budget
• --cache: Pebble block cache

Together they consume 50% of system memory by default. In a container, these must be set as fractions of the container limit, not the host. A common mistake is leaving them at host defaults when the container limit is much smaller.

The total memory footprint must satisfy: --max-sql-memory + --cache + Go runtime overhead + OS page cache needs, all within the available limit with 25% headroom. If it does not, reduce --cache or --max-sql-memory.

This change requires a node restart. Plan for rolling restarts.

Address goroutine leaks

If sys_goroutines is growing monotonically without a workload increase, goroutines are likely blocked on channels or locks that will never resolve. Each starts with a 2KB stack that grows on demand, and large counts directly inflate the Go heap.

# Get goroutine count
curl -s http://localhost:8080/debug/pprof/goroutine?debug=1 | head -1

For deeper analysis, capture a full goroutine profile and look for common blocking patterns. The fix depends on the root cause, which may require a CockroachDB bug report if the leak is internal.

CockroachDB sets the Go GC target internally (300% by default when a soft memory limit is configured) and honors externally-set GOGC and GOMEMLIMIT environment variables at startup, as well as the --go-gc-percent and --max-go-memory flags. Overriding these externally can cause unexpected behavior.

Expand memory or rebalance load

If none of the above resolves the issue, the node is fundamentally memory-constrained for its workload:

• Add memory to the node or increase the container limit
• Add nodes to reduce per-node range count, which reduces per-range Raft processing overhead and memory
• Move memory-intensive workloads to nodes with more capacity

Prevention

• Monitor GC CPU as a leading indicator. Alert when GC CPU exceeds 5% sustained or 10% at any point. This gives warning before pauses reach liveness-threatening durations.

• Watch RSS trends. Alert when RSS exceeds 75% of available memory or cgroup limit.

• Track SQL memory budget utilization. Alert when sql_mem_root_current exceeds 70% of --max-sql-memory with an upward trend. This catches memory-intensive queries before they drive the heap into GC thrashing.

• Validate container memory sizing. Ensure --max-sql-memory + --cache + Go runtime overhead + OS page cache needs all fit within the container limit with 25% headroom.

• Monitor goroutine count. Alert on 2x baseline without workload increase. Goroutine leaks are a slow-burning memory pressure source that eventually triggers the GC cascade.

• Use /health?ready=1 for load balancer checks. TCP health checks route traffic to nodes that are GC-thrashing or write-stalled but still listening on the port. The readiness endpoint returns 503 when the node is draining, decommissioning, or cannot reach a majority of the cluster.

Correlating with Netdata

Netdata collects CockroachDB’s Prometheus metrics at per-second resolution. This matters for the GC-to-liveness cascade because individual GC pause spikes can fall between 15-30 second scrape intervals from systems like Prometheus.

Practical use during an incident:

• Overlay sys_gc_pause_ns rate, RSS growth, sql_mem_root_current, and leases_transfers_success on a single timeline to confirm the oscillation pattern in seconds.
• Per-node dashboards prevent an overloaded leaseholder from hiding behind healthy cluster averages.
• Anomaly detection on GC cycle frequency can flag rising heap pressure before pauses reach dangerous durations.

For setup, see CockroachDB monitoring with Netdata.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls