CockroachDB memory pressure, GC thrashing, and Raft liveness failure

A CockroachDB node loses liveness in a repeating pattern: it drops out, the cluster redistributes its leases, it recovers, reacquires leases, then drops again. Each cycle lasts seconds to minutes. Application queries see intermittent timeouts, ambiguous results, and latency spikes that correlate with the node’s oscillation. The DB Console shows the node flapping between live and not-live states.

This is the memory pressure to GC thrashing to Raft liveness failure cascade. The Go runtime heap grows until garbage collection pauses become long enough to prevent the node from renewing its liveness heartbeat. Once the heartbeat interval lapses, the cluster marks the node dead and moves its leases. When GC completes and memory is freed, the node recovers and reacquires leases, restarting the cycle.

Each oscillation creates a burst of lease transfers and range unavailability. If another node is already marginal, the redistributed load can push it into the same cycle. Multiple nodes oscillating simultaneously can degrade the entire cluster.

What this means

CockroachDB nodes maintain liveness by renewing a heartbeat record on a fixed interval. If a node fails to renew within the expiry window, the cluster considers it dead and redistributes its range leases. Liveness depends on the node’s ability to process internal RPCs and write heartbeat records to the KV layer.

Go’s garbage collector runs stop-the-world pauses during certain collection phases. When the Go heap is large, these pauses can last hundreds of milliseconds or even seconds. If a GC pause exceeds the heartbeat renewal interval, the node cannot renew its liveness record in time. The cluster then treats it as dead, transfers its leases, and marks its ranges for up-replication.

The diagnostic signature is oscillation: the node loses liveness, recovers after GC completes, reacquires leases, then loses liveness again on the next GC cycle. GC pause duration directly correlates with liveness loss events. This distinguishes the pattern from a hard node crash (which does not recover on its own) or a network partition (which affects inter-node communication, not internal GC).

flowchart TD
    A["Go heap grows"] --> B["GC frequency and duration increase"]
    B --> C["GC pause exceeds heartbeat interval"]
    C --> D["Node loses liveness"]
    D --> E["Cluster redistributes leases"]
    E --> F["GC completes, memory freed"]
    F --> G["Node recovers, reacquires leases"]
    G --> B

Common causes

Quick checks

These are safe, read-only commands. Run them on the affected node.

# Check Go and CGo memory allocations
curl -s http://localhost:8080/_status/vars | grep -E 'sys\.(rss|go_allocbytes|cgo_allocbytes)'

# Check SQL memory budget utilization
curl -s http://localhost:8080/_status/vars | grep 'sql_mem_root_current'

# Check GC pause duration and cycle count
curl -s http://localhost:8080/_status/vars | grep -E 'sys_gc_(pause_ns|count)'

# Check process RSS from the OS
cat /proc/$(pgrep -x cockroach)/status | grep VmRSS

# Check range unavailability (should be zero)
curl -s http://localhost:8080/_status/vars | grep 'ranges_unavailable'

# Check lease transfer rate (elevated during liveness oscillation)
curl -s http://localhost:8080/_status/vars | grep 'leases_transfers_success'

# Check goroutine count
curl -s http://localhost:8080/_status/vars | grep 'sys_goroutines'

# Check SQL error count for resource exhaustion (53200)
curl -s http://localhost:8080/_status/vars | grep 'sql_failure_count'

How to diagnose it

1. Confirm the oscillation pattern. Check node liveness status over a 10-15 minute window. The node should be transitioning between live and not-live repeatedly, not a single hard failure. Use curl -s http://localhost:8080/_status/nodes and look for epoch increments or liveness state changes on the affected node.

2. Correlate GC pauses with liveness events. Look at sys_gc_pause_ns deltas between scrapes. If GC pause duration spikes align with liveness loss timestamps, the cascade is confirmed. Pauses above 500ms are concerning. Pauses above 1s carry risk of liveness loss. Pauses approaching the heartbeat interval will cause it.

3. Identify the memory consumer. Check whether pressure is from Go heap (sys_go_allocbytes), CGo allocations (sys_cgo_allocbytes, primarily Pebble block cache), or SQL execution memory (sql_mem_root_current):

   • Go heap growing rapidly with active queries: SQL execution is allocating heavily.
   • CGo allocations growing while Go heap is stable: the Pebble block cache may be too large for the container.
   • SQL memory budget above 70%: individual queries are consuming most of the per-node budget.

4. Check goroutine count. If sys_goroutines is above 100,000 or growing without workload increase, goroutine stacks (each roughly 8KB minimum) consume heap memory and drive GC pressure. Normal range is 5,000-30,000 depending on cluster size and connection count.

5. Identify the offending query. If SQL memory is the consumer, run SHOW CLUSTER SESSIONS to find active queries consuming large amounts of memory. Look for full table scans, large sorts, or hash joins on large tables. A single large query can starve all others because the SQL memory budget is per-node, not per-query.

6. Verify memory configuration. CockroachDB’s production recommendations specify that (2 * --max-sql-memory) + --cache should not exceed 80% of available system RAM. The remaining 20% covers Go runtime overhead, OS page cache, and burst allocations. If these flags are set too aggressively for the container or host, the node has no headroom.

7. Check for 53200 errors. A burst of SQL error code 53200 (resource exhaustion) indicates the SQL memory budget is exhausted. This often precedes an OOM kill if the budget is misconfigured.

Metrics and signals to monitor

Fixes

Kill the offending SQL query

If sql_mem_root_current is the primary consumer, a single large query (sort, hash join, full scan) may be starving the node. Identify it via SHOW CLUSTER SESSIONS and cancel it:

CANCEL QUERY 'query-id-here';

This terminates the query immediately. The node’s memory will be reclaimed on the next GC cycle. If the query is from a recurring application path, it will return unless the query or schema is fixed.

Reduce --max-sql-memory or --cache

If memory flags are set too aggressively for the container or host, reduce them and restart the node. CockroachDB’s production sizing formula is:

(2 * --max-sql-memory) + --cache <= 80% of available RAM

The remaining 20% covers Go runtime overhead, OS page cache, and burst allocations. In containers, base these values on the cgroup limit, not the host memory. The default --max-sql-memory is 25% of system memory, which may be too high for memory-constrained containers.

This change requires a node restart. Apply it during a rolling restart window, sequencing nodes with sufficient delay so the cluster maintains quorum and under-replication stays transient.

In systemd unit files, the % character must be escaped as %% when specifying percentage-based values for these flags.

Fix goroutine leaks

If goroutine count is growing without workload increase, check the goroutine profile to identify blocked stacks:

curl -s http://localhost:8080/debug/pprof/goroutine?debug=1 | head -50

Common causes include goroutines blocked on slow I/O (resolves when the I/O issue is fixed) or application-driven connection leaks (requires fixing client-side connection management). Each goroutine costs roughly 8KB minimum stack, so 100,000 goroutines consume approximately 800MB in stacks alone.

Address connection storms

If the pressure follows a node failure or restart, the cause is likely all clients reconnecting simultaneously to surviving nodes. Without proper backoff, hundreds or thousands of connections arrive at once, consuming goroutines and memory. The surviving nodes become overloaded not from the additional query load, but from the connection overhead itself.

Ensure client-side connection pools use exponential backoff with jitter on reconnect. Verify that PgBouncer or equivalent poolers are configured in transaction mode. Statement mode breaks multi-statement transactions and is not safe with CockroachDB.

Prevention

Right-size memory flags at deployment. Use the production formula (2 * --max-sql-memory) + --cache <= 80% of available RAM. In containers, compute against the cgroup limit. RSS should stay below 75% of available memory, leaving the remaining 25% for OS page cache, GC headroom, and burst allocations.

Alert on GC pause duration as a leading indicator. Individual GC pauses above 500ms indicate the node is approaching the cliff. Set alerting on the delta of sys_gc_pause_ns per scrape interval. If GC consumes more than 15% of CPU, the Go runtime is spending excessive time on memory management. The degradation curve is gradual then cliff: performance degrades slowly as GC consumes more CPU, then drops sharply when pauses exceed the heartbeat interval.

Monitor SQL memory budget utilization. sql_mem_root_current sustained above 70% means limited headroom for concurrent expensive queries. Normally this should be well below 50% to handle burst loads.

Alert on RSS approaching limits. RSS above 80% of available memory leaves no headroom for burst allocations. OOM risk and GC thrashing both accelerate past this point.

Use the readiness endpoint for load balancer health checks. Configure load balancers to use GET /health?ready=1, which returns 503 when the node is draining or quorum is lost. Plain TCP checks route traffic to nodes that are technically listening but functionally impaired by GC thrashing.

How Netdata helps

• Per-second collection of sys_go_allocbytes, sys_cgo_allocbytes, and sql_mem_root_current catches memory growth patterns that 15-30s Prometheus scrapes miss, especially the rapid spikes that precede liveness loss.
• GC pause duration from sys_gc_pause_ns deltas, correlated on the same timeline with node liveness transitions and lease transfer spikes, makes the oscillation pattern visible without manual timestamp matching.
• RSS monitoring with thresholds relative to container cgroup limits catches the gradual then cliff trajectory before GC pauses threaten liveness.
• ranges_unavailable and leases_transfers_success alongside GC and memory metrics on a single timeline let you distinguish GC-driven liveness failure from disk-stall-driven or CPU-saturation-driven causes.

For the full setup, see CockroachDB monitoring with Netdata.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls