CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling

CockroachDB is CPU-hungry by design. Every range runs its own Raft state machine. Every SQL statement parses, plans, and executes through Go. Compaction, encryption-at-rest, and checksumming all burn cycles. When CPU saturates, the failure mode is not a clean slowdown: Raft heartbeats get delayed, admission control starts queuing, GC pauses lengthen, and the node risks losing liveness. The cluster average can look healthy while one leaseholder melts.

CockroachDB recommends keeping sustained utilization below 60-70% so Raft processing and background tasks have headroom. The documented ceiling is 80% — above it, degradation is nonlinear: Raft, GC, and admission control contend for the remaining cycles simultaneously, and the node starts losing leases.

What this means

CockroachDB exposes CPU consumption as cumulative nanosecond counters: sys_cpu_user_ns for application (user) time and sys_cpu_sys_ns for kernel (system) time. Both are Prometheus counters, so compute rate() over a time window and divide by core_count * 1e9 to get utilization percentage. The split between user and system is the first diagnostic fork:

• High user CPU means SQL execution, Raft processing, or Go GC is dominating. Expected pattern for a CPU-bound workload.
• High system CPU with low user CPU means the kernel is doing heavy work on behalf of CockroachDB: filesystem operations from compaction, or network packet processing. When you see this, look at I/O, not SQL execution.

The three CPU consumers

Raft ticking is a per-range overhead that scales with range count, not query load. Each range is an independent Raft state machine that consumes CPU for ticking, proposing, and applying entries. The estimate is roughly 1% of a core per 500-1000 ranges. A node with 50,000 ranges needs significantly more CPU for Raft ticking alone than a node with 5,000 ranges, even at identical query throughput. At extreme range counts (above 100,000 per node), Raft ticking can consume multiple cores. This is the non-obvious resource multiplier that catches teams off guard.

SQL execution covers parsing, planning, optimization, and the DistSQL execution engine. Complex queries, missing indexes, and plan regressions all increase CPU per statement. This correlates directly with query throughput and is the most visible consumer.

Go GC runs concurrently but still causes stop-the-world pauses for certain phases. GC CPU above 10-15% of total indicates excessive garbage collection pressure, often from memory-intensive queries or a growing heap. Individual pauses above 500ms are a warning sign. Above 1s, there is risk of liveness loss.

flowchart TD
    subgraph Consumers
        R[Raft ticking: per-range state machines]
        S[SQL execution: parse, plan, DistSQL]
        G[Go GC: concurrent + STW pauses]
        C[Compaction: Pebble LSM merges]
    end
    R --> CPU[Node CPU pool]
    S --> CPU
    G --> CPU
    C --> CPU
    CPU --> L{Utilization level}
    L -- "Below 60-70%" --> OK[Healthy: bursts absorbed, Raft responsive]
    L -- "70-80%" --> AC[Admission control queuing, latency rising]
    L -- "Above 80%" --> CLIFF[Cliff: heartbeat delays, GC stalls, liveness risk]

Common causes

Quick checks

# CPU utilization per node (user vs system split)
# These are cumulative counters; compute rate() to get per-second consumption
curl -s http://localhost:8080/_status/vars | grep -E 'sys_cpu_(user|sys)_ns'

# Go GC pressure (pause duration and frequency)
curl -s http://localhost:8080/_status/vars | grep -E 'sys_gc_(pause|count)'

# Range count per node (primary driver of Raft tick overhead)
curl -s http://localhost:8080/_status/vars | grep -E '^ranges\b|leases'

# Admission control queue depth and wait times
curl -s http://localhost:8080/_status/vars | grep admission

# Goroutine count (concurrency pressure indicator)
curl -s http://localhost:8080/_status/vars | grep sys_goroutines

# OS-level CPU breakdown for the cockroach process
top -p $(pgrep -x cockroach) -bn1 | tail -1

# Disk I/O when system CPU is elevated
iostat -xz 1 3

How to diagnose it

1. Check per-node CPU, not the cluster average. The cluster average hides individual node saturation. Pull sys_cpu_user_ns and sys_cpu_sys_ns for each node separately. A single leaseholder can be at 90% while the cluster average reads 30%.

2. Split user vs system time. If user CPU dominates, the bottleneck is inside CockroachDB (Raft, SQL, GC). If system CPU dominates, the bottleneck is kernel-level I/O or network. This determines whether you investigate the database workload or the storage layer.

3. Correlate with range count. If CPU is elevated even when query throughput is low, check ranges per node. High range count means high Raft tick overhead regardless of workload. Compare against the 50,000-per-node practical ceiling.

4. Check GC pressure. If sys_gc_pause_ns is growing rapidly relative to sys_gc_count, individual pauses are getting longer. GC CPU above 10-15% of total means the Go runtime is spending excessive time on memory management. This is a leading indicator of memory pressure, GC thrashing, and Raft liveness failure cascades.

5. Check admission control. Sustained queuing in the kv or sql-kv-response queues means the system is at capacity and maintaining throughput by adding latency. elastic-cpu queue queuing is often acceptable (correctly prioritizing foreground over background), but foreground queue wait times above 10ms sustained indicate zero burst headroom.

6. Look for asymmetry across nodes. If one node has 2x or higher CPU than others with similar range counts, suspect a hot range. Confirm via per-range QPS distribution or the Hot Ranges page in the DB Console.

Metrics and signals to monitor

Fixes

Range count overhead

If range count is the driver, the fix is structural. Add nodes to reduce ranges per node. Range merges can consolidate sparse ranges, but are only effective for tables with deleted or sparse data. The total range count approximates total data divided by 512 MiB (the default range max size). With default 3-way replication, each node hosts roughly 3/N of the total ranges as replicas. A 100 TB cluster across 5 nodes has roughly 120,000 replicas per node, far above the practical ceiling.

Do not increase the range size limit without testing. Larger ranges mean larger Raft snapshots during rebalancing and recovery, which increases I/O pressure during node failures.

Hot range bottleneck

One range receiving 10x or higher average QPS saturates its leaseholder while the rest of the cluster idles. Short-term mitigation: ALTER TABLE ... SPLIT AT to manually split the hot range. Manual splits require forethought about split points and should be re-evaluated as data grows, since writes accumulate around the chosen boundaries. Long-term fix: redesign primary keys to avoid sequential or monotonically increasing patterns. UUID-based or hash-prefixed keys distribute writes across ranges naturally.

GC thrashing

GC CPU above 15% means the Go runtime is spending too much time reclaiming memory. Identify the consumer: check sys_go_allocbytes and sys_cgo_allocbytes to determine whether the Go heap or the Pebble cache is growing. If SQL memory is the culprit, find and kill expensive queries. If --cache or --max-sql-memory are set too aggressively for available memory, reduce them. Changing these flags requires a node restart. In containers, set them as fractions of the container limit, not the host.

I/O-driven system CPU

When sys_cpu_sys_ns dominates, the CPU is spending time in kernel filesystem and I/O code paths. Check iostat for disk utilization and latency. If compaction is the driver, check L0 sublevel count: elevated L0 means compaction is working harder, consuming more I/O and system CPU. Separating WAL onto a dedicated device eliminates the most latency-sensitive I/O from the compaction bottleneck.

Admission control at capacity

Sustained admission control queuing means the system is at capacity. Admission control provides temporary protection, not a permanent throughput limiter. If foreground queues (kv, sql-kv-response) are consistently queuing, the node needs more CPU or fewer ranges. If only elastic-cpu is queuing, the system is correctly prioritizing foreground work over background tasks, which may be acceptable.

Prevention

Size for N+1. CPU headroom must absorb the loss of one node. When a node fails, its work redistributes to the remaining nodes. For an N-node cluster, steady-state per-node CPU should be below 80% multiplied by (N-1)/N. For a 3-node cluster, that means approximately 53%. For a 5-node cluster, approximately 64%. This is stricter than the 60-70% steady-state target because it accounts for failover load.

Monitor per-node, not cluster-wide. The cluster average is meaningless for CPU diagnosis. One melting leaseholder can hide behind healthy global metrics. Always alert on per-node CPU with the user/system split.

Track range count as a scaling dimension. Most teams monitor data volume and query throughput but miss range count growth. Each range adds Raft processing overhead proportional to CPU, not to query load. Track range count per node and plan node additions before hitting 50,000.

Avoid burstable or shared-core VMs. CockroachDB recommends dedicated CPU resources. Burstable instances that limit CPU under sustained load cause unpredictable latency spikes and can trigger admission control throttling during normal operation.

Watch for P50 latency creep. When CPU approaches saturation, P50 latency starts rising before P99. A rising P50 with stable P99 means the system is uniformly slowing down, not just experiencing tail latency. This is an early warning that CPU capacity is exhausted.

How Netdata helps

• Per-second granularity on sys_cpu_user_ns and sys_cpu_sys_ns exposes the user/system split at resolution fine enough to catch transient spikes that 15-30 second scrape intervals miss.
• Per-node dashboards prevent the cluster-average trap by showing each node’s CPU independently, making hot leaseholder asymmetry immediately visible.
• Correlation across subsystems lets you see CPU saturation alongside L0 sublevel count, admission control queue depth, and SQL latency in a single timeline, shortening the path from symptom to root cause.
• Range count tracking per node provides early warning when data growth is silently increasing Raft tick overhead.

Netdata’s database monitoring brings these signals together with per-second collection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls