CockroachDB range count per node too high: Raft ticking as a scaling dimension

CPU utilization is climbing across one or more CockroachDB nodes, and the usual explanations do not fit. Query throughput is flat. Disk I/O looks healthy. Admission control is not queuing. L0 sublevels are in single digits. CPU keeps trending upward, and the only change is that the database holds more data.

The likely cause is the scaling dimension most teams never instrument: range count per node. Every range in CockroachDB is an independent Raft consensus group. Even idle, each range’s Raft state machine ticks at a fixed interval, consuming CPU separate from query execution, compaction, or any workload-driven process. This baseline overhead grows linearly with range count per node.

At 5,000 ranges per node, this overhead is rounding error. At 50,000, it is measurable. At 200,000, it is a different operating regime where Raft ticking alone consumes multiple CPU cores.

What this means

The entire CockroachDB keyspace is divided into ranges, each approximately 512 MiB by default (increased from 64 MiB in v20.1). Each range is replicated (typically 3x) and every replica participates in a Raft consensus group. One replica is the Raft leader, one is the leaseholder, and in the common case they are co-located on the same node.

Raft is not idle when there are no writes. Every range’s Raft state machine ticks at a fixed interval. The default is 500ms, controlled by the COCKROACH_RAFT_TICK_INTERVAL environment variable. Each tick involves processing heartbeat messages, checking election timers, and maintaining state. This work happens for every range on the node, every tick interval, regardless of traffic.

Rough guideline: approximately 1% of a CPU core per 500 to 1,000 ranges, just for Raft ticking. A node holding 50,000 ranges consumes roughly half a core to a full core before any query executes. At 100,000 or more, Raft ticking alone consumes multiple cores.

Two distinctions matter for diagnosis:

• Replica count vs. lease count: The ranges metric reports total replicas per store. The leases_count metric reports leases held. A node may hold 10,000 replicas but only 3,000 leases. Replica count drives Raft processing overhead. Lease count drives query serving load. Both consume CPU, but through different mechanisms.
• Quiesced ranges still cost CPU: CockroachDB quiesces ranges that have had no proposals for several tick intervals, reducing heartbeat network traffic. But quiesced ranges still require processing when unpacked. Quiescence reduces but does not eliminate the marginal cost of maintaining inactive ranges.

When range count consumes enough baseline CPU to shrink headroom, the node becomes brittle. Burst traffic, compaction spikes, or GC pauses that previously fit within idle capacity now collide with the Raft ticking floor. If CPU saturation delays Raft heartbeats, the node loses leases or liveness. The cluster redistributes ranges to surviving nodes, increasing their range count and baseline CPU, accelerating the cycle.

flowchart TD
    A[Data grows, no nodes added] --> B[Range count per node rises]
    B --> C[Raft ticking CPU grows]
    C --> D[Less headroom for queries and compaction]
    D --> E[CPU saturates during bursts]
    E --> F[Raft heartbeats delayed]
    F --> G[Node loses leases or liveness]
    G --> H[Ranges redistributed to survivors]
    H --> B

Common causes

Quick checks

# Check range and lease count per store
curl -s http://localhost:8080/_status/vars | grep -E '^ranges\b|leases'

# Check CPU utilization
curl -s http://localhost:8080/_status/vars | grep sys_cpu

# Check for under-replication (rebalancing may be stuck)
curl -s http://localhost:8080/_status/vars | grep ranges_underreplicated

# Check store capacity to estimate expected range count
curl -s http://localhost:8080/_status/vars | grep -E 'capacity'

# Check lease transfer rate (elevated rate may indicate instability)
curl -s http://localhost:8080/_status/vars | grep leases_transfers

# Per-node range distribution (admin-only, expensive, diagnosis only)
<!-- TODO: verify column name in crdb_internal.ranges for the target CockroachDB version -->
# cockroach sql -e "SELECT node_id, count(*) FROM crdb_internal.ranges GROUP BY node_id ORDER BY count DESC;"

How to diagnose it

1. Get per-store range counts. Scrape ranges per store. With replication factor 3, each unique range produces 3 replicas. The sum of ranges across all stores divided by the replication factor gives the cluster’s unique range count. Expected total unique ranges is approximately total data divided by 512 MiB.

2. Check for imbalance. Calculate the mean range count per node. If any node deviates more than 20% from the mean, the allocator is not distributing evenly. Common causes: zone constraint conflicts, heterogeneous node sizes, or a node with insufficient disk space to accept more replicas.

3. Compare range count to data volume. If your cluster holds 10 TB of data with RF=3, you have approximately 20,000 unique ranges. Across 5 nodes, each holds roughly 12,000 replicas. If actual range count is significantly higher, you have many small ranges from excessive splitting on sequential key patterns.

4. Correlate CPU with range count. If CPU is elevated but query throughput, disk I/O, and compaction are normal, Raft ticking is likely a significant consumer. Profile with Go’s pprof endpoint (/debug/pprof/profile) to confirm. Look for Raft-related functions consuming a disproportionate share of CPU time.

5. Check startup time. Higher range count means longer startup. A node that restarted in 30 seconds six months ago and now takes several minutes is carrying significantly more ranges. Track sys_uptime after restarts to observe the trend.

6. Verify lease distribution. Compare ranges (replicas) to leases_count per store. If one node holds disproportionately many leases, it bears both Raft overhead and query serving load, compounding CPU pressure. Note that crdb_internal.ranges queries are expensive and perform cluster-wide RPC fan-out. Use them for manual diagnosis only, never for continuous monitoring.

Metrics and signals to monitor

Fixes

Add nodes

The most direct fix. Adding nodes distributes ranges across more machines, reducing per-node Raft overhead. The allocator automatically rebalances replicas and leases. On v23.1+, the load-based rebalancing objective defaults to cpu.

Monitor the rebalancing process. Check that ranges_underreplicated converges to zero and that the new node’s range count trends toward the cluster mean. If the new node is not absorbing ranges, check for zone constraint conflicts or insufficient disk space.

Address range imbalance

If one or two nodes hold significantly more ranges than others (>20% deviation), investigate three common causes:

• Zone constraint conflicts: Zone configs may restrict where replicas can be placed. Tight constraints prevent even distribution. Check with SHOW ZONE CONFIGURATIONS;.
• Heterogeneous node sizes: Smaller nodes may be full, preventing the allocator from placing replicas even when aggregate free space exists.
• Disk space: A node near its capacity limit cannot accept new replicas.

Reduce range count via merges

If your cluster has many small ranges from deleted data or sparse key regions, range merges can consolidate them. CockroachDB performs merges automatically for sufficiently empty adjacent ranges. If your data has large sparse regions (TTL-expired data, dropped tables), ensure the merge queue is active and not blocked.

Merges are only effective for sparse or deleted regions. Active data at the default 512 MiB range size cannot be merged further.

Review range size configuration

If you are on a version prior to v20.1 or have manually overridden range size, you may have smaller ranges than necessary. The default is approximately 512 MiB. Larger ranges mean fewer ranges for the same data volume, directly reducing Raft ticking overhead. Very large ranges increase the cost of Raft snapshots (up to the range size per snapshot) and can slow recovery.

Short-term mitigation: reduce background work

If you cannot immediately add nodes, reduce non-essential background work to free CPU for Raft processing. Pause bulk imports, defer schema changes, and ensure admission control is active. This buys time but does not address the underlying range count growth.

Prevention

• Track range count per node as a capacity metric. It belongs alongside CPU, memory, and disk in capacity planning. The per-node target is below 50,000 for typical workloads.
• Project range count growth from data growth. Total unique ranges is approximately total data divided by 512 MiB. If data grows 10% per year and node count is fixed, range count per node grows at the same rate. Plot the trajectory and plan node additions before crossing the threshold.
• Alert on range imbalance exceeding 20% deviation. Persistent imbalance indicates allocator constraints that will compound as range count grows.
• Watch startup time as a leading indicator. Increasing startup time signals growing range count before CPU metrics reflect the problem.
• Account for replication factor in capacity planning. Increasing RF from 3 to 5 increases total replica count by 67% without adding data, directly increasing per-node range count.

How Netdata helps

• Per-second range and lease metrics. Netdata scrapes ranges and leases_count per store at high frequency, giving immediate visibility when range count trends upward or rebalancing creates imbalance.
• CPU correlation. Overlay range count with sys_cpu_user_ns and sys_cpu_sys_ns to confirm whether baseline CPU is growing in lockstep with range count. When Raft ticking is the driver, the correlation is tight and query throughput stays flat.
• Startup time tracking. sys_uptime after restart events reveals whether startup time is creeping upward as range count grows.
• Lease transfer and liveness correlation. High range count can destabilize a node during CPU bursts. Correlating leases_transfers_success and node liveness status with CPU saturation distinguishes range-count-driven instability from other causes.
• Growth trajectory. Long-term retention of per-store range count lets you project when you will cross the 50,000 per node threshold, giving lead time to add capacity.

See CockroachDB monitoring with Netdata for setup details.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability