CockroachDB under-replicated ranges: ranges_underreplicated and the healing margin

When ranges_underreplicated rises above zero, CockroachDB has ranges with fewer live replicas than the configured replication factor. The replicate queue is already working to heal them by transferring Raft snapshots to new target nodes, but until healing completes, those ranges have reduced or zero fault tolerance margin.

The key operational question is not whether under-replicated ranges exist (they will, transiently, after almost any node event), but whether the cluster is healing fast enough to stay ahead of the next failure. This is the healing margin: the buffer between the current replication state and the point where a range loses quorum and becomes unavailable.

With a default replication factor of 3, a fully healthy range tolerates one node failure. An under-replicated range with only 2 replicas still has quorum (2 of 3) and remains available for reads and writes, but it cannot survive another failure. If another replica is lost before the replicate queue up-replicates, the range drops to 1 replica, loses quorum, and moves from ranges_underreplicated to ranges_unavailable. That transition is the difference between a risk state and live data unavailability.

The ranges_underreplicated metric is a per-store gauge exposed at the Prometheus endpoint. It counts ranges whose number of live replicas is below the configured replication factor for their zone. Non-voting replicas are not counted toward the replication factor.

# Check current under-replicated range count
curl -s http://localhost:8080/_status/vars | grep ranges_underreplicated

When the healing margin is zero (under-replicated with no progress), the cluster is in a race it cannot afford to lose.

flowchart TD
    A["All replicas live
ranges_underreplicated = 0"] -->|"node loss or disk failure"| B["Under-replicated
ranges_underreplicated above 0"]
    B -->|"replicate queue heals
via Raft snapshots"| A
    B -->|"healing blocked
30+ min, not decreasing"| D["Sustained under-replication
zero fault tolerance"]
    B -->|"another failure before healing"| C["Quorum lost
ranges_unavailable above 0"]
    D -->|"any additional failure"| C

CockroachDB ships a Prometheus alerting rule for UnderreplicatedRanges that fires on ranges_underreplicated > 0 with a for: 10m0s duration. This is a reasonable starting point, but it does not distinguish between expected healing after maintenance and a stuck recovery that leaves the cluster one failure away from unavailability.

Severity triage

• TICKET: Any nonzero value. Expected after planned maintenance (node restarts, decommissions) and should converge to zero within minutes to hours depending on data volume. Track it, but do not wake anyone.
• PAGE: Under-replication sustained for more than 30 minutes, the count is not decreasing, and another failure or risk is simultaneously present. This combination indicates the cluster cannot heal on its own and is at elevated risk of data unavailability.

The “not decreasing” qualifier is critical. A cluster that lost a node and is actively up-replicating 10,000 ranges is in a known, recovering state. A cluster that has had 500 under-replicated ranges for an hour with no downward trend has a blocked replicate queue.

Common causes

Quick checks

# Under-replicated range count (per-store gauge)
curl -s http://localhost:8080/_status/vars | grep ranges_underreplicated

# Unavailable ranges (quorum lost - this is the emergency state)
curl -s http://localhost:8080/_status/vars | grep ranges_unavailable

# Live node count
curl -s http://localhost:8080/_status/vars | grep liveness_livenodes

# Per-node liveness detail
curl -s http://localhost:8080/_status/liveness | python3 -c "
import json, sys
data = json.load(sys.stdin)
for e in data.get('livenesses', []):
    nid = e.get('node_id', '?')
    alive = e.get('alive', 'UNKNOWN')
    print(f'Node {nid}: alive={alive}')"
# <!-- TODO: verify exact JSON field names in _status/liveness response -->

# Store capacity across all nodes
curl -s http://localhost:8080/_status/vars | grep -E 'capacity_(used|available)'

# Raft snapshot activity (should be elevated during healing)
curl -s http://localhost:8080/_status/vars | grep range_snapshots
# <!-- TODO: verify exact metric name (range_snapshots_generated vs rangesnapshots_generated) -->

# Lease transfer rate (elevated during node recovery)
curl -s http://localhost:8080/_status/vars | grep leases_transfers

How to diagnose it

1. Establish the timeline. When did ranges_underreplicated start rising? Correlate with node liveness events, decommissions, rolling restarts, or zone config changes. If the increase coincides with planned maintenance and the count is decreasing, the cluster is healing normally.

2. Check whether the count is decreasing. Compare the current value against the value 10 minutes ago. If it is flat or increasing after 30 minutes, healing is blocked. If it is decreasing, estimate the convergence rate and verify it reaches zero.

3. Identify the blocking condition. The replicate queue needs three things to up-replicate a range: a valid target node (matching zone constraints), available disk space on that target, and snapshot transfer bandwidth. Check each in order.

4. Verify target node availability. Are there enough live nodes with matching zone constraints to accept new replicas? Heterogeneous node sizes can prevent placement even when aggregate free space exists.

5. Check disk space on target stores. If candidate target nodes are near capacity, the allocator cannot place replicas there. CockroachDB treats stores below approximately 20% free space as invalid targets for replica placement.

6. Inspect snapshot transfer rates. During healing, range_snapshots_generated and related snapshot metrics should be elevated. If they are low or zero while under-replicated count is high, snapshots are being throttled or failing.

7. Check for zone config conflicts. If zone constraints or locality requirements are too restrictive, the allocator may have no valid targets. Compare the constraints in your zone configs against the actual node topology.

8. Look for decommissioned node residue. After decommissioning a node, a range may still map one of its replicas to the decommissioned node, keeping it under-replicated until the replicate queue processes the removal. On large clusters, the replicate queue scanner can take significant time to process all ranges.

Metrics and signals to monitor

Fixes

Node loss

If a node crashed or was OOM-killed, the cluster will up-replicate its ranges once replacements are available. Do not restart services as a first action. Instead:

1. Verify the node is actually down (not just slow). Check liveness status.
2. If the node will recover (restart, replace), let it come back. The replicate queue will reconcile the stale entries.
3. If the node is permanently lost, decommission it so the cluster removes its replicas from the range maps.

Disk full on target nodes

If candidate target stores lack space, up-replication cannot proceed. Free space on the affected stores or add capacity. Deleting data does not immediately free space because MVCC tombstones must propagate through compaction. If multiple stores are full, the cluster has a systemic capacity problem that no queue tuning will fix.

Snapshot throttling

The kv.snapshot_rebalance.max_rate and kv.snapshot_recovery.max_rate cluster settings control how fast Raft snapshots transfer during rebalancing and recovery respectively. If these are set too conservatively, healing takes longer than necessary. Increasing them speeds recovery but the additional I/O competes with foreground traffic. On clusters with network-attached storage, measure compaction and snapshot I/O headroom before raising limits. Check current values with:

SHOW CLUSTER SETTING kv.snapshot_rebalance.max_rate;
SHOW CLUSTER SETTING kv.snapshot_recovery.max_rate;

Zone configuration conflicts

If zone constraints specify localities or attributes that no available node matches, the allocator cannot place replicas. Review zone configs against actual node topology. Common issues: nodes in a locality that was removed, constraints referencing store attributes that changed after node replacement, or survivability requirements that exceed the current node count.

Decommission residue

After decommissioning a node, some ranges may still map replicas to the decommissioned node until the replicate queue processes the removal. If the under-replicated count persists long after decommission, verify the decommission completed successfully and the node is fully removed from the cluster. Re-running cockroach node status against the decommissioned node confirms whether any replicas remain.

Recovery-induced saturation

When the cluster heals after a major failure, the healing work itself (snapshot transfers, rebalancing) can saturate disk and network, degrading foreground traffic. If foreground latency is impacted during healing:

1. Confirm healing is progressing, not thrashing.
2. Reduce concurrent background jobs (backups, schema changes, stats collection).
3. Consider lowering snapshot rate limits temporarily to prioritize foreground I/O.

Prevention

• Size for one-node loss. In an N-node cluster, steady-state resource utilization should be low enough that losing one node and redistributing its work does not push surviving nodes past their limits. For a 3-node cluster, target below approximately 53% utilization. For 5 nodes, below approximately 64%. This formula comes from 80% times (N-1)/N.

• Keep disk space above 20% on all stores. Below this threshold, stores become invalid targets for replica placement, and compaction may fail. CockroachDB documentation recommends at least 20% free at all times, with 30% preferred for compaction amplification and snapshot staging.

• Monitor during rolling restarts. Brief under-replication during rolling restarts is expected if restarts are sequenced with sufficient delay. Suppress or downgrade under-replication alerts during planned maintenance windows, or when node liveness shows a draining state.

• Validate zone configs after topology changes. When nodes are added, removed, or relocated, verify that zone constraints still match the actual topology. A zone config referencing a locality that no longer exists silently blocks replica placement.

• Watch the replicate queue on large clusters. Clusters with hundreds of thousands of ranges can take significant time for the replicate queue to scan and process. The scanner does not prioritize under-replicated ranges by severity. Plan maintenance windows with enough recovery time for the queue to complete a full pass.

• Track snapshot rate as a leading indicator. During healthy healing, range_snapshots_generated should be elevated. If under-replicated count is high but snapshot generation is near zero, the replicate queue is blocked before it even attempts a transfer.

How Netdata helps

• Per-second granularity on ranges_underreplicated lets you see the healing trend (decreasing, flat, or increasing) within minutes rather than waiting for multiple scrape intervals to establish a direction.

• Correlation with ranges_unavailable distinguishes between under-replication (risk state, still available) and unavailability (quorum lost, live impact) without switching dashboards.

• Node liveness and store capacity alongside replication metrics helps identify whether healing is blocked by a missing target node, disk space exhaustion, or a zone config conflict in the same view.

• ML-based anomaly detection on snapshot rates and lease transfer rates surfaces stuck snapshots or lease thrashing that raw threshold alerts miss.

Netdata’s database monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum