CockroachDB Raft snapshot storm: when followers fall too far behind

Raft snapshots are CockroachDB’s fallback when incremental log replication is no longer possible. A leader sends a full copy of a range (up to 512 MiB by default) to a follower that has fallen so far behind that the log entries it needs have already been truncated. In steady state, this almost never happens. When it starts happening at scale, it creates a resource-draining cycle that can degrade an entire cluster.

The symptom: range_snapshots_generated and range_snapshots_applied_initial climb from near zero to a sustained elevated rate. Each snapshot transfers a large block over the network and writes it to disk on the receiving node. That I/O competes with foreground traffic and with the node’s own log application, so the lagging follower falls further behind while applying the snapshot. The next range that needs a snapshot hits the same overloaded node. The pattern feeds itself.

The precursor signal is raftlog_behind, a per-store gauge that sums the log entry deficit across all ranges where that node’s replica trails the leader. Once this begins climbing on a single node, you have a warning window before snapshots become necessary. Once snapshots fire at volume, you are already in the cycle and need to act.

What this means

Under normal operation, CockroachDB replicates writes through the Raft log. The leader proposes entries, followers receive and persist them, and once a quorum has persisted an entry it is committed and applied to each replica’s Pebble state machine. The leader periodically truncates old entries that all followers have acknowledged, keeping the log bounded.

A follower needs a full snapshot when the leader no longer holds the log entries it requires. This happens when the follower’s apply progress falls behind the leader’s truncation point. The leader then sends the entire range state. With ranges sized at up to 512 MiB by default, each snapshot is a substantial I/O operation on both sender and receiver.

Snapshot application is not free. While the receiving node writes the snapshot data into its LSM tree, it continues accumulating lag on its other ranges. The I/O competes with compaction, foreground writes, and Raft log application. A node that was slightly behind can spiral into a state where it needs snapshots across many ranges simultaneously, each consuming disk bandwidth and network capacity.

flowchart TD
    A[Steady state: followers
replicate via log entries] --> B[Follower slows down
disk, CPU, or network]
    B --> C[raftlog_behind climbs]
    C --> D{Leader still has
needed log entries?}
    D -->|Yes| E[Catch up via
log replication]
    D -->|No, truncated| F[Full Raft snapshot
up to 512 MiB]
    F --> G[Snapshot I/O competes
with foreground traffic]
    G --> H[Follower falls
further behind]
    H --> C
    G --> I[Cluster-wide latency
and throughput degrade]

A single slow node recovering thousands of ranges can generate sustained snapshot traffic that saturates its disk and network for hours. If the cluster is also handling foreground load, the snapshot traffic degrades query latency across all nodes, not just the one being healed.

Common causes

Quick checks

Run these read-only commands on any node in the cluster. All are safe for production use during an incident.

# Check Raft log behind per node (the precursor signal)
curl -s http://localhost:8080/_status/vars | grep raftlog_behind

# Check snapshot generation and application rates
curl -s http://localhost:8080/_status/vars | grep range_snapshots

# Check under-replicated range count
curl -s http://localhost:8080/_status/vars | grep ranges_underreplicated

# Check inter-node RPC latency
curl -s http://localhost:8080/_status/vars | grep round_trip_latency

# Check L0 sublevels per store (snapshot ingest can cause storage pressure)
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

# Check write stalls (a consequence of snapshot I/O saturating the disk)
curl -s http://localhost:8080/_status/vars | grep storage_write_stalls

# Check WAL fsync latency (write-path health on the receiving node)
curl -s http://localhost:8080/_status/vars | grep storage_wal_fsync_latency

# Check disk I/O on the lagging node
iostat -xz 1 3

How to diagnose it

1. Confirm the snapshot rate is abnormal. Check the rate of change of range_snapshots_generated (a counter) over a 10-minute window. In steady state, this should be near zero. During planned rebalancing or after a known node failure, some snapshot activity is expected. If the rate is elevated without an operational cause, proceed.

2. Identify which nodes are lagging. Check raftlog_behind per store. The aggregate metric can mislead: 10,000 entries behind across 5,000 ranges (2 per range) is fine. 10,000 entries behind concentrated on a single node indicates that node cannot apply log entries fast enough. Focus on the node with the highest sustained value.

3. Determine why the lagging node cannot keep up. Check three resources on the affected node:

   • Disk I/O: Run iostat -xz 1 and look at await and utilization. If the disk is saturated, the node cannot persist Raft log entries or apply them fast enough.
   • CPU: Check sys_cpu_user_ns and sys_cpu_sys_ns rates. CPU saturation delays Raft processing. A node with 50,000 ranges needs significant CPU just for Raft ticking.
   • Network: Check round_trip_latency for the affected node’s pairs. Elevated latency means log entries arrive slowly, and the follower falls behind before it can begin applying.

4. Check for secondary storage pressure. Snapshot ingest writes a large block of data into the LSM tree. Check storage_l0_sublevels on the receiving node. If L0 is climbing past 10-20, the snapshot ingest is causing compaction to fall behind, which further degrades the node’s ability to apply entries. This is the mechanism by which snapshot storms trigger the LSM compaction death spiral.

5. Assess whether this is recovery-induced. Check ranges_underreplicated. If it is elevated and the cluster recently lost a node or completed a decommission, the snapshot traffic is the cluster healing itself. The question is whether the healing rate is acceptable or degrading foreground traffic too aggressively.

6. Evaluate foreground impact. Check SQL latency and admission control queue depth. If admission.io.overload is elevated or the store-write queue is deep, snapshot I/O is competing with foreground writes and the system is throttling user traffic to protect itself.

Metrics and signals to monitor

Fixes

Chronically slow node

If one node consistently shows elevated raftlog_behind and its disk or CPU is saturated, the node is undersized for its workload. Options:

• Address the resource bottleneck. Add disk I/O capacity (faster storage, more provisioned IOPS on cloud volumes) or add CPU. This is the right fix if the node is otherwise healthy.
• Reduce the node’s load. If the node holds too many ranges, the cluster may be under-provisioned for its range count. CockroachDB recommends keeping per-node range count below 50,000 for typical workloads.
• Decommission the node. If the node is persistently impaired (failing disk, insufficient resources), decommission it so the cluster redistributes ranges to healthy nodes. Run cockroach node decommission <node_id> --host=<host:port> and monitor progress. Decommissioning itself generates snapshot traffic as ranges move off the node, so only do this when other nodes have capacity to absorb the transfer.

Network bottleneck

If round_trip_latency is elevated for specific node pairs, the network path between them is degraded. Check for NIC errors, cloud bandwidth limits, or asymmetric routing. CockroachDB multiplexes all inter-node traffic over a single gRPC connection per node pair, so there is no application-level way to prioritize Raft heartbeats over snapshot data. If snapshot transfers saturate the link, heartbeat delivery also suffers, which can cause liveness issues.

Recovery overload

If the snapshot storm is the cluster healing after a node failure, the traffic is legitimate but may be too aggressive. Two cluster settings control snapshot bandwidth:

-- Check current snapshot rate limits
SHOW CLUSTER SETTING kv.snapshot_rebalance.max_rate;
-- kv.snapshot_recovery.max_rate existed until v23.2 (default 32 MiB) and was removed in v24.1

Defaults: kv.snapshot_rebalance.max_rate is 32 MiB. kv.snapshot_recovery.max_rate was also 32 MiB but was deprecated in v23.2 and removed in v24.1, so on current releases kv.snapshot_rebalance.max_rate governs all snapshots.

kv.snapshot_rebalance.max_rate throttles snapshots during load-based rebalancing and up-replication. kv.snapshot_recovery.max_rate throttled recovery snapshots until it was deprecated in v23.2 and removed in v24.1. If set too high, snapshot traffic saturates disk and network, degrading foreground queries. If set too low, recovery takes excessively long and under-replication persists.

To reduce foreground impact during recovery:

-- Reduce recovery snapshot rate (this is a conservative example; adjust to your environment)
-- This takes effect immediately at runtime.
SET CLUSTER SETTING kv.snapshot_rebalance.max_rate = '4 MiB';

This is a tradeoff. Lower values mean slower recovery but less foreground impact. Monitor ranges_underreplicated to confirm the cluster is still converging.

Foreground load reduction

If snapshot traffic is degrading user-facing queries, reduce competing load:

• Pause bulk operations (IMPORT, RESTORE, large schema changes) that generate write I/O competing with snapshot apply.
• If admission control is throttling, the system is already at capacity. Reducing foreground load is the only way to give snapshots room to complete without degrading user queries.

Prevention

• Monitor raftlog_behind proactively. This is the precursor signal. Sustained elevation on any single node means that node is at risk of triggering snapshot storms. Investigate before snapshots begin.
• Track snapshot rate as a baseline metric. Know what zero looks like in your cluster. Any deviation should be attributable to a known operational event.
• Ensure disk I/O headroom. Compaction throughput should be at least 2x sustained write ingestion rate. Without this headroom, snapshot ingest will tip the node into LSM compaction debt.
• Size nodes for their range count. A node with 50,000 ranges needs substantially more CPU for Raft ticking alone than a node with 5,000. Heterogeneous node sizes create imbalances where smaller nodes become chronic laggards.
• Watch for range count growth. As data grows, range count grows proportionally. Plan node additions before per-node range count exceeds 50,000.
• Validate recovery behavior during maintenance. During planned decommissioning or rolling restarts, watch range_snapshots_generated and ranges_underreplicated. If recovery is too slow or too aggressive, adjust rate limits before it becomes an incident.

How Netdata helps

• Per-second metric resolution on raftlog_behind, range_snapshots_generated, and range_snapshots_applied_initial lets you see the onset of a snapshot storm in real time, not minutes after it begins.
• Per-store dashboards for storage_l0_sublevels and storage_write_stalls show immediately when snapshot ingest is causing secondary storage pressure on the receiving node.
• Correlation across nodes lets you identify the single lagging node and its disk, CPU, and network metrics in one view, rather than grepping through per-node Prometheus endpoints during an incident.
• Anomaly detection on raftlog_behind can surface a node trending toward snapshot territory before the rate spikes, giving you the warning window that static threshold alerting often misses.
• Inter-node RPC latency dashboards help distinguish network-caused lag from disk-caused lag, which determines whether the fix is infrastructure or node-level.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and anomaly detection.

Related guides

• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB monitoring checklist: the signals every production cluster needs