CockroachDB disk stall detected: storage_disk_stalled and node self-termination

When storage_disk_stalled increments on a CockroachDB node, the Pebble storage engine has detected that disk I/O is no longer completing within the expected time window. The node is on a countdown to self-termination. This is not a performance degradation warning. It is a safety mechanism preparing to fire.

CockroachDB writes every committed transaction through a write-ahead log (WAL). The WAL fsync is on the critical path for every write: Raft cannot acknowledge a commit until the log entry is persisted to disk. If that fsync blocks for long enough, the node cannot process Raft heartbeats, cannot commit writes, and cannot renew its liveness record. Rather than continue operating in a state that could produce data inconsistency, CockroachDB terminates the process.

The default self-termination threshold is 20 seconds. If a disk operation remains incomplete past that window, the cockroach process exits. On cloud-attached storage (AWS EBS, GCP Persistent Disk), this typically happens during volume throttling events or burst credit exhaustion. On bare metal, it usually means a failing SSD or a kernel I/O subsystem problem.

What this means

CockroachDB’s disk stall detection is built into the Pebble storage engine. Every 5 seconds, Pebble checks whether the most recent WAL data has been synced to disk via fsync. If the sync has not completed within that interval, Pebble logs a warning on the STORAGE logging channel — “disk slowness detected: %s on file %s (%d bytes) has been ongoing for %0.1fs”. The storage_disk_stalled counter increments.

If the stall persists past the configurable termination threshold (default 20 seconds, controlled by the COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT environment variable), the cockroach process exits. A runtime cluster setting, storage.max_sync_duration, can also control this threshold; if explicitly set, it takes precedence over the environment variable. The self-termination is deliberate. A node with a stalled disk can become a stale leaseholder: it holds range leases it cannot serve while still appearing live to the cluster’s liveness system if the stall is asymmetric (some I/O completes, other operations do not). Exiting the process forces the cluster to treat the node as dead, redistribute its leases to healthy nodes, and restore availability.

Do not confuse disk stalls with write stalls. storage_write_stalls is a counter that increments when Pebble deliberately pauses writes because L0 sublevels or memtable count exceeded safe thresholds. storage_disk_stalled is a counter that increments when the underlying disk device stops completing I/O within the max sync duration. Write stalls are Pebble protecting the LSM tree. Disk stalls are hardware or volume failure.

flowchart TD
    A[WAL fsync blocks] --> B[Pebble logs stall warning]
    B --> C{Persists past
20s threshold?}
    C -->|Yes| D[Process self-terminates]
    C -->|No| E[Stall clears]
    D --> F[Cluster redistributes leases]
    F --> G[Under-replicated ranges heal]
    D --> H[Fix root cause
before restart]

Common causes

Quick checks

Run these read-only checks on the affected node. If the process has already exited, focus on OS-level checks and logs.

# Check if the process is still running or has self-terminated
pgrep -x cockroach && echo "running" || echo "process not found"

# Check disk stall counter (any increase = stall detected)
curl -s http://localhost:8080/_status/vars | grep storage_disk_stalled

# Check WAL fsync latency histogram
curl -s http://localhost:8080/_status/vars | grep storage_wal_fsync_latency

# Check slow disk operation counter
curl -s http://localhost:8080/_status/vars | grep storage_disk_slow

# Check node uptime (short uptime after a stall = recent self-termination)
curl -s http://localhost:8080/_status/vars | grep sys_uptime

# Check for unavailable ranges (downstream impact)
curl -s http://localhost:8080/_status/vars | grep ranges_unavailable

# OS-level disk latency (watch the await column)
iostat -xz 1 3

# Check disk space on the store device
df -h /path/to/cockroach-data

If the process has exited, check the CockroachDB logs for the stall message. The entry appears on the STORAGE channel and includes the file path that could not be synced and the duration of the stall.

How to diagnose it

1. Confirm the stall. If the node is still running, check storage_disk_stalled. If it has incremented, a stall has been detected. If the process has exited, check sys_uptime: a recently restarted node with short uptime after a stall event confirms self-termination.

2. Check WAL fsync latency trend. Pull storage_wal_fsync_latency and look at the P99 bucket. On healthy local SSDs, WAL fsync P99 should be under 10ms. Sustained P99 above 50ms is concerning. Above 200ms is critical and indicates the device is either failing or being throttled. On cloud volumes, compare against provisioned IOPS and throughput limits.

3. Identify the I/O bottleneck. Run iostat -xz 1 and watch the await column. On SSDs, average write latency above 5ms indicates queueing or throttling. Do not rely on %util alone: on NVMe and cloud volumes, %util is misleading because of device parallelism. Latency is the signal that matters.

4. Check for cloud volume throttling. On AWS, check the EBS BurstBalance metric in CloudWatch. gp2 volumes under 1 TB have limited burst credits. When credits exhaust, IOPS drop to the baseline rate of 3 IOPS per GB of volume size. A 100 GB gp2 volume drops to 300 IOPS, which cannot sustain CockroachDB. On GCP, check Persistent Disk IOPS and throughput against provisioned limits.

5. Check disk space. Run df -h on the store directory. CockroachDB requires at least 20 percent free space for compaction. If free space is below 15 percent, compaction may be unable to run, creating a feedback loop where the LSM tree grows, read amplification increases, I/O demand rises, and fsync latency climbs.

6. Check for hardware failure. Run smartctl -a /dev/<device> if the storage is a local SSD. Look for reallocated sectors, pending sectors, and medium errors. Check dmesg for I/O errors, link resets, or filesystem corruption messages.

7. Determine scope. Check whether the stall is isolated to one node or affecting multiple nodes simultaneously. Multiple nodes stalling at the same time points to shared infrastructure: a storage backend degradation, a network-attached storage controller problem, or a cloud provider incident. If multiple nodes in the same failure domain stall, quorum loss across many ranges is possible.

Metrics and signals to monitor

Fixes

Cloud volume throttling and EBS burst credit exhaustion

If the stall is caused by cloud volume throttling, the provisioned IOPS or throughput is insufficient for the workload.

For AWS EBS:

• gp2 volumes under 1 TB rely on burst credits. When credits exhaust, IOPS drop to 3 IOPS per GB of volume size. A 100 GB gp2 volume drops to 300 IOPS, which cannot sustain CockroachDB.
• Migrate to gp3 volumes with provisioned IOPS. gp3 provides a baseline of 3,000 IOPS and 125 MiB/s throughput, with the ability to provision additional IOPS independently of capacity.
• Size provisioned IOPS based on compaction and WAL throughput requirements, not just steady-state query rate. Compaction generates significant write amplification (10-30x for a leveled LSM tree).

For GCP Persistent Disk, check provisioned IOPS limits. PD-SSD and PD-Balanced have IOPS proportional to disk size. Resize the disk or switch to a higher-performance tier if IOPS are insufficient.

After changing the volume configuration, restart or replace the node. The stall will recur if the underlying capacity problem is not resolved.

Failing SSD

If smartctl reports reallocated sectors, uncorrectable read errors, or medium errors, replace the disk. CockroachDB’s replication factor 3 means the data is safe on other nodes, but the affected node must be decommissioned, the disk replaced, and the node re-provisioned before rejoining the cluster.

Do not attempt to continue running on a failing SSD. Transient I/O errors can escalate into silent data corruption that CockroachDB’s consistency checker may not catch until a later read.

Disk space exhaustion

If the stall coincides with disk space dropping below 15-20 percent free, compaction cannot run, which worsens read amplification, which increases I/O demand, creating a feedback loop. See CockroachDB disk space running out: capacity_available trends and the 20% rule for detailed guidance.

Immediate actions:

• Add capacity to the volume.
• Check for MVCC garbage accumulation that protected timestamps are blocking from being collected.
• Reduce the write rate temporarily to let compaction catch up.

Filesystem or kernel I/O issues

If the disk hardware is healthy but fsync latency is high, investigate the filesystem and kernel I/O stack. Filesystem journal contention (ext4 journaling) or allocation group contention (XFS) can bottleneck fsync under heavy mixed read/write workloads. Check kernel logs for I/O errors, driver issues, or write barrier problems. Review the kernel version for known I/O regressions in your distribution.

Prevention

Monitor WAL fsync latency proactively. The most common monitoring gap is watching disk utilization and IOPS but not WAL fsync latency. Fsync latency is the direct signal of write-path health. On SSDs, P99 should stay under 10ms. Set alerts at 50ms and investigate immediately if it trends upward. Disk stalls cause node self-termination before utilization metrics react.

Use provisioned IOPS volumes. If running on AWS, use gp3 volumes with provisioned IOPS sized for compaction and WAL throughput, not just steady-state query rate. If running on GCP, size Persistent Disks to provide sufficient IOPS for both foreground writes and background compaction.

Keep 20 percent disk space free at all times. CockroachDB requires free space for compaction, Raft snapshots, and WAL files. Below 15 percent, compaction may fail to run, creating a feedback loop that accelerates toward disk stall.

Consider WAL failover. WAL failover relocates the WAL to a secondary store when the primary volume stalls. When enabled, the recommended max sync duration threshold increases from 20s to 40s to account for failover time. Configure via --wal-failover=among-stores or the COCKROACH_WAL_FAILOVER=among-stores environment variable. Caveats:

• WAL failover only protects WAL writes. Data files (SSTables) remain on the primary volume. Reads that miss the Pebble block cache and OS page cache can still stall if the primary disk is impaired.
• The secondary failover volume must be durable. Using an ephemeral volume risks permanent data loss if the VM restarts.
• If using file-based logging, enable asynchronous buffering of file-group log sinks (configure a buffering: block under file-defaults, for example buffered-writes: false with max-staleness: 1s). Without buffering, log writes can block indefinitely during a disk stall, negating WAL failover’s benefit.

Check for log disk stalls separately. The environment variable COCKROACH_LOG_MAX_SYNC_DURATION (default 20s) controls the timeout for log file writes. If logs are on a different partition than the data store and the log disk stalls, the node can self-terminate even if the data disk is healthy.

Monitor EBS burst credit balance. If still using gp2 volumes, monitor the BurstBalance metric and set alerts at 50 percent remaining. Plan migration to gp3 before credits become a recurring problem.

How Netdata helps

• Per-second storage_disk_stalled monitoring. Standard Prometheus scrapes at 10-30 second intervals can miss sub-second stall events. Netdata’s per-second collection captures the counter increment as it happens, before the 20-second termination threshold fires.
• WAL fsync latency histograms at per-second resolution. The storage_wal_fsync_latency histogram is collected continuously, letting you see the latency trend leading up to a stall rather than just the aftermath.
• OS-level disk I/O correlation. Netdata collects iostat-equivalent metrics (await, queue depth, IOPS) alongside CockroachDB metrics. Correlating a spike in OS-level write latency with a rise in storage_disk_slow confirms whether the bottleneck is at the device level or the application level.
• Node uptime tracking. sys_uptime resets to near-zero on self-termination. Netdata flags the restart event and correlates it with preceding disk latency signals.
• Downstream impact visibility. When a node self-terminates, ranges_unavailable and ranges_underreplicated spike. Netdata surfaces these alongside the stall metrics, making the causal chain visible in a single view.

For pre-built dashboards and anomaly detection on these metrics, see CockroachDB monitoring with Netdata.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs