CockroachDB WAL fsync latency high: the most direct write-path health signal

Elevated storage_wal_fsync_latency means the storage device is taking too long to durably persist write-ahead log entries. Every write in CockroachDB must complete a WAL fsync before Raft can acknowledge it. Fsync latency is the gating signal for the entire write path.

On healthy local SSDs, WAL fsync P99 should be under 10ms. Sustained values above 50ms mean the device is struggling. Above 200ms, the node is approaching disk stall territory, where CockroachDB’s built-in stall detection may terminate the process to prevent data inconsistency. Disk stalls cause self-termination before utilization metrics react: a volume reporting 60% utilization can stall on an individual fsync for seconds if the underlying hardware is failing, the I/O scheduler is misbehaving, or a cloud provider is throttling IOPS.

What this means

Every write in CockroachDB follows this path: the client sends the write to the leaseholder, the leaseholder proposes it through Raft, and the Raft log entry must be persisted to the WAL via fsync before Raft considers it committed. The fsync is synchronous and on the critical path. If the fsync takes 200ms instead of 2ms, every write on that store waits 200ms.

Other disk metrics (utilization, throughput, IOPS) are indirect proxies. They can look healthy while individual fsync operations stall.

CockroachDB has built-in disk stall detection. Every 10 seconds, the storage engine checks whether the WAL has been synced since the last check. If it has not, a warning is logged to the STORAGE channel. If the stall persists past the threshold controlled by the COCKROACH_ENGINE_MAX_SYNC_DURATION environment variable , the cockroach process terminates itself.

flowchart TD
    A[Write arrives at leaseholder] --> B[Raft proposal]
    B --> C[WAL fsync on critical path]
    C --> D{fsync latency}
    D -->|Normal: P99 under 10ms| E[Raft commit, client ACK]
    D -->|Elevated: 50 to 200ms| F[All write latency rises]
    D -->|Stall: no sync for 20s| G[disk stall detected log]
    F --> H[Admission control throttles writes]
    G --> I[Node self-terminates]

The self-termination is a safety measure. A disk that cannot fsync cannot guarantee durability. Continuing to accept writes would risk acknowledging operations that may not survive a crash.

The key operational implication: WAL fsync latency starts climbing well before the stall detection threshold fires. If you are not monitoring it, your first indication of a storage problem may be a node unexpectedly terminating.

Common causes

CauseWhat it looks likeFirst thing to check
Cloud volume throttlingPeriodic spikes, often correlating with burst credit or provisioned IOPS exhaustionCloud provider IOPS metrics
Failing SSDProgressive latency increase, eventually stallssmartctl health, iostat await trend
Shared device contentionFsync latency rises during compaction bursts or backupsWhether WAL and data files share the same device
SELinux overheadPersistently elevated fsync without hardware cause on one nodeSELinux mode and audit logs
Kernel or filesystem issueFsync high on one node, normal on identical hardwareKernel version, filesystem type, mount options

Quick checks

# Check WAL fsync latency histogram (per store)
curl -s http://localhost:8080/_status/vars | grep storage_wal_fsync_latency

# Check disk stall detection status
curl -s http://localhost:8080/_status/vars | grep -E 'storage_disk_stalled|storage_disk_slow'

# Check Raft log commit latency (should track WAL fsync)
curl -s http://localhost:8080/_status/vars | grep -E 'exec_latency|raft_process_logcommit'

# Check OS-level disk write latency (run for several intervals)
iostat -xz 1 5

# Check disk SMART health
smartctl -H /dev/nvme0n1

# Check admission control store-write queue (write throttling signal)
curl -s http://localhost:8080/_status/vars | grep admission

# Check for disk stall log messages
grep -i "disk stall" /path/to/cockroach-data/logs/cockroach.log

# Check SELinux mode
getenforce

How to diagnose it

  1. Confirm the metric is actually elevated. Pull the storage_wal_fsync_latency histogram and compute P99 from the bucket distribution. On local SSD, P99 above 50ms is concerning. Above 200ms is critical. On cloud volumes, compare against your provisioned performance baseline rather than absolute thresholds.

  2. Correlate with Raft log commit latency. Check raft_process_logcommit_latency in the Prometheus output. If it tracks WAL fsync latency closely, the bottleneck is in the fsync path itself, not in Raft processing or network round-trips. If Raft commit latency is high but fsync latency is normal, the problem is elsewhere (slow quorum replica, network latency).

  3. Check OS-level disk latency. Run iostat -xz 1 and watch w_await (write latency) and %util. On SSDs, w_await above 5ms indicates queueing. A sudden spike in w_await usually means the device write cache flushed or the device is throttling. Note that %util is misleading for NVMe and multi-queue devices: a device at 100% util with sub-millisecond await is healthy.

  4. Determine if the problem is isolated to one store. The metric is per-store. A node with multiple stores can have one healthy and one failing. Check all stores individually. If only one store is affected, suspect hardware degradation on that device.

  5. Check for cloud volume throttling. On AWS, check CloudWatch VolumeConsumedReadWriteOps (gp3) or BurstBalance (gp2). On GCP, check PD IOPS utilization against provisioned limits. EBS gp3 volumes have a baseline of 3000 IOPS and 125 MiB/s, which moderate CockroachDB workloads can exceed.

  6. Look for disk stall log messages. Search logs for “disk stall detected.” Any occurrence means the stall detection mechanism fired. If the process is still running, it may be within the grace window. If it terminated, you will find the critical message followed by a process exit.

  7. Check whether WAL and data share a device. If the WAL directory and the store data directory are on the same physical device, compaction I/O competes directly with WAL fsync for disk bandwidth. This is the most common avoidable cause of fsync latency spikes.

  8. Rule out SELinux. In rare cases, SELinux policy evaluation adds measurable overhead to fsync system calls. Check getenforce output. If Enforcing, check audit logs for denied operations related to the CockroachDB data directory.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
storage_wal_fsync_latencyDirect measure of write-path healthP99 above 50ms on SSD
storage_disk_stalledNode has detected a stalled disk and may self-terminateAny nonzero value
storage_disk_slowCounter of slow disk operationsIncrementing regularly
raft_process_logcommit_latencyRaft commit waits on WAL fsyncRising in lockstep with fsync latency
exec_latencyKV execution includes storage I/O timeRising alongside fsync latency
iostat w_awaitOS-level write latency per deviceAbove 5ms on SSD
Admission store-write queueSystem throttling writes to protect LSM healthSustained nonzero queue depth
L0 sublevel count (storage_l0_sublevels)Compaction debt creates I/O pressure on shared devicesAbove 10 sustained

Fixes

Cloud volume throttling

Cloud providers throttle block storage when provisioned IOPS or burst budgets are exhausted. Transient stalls that appear as fsync latency spikes can occur frequently under sustained write pressure. The fix is to provision adequate IOPS and throughput for your sustained write rate, not just your average.

For EBS gp3, increase provisioned IOPS and throughput above the 3000 IOPS / 125 MiB/s baseline. For GCP PD, use pd-ssd or pd-extreme rather than pd-standard. Monitor burst credit balance on gp2 volumes: exhaustion forces a drop to baseline performance that causes sustained fsync latency spikes.

If throttling is chronic and budget-constrained, separate the WAL onto a dedicated volume with lower throughput but consistent latency characteristics.

Failing SSD

Check SMART data with smartctl -a /dev/<device>. Look for reallocated sectors, pending sectors, or media errors. A degrading SSD often shows progressively increasing fsync latency before it fails completely.

If the device is failing, drain and decommission the node, replace the underlying storage, and let the cluster up-replicate. Do not raise COCKROACH_ENGINE_MAX_SYNC_DURATION to mask the problem. The node will continue to accept writes it cannot durably persist, risking data loss on crash.

Shared device contention

If WAL and data files share the same physical device, compaction I/O and backup I/O compete with WAL fsync for disk bandwidth. During compaction bursts or backup operations, fsync latency can spike even on a healthy SSD.

Place the WAL on a dedicated fast device to eliminate the most latency-sensitive I/O from the compaction bottleneck.

WAL failover (v24.3+)

WAL failover writes to a secondary WAL device when the primary device stalls, providing resilience against transient disk stalls.

When WAL failover is enabled, the recommended COCKROACH_ENGINE_MAX_SYNC_DURATION increases to 40 seconds (default 20s). There is also COCKROACH_LOG_MAX_SYNC_DURATION for log file sink stalls, also recommended at 40s with WAL failover.

One operational gotcha: the storage_wal_fsync_latency metric does not currently distinguish between the primary and secondary WAL directories. If WAL failover is active, you cannot tell from the metric alone which device is slow. You must check OS-level disk metrics for both devices.

SELinux overhead

If SELinux is in Enforcing mode and fsync latency is elevated without hardware cause, check audit logs for denied operations. Consider testing with SELinux in Permissive mode to isolate the effect. If confirmed, work with your security team to tune the policy rather than disabling SELinux entirely.

Prevention

  • Monitor WAL fsync latency per store. A per-second collection interval catches transient spikes that 15-30 second scrapes miss.
  • Use dedicated WAL devices on latency-sensitive workloads. Separating WAL from data eliminates compaction contention for the most latency-critical I/O.
  • Right-size cloud volumes. Provisioned IOPS and throughput caps can silently throttle CockroachDB under moderate load. Provision for sustained write rate, not average.
  • Do not raise stall thresholds to mask hardware issues. Increasing COCKROACH_ENGINE_MAX_SYNC_DURATION hides the symptom while the underlying storage degrades toward failure.
  • Distinguish chronic throttling from acute degradation. Track fsync latency patterns over hours. Cloud volume throttling produces periodic spikes; failing hardware produces progressive increase.
  • Consider WAL failover on v24.3+. If your deployment experiences periodic disk stalls from cloud volume throttling, WAL failover provides resilience by writing to a secondary device.

How Netdata helps

  • Per-second WAL fsync latency histograms. Netdata collects storage_wal_fsync_latency at one-second resolution. Individual fsync stalls can cause Raft heartbeat delays even when the average looks fine.
  • Disk stall detection correlation. Netdata surfaces storage_disk_stalled and storage_disk_slow alongside WAL fsync latency in the same dashboard. A nonzero storage_disk_stalled with rising fsync latency confirms the node is in active stall territory.
  • OS-level disk I/O alongside database metrics. Netdata’s disk collector provides iostat-equivalent latency, utilization, and queue depth per device at per-second granularity.
  • Write-path signal chain. Correlating WAL fsync latency with raft_process_logcommit_latency, exec_latency, and sql_service_latency in one view shows where latency enters the write path and how far it propagates to clients.
  • Admission control queue depth. When fsync latency rises, the store-write admission control queue deepens as CockroachDB throttles writes. Netdata surfaces this alongside the fsync metric.
  • ML anomaly detection on fsync latency. Netdata’s anomaly engine learns the normal fsync latency baseline per store, which is especially useful for cloud volumes where the baseline varies with provisioned IOPS.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.