$ guides / cockroachdb / cockroachdb-wal-fsync-latency-high ▌

Operations Guides

CockroachDB WAL fsync latency high: the most direct write-path health signal

Elevated storage_wal_fsync_latency means the storage device is taking too long to durably persist write-ahead log entries. Every write in CockroachDB must complete a WAL fsync before Raft can acknowledge it. Fsync latency is the gating signal for the entire write path.

On healthy local SSDs, WAL fsync P99 should be under 10ms. Sustained values above 50ms mean the device is struggling. Above 200ms, the node is approaching disk stall territory, where CockroachDB’s built-in stall detection may terminate the process to prevent data inconsistency. Disk stalls cause self-termination before utilization metrics react: a volume reporting 60% utilization can stall on an individual fsync for seconds if the underlying hardware is failing, the I/O scheduler is misbehaving, or a cloud provider is throttling IOPS.

What this means

Every write in CockroachDB follows this path: the client sends the write to the leaseholder, the leaseholder proposes it through Raft, and the Raft log entry must be persisted to the WAL via fsync before Raft considers it committed. The fsync is synchronous and on the critical path. If the fsync takes 200ms instead of 2ms, every write on that store waits 200ms.

Other disk metrics (utilization, throughput, IOPS) are indirect proxies. They can look healthy while individual fsync operations stall.

CockroachDB has built-in disk stall detection. The storage engine measures the duration of each WAL sync; when one exceeds the configured threshold, a disk stall detected: message is logged to the STORAGE channel. The threshold defaults to 20s and is controlled by the storage.max_sync_duration cluster setting, whose default comes from the COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT environment variable; when a sync exceeds it, the cockroach process terminates itself (unless storage.max_sync_duration.fatal.enabled is set to false).

flowchart TD
    A[Write arrives at leaseholder] --> B[Raft proposal]
    B --> C[WAL fsync on critical path]
    C --> D{fsync latency}
    D -->|Normal: P99 under 10ms| E[Raft commit, client ACK]
    D -->|Elevated: 50 to 200ms| F[All write latency rises]
    D -->|Stall: single sync exceeds 20s| G[disk stall detected log]
    F --> H[Admission control throttles writes]
    G --> I[Node self-terminates]

The self-termination is a safety measure. A disk that cannot fsync cannot guarantee durability. Continuing to accept writes would risk acknowledging operations that may not survive a crash.

The key operational implication: WAL fsync latency starts climbing well before the stall detection threshold fires. If you are not monitoring it, your first indication of a storage problem may be a node unexpectedly terminating.

Common causes

Cause	What it looks like	First thing to check
Cloud volume throttling	Periodic spikes, often correlating with burst credit or provisioned IOPS exhaustion	Cloud provider IOPS metrics
Failing SSD	Progressive latency increase, eventually stalls	`smartctl` health, `iostat` await trend
Shared device contention	Fsync latency rises during compaction bursts or backups	Whether WAL and data files share the same device
SELinux overhead	Persistently elevated fsync without hardware cause on one node	SELinux mode and audit logs
Kernel or filesystem issue	Fsync high on one node, normal on identical hardware	Kernel version, filesystem type, mount options

Quick checks

# Check WAL fsync latency histogram (per store)
curl -s http://localhost:8080/_status/vars | grep storage_wal_fsync_latency

# Check disk stall detection status
curl -s http://localhost:8080/_status/vars | grep -E 'storage_disk_stalled|storage_disk_slow'

# Check Raft log commit latency (should track WAL fsync)
curl -s http://localhost:8080/_status/vars | grep -E 'exec_latency|raft_process_logcommit'

# Check OS-level disk write latency (run for several intervals)
iostat -xz 1 5

# Check disk SMART health
smartctl -H /dev/nvme0n1

# Check admission control store-write queue (write throttling signal)
curl -s http://localhost:8080/_status/vars | grep admission

# Check for disk stall log messages
grep -i "disk stall" /path/to/cockroach-data/logs/cockroach.log

# Check SELinux mode
getenforce

How to diagnose it

Confirm the metric is actually elevated. Pull the storage_wal_fsync_latency histogram and compute P99 from the bucket distribution. On local SSD, P99 above 50ms is concerning. Above 200ms is critical. On cloud volumes, compare against your provisioned performance baseline rather than absolute thresholds.
Correlate with Raft log commit latency. Check raft_process_logcommit_latency in the Prometheus output. If it tracks WAL fsync latency closely, the bottleneck is in the fsync path itself, not in Raft processing or network round-trips. If Raft commit latency is high but fsync latency is normal, the problem is elsewhere (slow quorum replica, network latency).
Check OS-level disk latency. Run iostat -xz 1 and watch w_await (write latency) and %util. On SSDs, w_await above 5ms indicates queueing. A sudden spike in w_await usually means the device write cache flushed or the device is throttling. Note that %util is misleading for NVMe and multi-queue devices: a device at 100% util with sub-millisecond await is healthy.
Determine if the problem is isolated to one store. The metric is per-store. A node with multiple stores can have one healthy and one failing. Check all stores individually. If only one store is affected, suspect hardware degradation on that device.
Check for cloud volume throttling. On AWS, check CloudWatch VolumeConsumedReadWriteOps (gp3) or BurstBalance (gp2). On GCP, check PD IOPS utilization against provisioned limits. EBS gp3 volumes have a baseline of 3000 IOPS and 125 MiB/s, which moderate CockroachDB workloads can exceed.
Look for disk stall log messages. Search logs for “disk stall detected.” This is logged to the STORAGE channel, which writes to cockroach.log by default. Any occurrence means the stall detection mechanism fired. If the process is still running, it may be within the grace window. If it terminated, you will find the critical message followed by a process exit.
Check whether WAL and data share a device. If the WAL directory and the store data directory are on the same physical device, compaction I/O competes directly with WAL fsync for disk bandwidth. This is the most common avoidable cause of fsync latency spikes.
Rule out SELinux. In rare cases, SELinux policy evaluation adds measurable overhead to fsync system calls. Check getenforce output. If Enforcing, check audit logs for denied operations related to the CockroachDB data directory.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`storage_wal_fsync_latency`	Direct measure of write-path health	P99 above 50ms on SSD
`storage_disk_stalled`	Node has detected a stalled disk and may self-terminate	Any nonzero value
`storage_disk_slow`	Counter of slow disk operations	Incrementing regularly
`raft_process_logcommit_latency`	Raft commit waits on WAL fsync	Rising in lockstep with fsync latency
`exec_latency`	KV execution includes storage I/O time	Rising alongside fsync latency
`iostat` w_await	OS-level write latency per device	Above 5ms on SSD
Admission `store-write` queue	System throttling writes to protect LSM health	Sustained nonzero queue depth
L0 sublevel count (`storage_l0_sublevels`)	Compaction debt creates I/O pressure on shared devices	Above 10 sustained

Fixes

Cloud volume throttling

Cloud providers throttle block storage when provisioned IOPS or burst budgets are exhausted. Transient stalls that appear as fsync latency spikes can occur frequently under sustained write pressure. The fix is to provision adequate IOPS and throughput for your sustained write rate, not just your average.

For EBS gp3, increase provisioned IOPS and throughput above the 3000 IOPS / 125 MiB/s baseline. For GCP PD, use pd-ssd or pd-extreme rather than pd-standard. Monitor burst credit balance on gp2 volumes: exhaustion forces a drop to baseline performance that causes sustained fsync latency spikes.

If throttling is chronic and budget-constrained, separate the WAL onto a dedicated volume with lower throughput but consistent latency characteristics.

Failing SSD

Check SMART data with smartctl -a /dev/<device>. Look for reallocated sectors, pending sectors, or media errors. A degrading SSD often shows progressively increasing fsync latency before it fails completely.

If the device is failing, drain and decommission the node, replace the underlying storage, and let the cluster up-replicate. Do not raise COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT to mask the problem. The node will continue to accept writes it cannot durably persist, risking data loss on crash.

Shared device contention

If WAL and data files share the same physical device, compaction I/O and backup I/O compete with WAL fsync for disk bandwidth. During compaction bursts or backup operations, fsync latency can spike even on a healthy SSD.

Place the WAL on a dedicated fast device to eliminate the most latency-sensitive I/O from the compaction bottleneck. CockroachDB has no --wal-dir flag; use WAL failover (--wal-failover=among-stores for multi-store nodes, or --wal-failover=path=/wal_device for a single store with a side disk) to move WAL writes to a separate device.

WAL failover (v24.3+)

WAL failover writes to a secondary WAL device when the primary device stalls, providing resilience against transient disk stalls.

When WAL failover is enabled, the recommended COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT increases to 40 seconds (default 20s). There is also COCKROACH_LOG_MAX_SYNC_DURATION for log file sink stalls, also recommended at 40s with WAL failover. The engine threshold is configurable as the storage.max_sync_duration cluster setting (default 20s), which takes precedence over the environment variable when set.

One operational gotcha: the storage_wal_fsync_latency metric does not currently distinguish between the primary and secondary WAL directories. If WAL failover is active, you cannot tell from the metric alone which device is slow. You must check OS-level disk metrics for both devices.

SELinux overhead

If SELinux is in Enforcing mode and fsync latency is elevated without hardware cause, check audit logs for denied operations. Consider testing with SELinux in Permissive mode to isolate the effect. If confirmed, work with your security team to tune the policy rather than disabling SELinux entirely.

Prevention

Monitor WAL fsync latency per store. A per-second collection interval catches transient spikes that 15-30 second scrapes miss.
Use dedicated WAL devices on latency-sensitive workloads. Separating WAL from data eliminates compaction contention for the most latency-critical I/O.
Right-size cloud volumes. Provisioned IOPS and throughput caps can silently throttle CockroachDB under moderate load. Provision for sustained write rate, not average.
Do not raise stall thresholds to mask hardware issues. Increasing COCKROACH_ENGINE_MAX_SYNC_DURATION_DEFAULT hides the symptom while the underlying storage degrades toward failure.
Distinguish chronic throttling from acute degradation. Track fsync latency patterns over hours. Cloud volume throttling produces periodic spikes; failing hardware produces progressive increase.
Consider WAL failover on v24.3+. If your deployment experiences periodic disk stalls from cloud volume throttling, WAL failover provides resilience by writing to a secondary device.

How Netdata helps

Per-second WAL fsync latency histograms. Netdata collects storage_wal_fsync_latency at one-second resolution. Individual fsync stalls can cause Raft heartbeat delays even when the average looks fine.
Disk stall detection correlation. Netdata surfaces storage_disk_stalled and storage_disk_slow alongside WAL fsync latency in the same dashboard. A nonzero storage_disk_stalled with rising fsync latency confirms the node is in active stall territory.
OS-level disk I/O alongside database metrics. Netdata’s disk collector provides iostat-equivalent latency, utilization, and queue depth per device at per-second granularity.
Write-path signal chain. Correlating WAL fsync latency with raft_process_logcommit_latency, exec_latency, and sql_service_latency in one view shows where latency enters the write path and how far it propagates to clients.
Admission control queue depth. When fsync latency rises, the store-write admission control queue deepens as CockroachDB throttles writes. Netdata surfaces this alongside the fsync metric.
ML anomaly detection on fsync latency. Netdata’s anomaly engine learns the normal fsync latency baseline per store, which is especially useful for cloud volumes where the baseline varies with provisioned IOPS.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

CockroachDB monitoring with Netdata

Netdata monitors CockroachDB with per-second metrics and automatic dashboards. Watch LSM compaction, Raft liveness, clock skew, hot ranges, and intent buildup so the distributed-systems failure modes in these runbooks surface early.

See CockroachDB monitoring → Start monitoring free