CockroachDB context deadline exceeded: timeouts, slow ranges, and overload

“context deadline exceeded” tells you something is slow. It does not tell you what. The same error appears whether a leaseholder is overloaded, a network path is congested, L0 sublevels are climbing past write-stall territory, or the application set a 2-second statement timeout on a query that normally takes 50 milliseconds.

Treat the error as a symptom, not a diagnosis. The diagnostic question is: which layer is slow?

What this means

Every internal CockroachDB RPC carries a context with a deadline. SQL statements sent from a gateway node to a leaseholder, KV batches forwarded to range replicas, and Raft heartbeats all have timeouts. When a deadline expires, the caller receives a “context deadline exceeded” error that propagates up through the SQL layer to the client.

A per-replica circuit breaker adds a second mechanism. If a replica proposal (such as a lease request) does not succeed within the slow replication threshold, the circuit breaker trips and returns a ReplicaUnavailableError, which can surface to clients as context deadline exceeded.

The error is the standard Go context.DeadlineExceeded. It appears in three distinct situations:

  1. An application-level timeout (statement_timeout, connection pool timeout, client-side deadline) fires before the query finishes.
  2. An internal CockroachDB RPC deadline fires because a downstream layer (storage, Raft, network) is slow.
  3. A replica circuit breaker trips because a proposal stalled.

Distinguishing these three is the entire diagnostic exercise.

Common causes

CauseWhat it looks likeFirst thing to check
LSM compaction debtKV exec latency spiking, storage_l0_sublevels climbing past 10, write stalls in logsgrep storage_l0_sublevels in metrics
CPU saturation or admission throttlingAdmission control queue depth non-zero, round_trip_latency elevated on one node, GC pauses risinggrep admission in metrics, check CPU per node
Network issue between nodesround_trip_latency elevated for specific node pairs, asymmetric latencygrep round_trip_latency in metrics
Node liveness failure or Raft stallranges_unavailable nonzero, lease transfers spiking, liveness heartbeat failures in logsgrep ranges_unavailable and check node liveness
Clock skewreadwithinuncertainty restarts climbing, clock_offset_meannanos elevatedgrep clock_offset and chronyc tracking
Application timeout too aggressiveDatabase metrics look fine, error correlates with specific queries or traffic burstsCompare sql_service_latency P99 against client timeout

Quick checks

Run these read-only commands on any node. They pull from the Prometheus endpoint at /_status/vars.

# Check if any ranges are unavailable (active availability loss)
curl -s http://localhost:8080/_status/vars | grep ranges_unavailable

# Check KV execution latency (storage layer health)
curl -s http://localhost:8080/_status/vars | grep exec_latency

# Check inter-node RPC latency (network health)
curl -s http://localhost:8080/_status/vars | grep round_trip_latency

# Check L0 sublevel count (storage engine health)
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

# Check admission control queue state (overload indicator)
curl -s http://localhost:8080/_status/vars | grep admission

# Check lease transfer rate (liveness churn)
curl -s http://localhost:8080/_status/vars | grep leases_transfers

# Check clock offset between nodes
curl -s http://localhost:8080/_status/vars | grep clock_offset

# Check WAL fsync latency (disk write path)
curl -s http://localhost:8080/_status/vars | grep wal_fsync_latency

# Check write stalls (storage engine refusing writes)
curl -s http://localhost:8080/_status/vars | grep storage_write_stalls

# Synthetic SQL probe (full client path; requires auth in secure clusters)
time cockroach sql -e "SELECT 1" --host=localhost:26257

How to diagnose it

Localize the slow layer. Work from the most severe signals (range unavailability, node liveness) down to per-query timing.

flowchart TD
    A["context deadline exceeded"] --> B{"ranges_unavailable > 0?"}
    B -->|Yes| C["Quorum or leaseholder failure"]
    B -->|No| D{"round_trip_latency elevated?"}
    D -->|Yes| E["Network or CPU scheduler delay"]
    D -->|No| F{"storage_l0_sublevels > 10?"}
    F -->|Yes| G["Compaction debt or storage overload"]
    F -->|No| H{"admission queue wait > 0?"}
    H -->|Yes| I["Admission control throttling"]
    H -->|No| J{"clock_offset > 250ms?"}
    J -->|Yes| K["Clock skew causing restarts"]
    J -->|No| L["Compare query latency vs client timeout"]

  1. Check range availability first. If ranges_unavailable is nonzero, some ranges cannot serve reads or writes. The timeout errors are a consequence, not the root cause. Investigate why ranges lost quorum or have no leaseholder. Check node liveness and whether a node is down or partitioned.

  2. Check node liveness. A slow node that cannot process liveness heartbeats will have its leases redistributed. This creates a cascade: lease transfers cause brief unavailability, which surfaces as timeouts. Look at lease transfer rate for spikes, and check whether any node appears impaired in /_status/nodes.

  3. Check inter-node RPC latency. Elevated round_trip_latency for specific node pairs means either network congestion or a CPU-saturated node. Kernel scheduling delay inflates RPC latency even with a healthy network. Asymmetric latency (A to B fast, B to A slow) can cause Raft leadership oscillation, which produces intermittent timeouts.

  4. Check KV execution latency. If KV exec latency is elevated but round_trip_latency is normal, the storage layer is the bottleneck. Check raft.process.logcommit.latency to distinguish WAL write delays from general KV processing delays. Rising Raft log commit latency means disk I/O is the constraint.

  5. Check L0 sublevels. If storage_l0_sublevels is above 10, read amplification is degrading performance. Above 20, write stalls are imminent or active. Check storage_write_stalls for confirmation. Admission control’s store-write queue should show queuing when L0 is elevated.

  6. Check admission control. Non-zero queue depth with growing wait times means the system is at capacity. The store-write queue is tied to LSM health. The kv and sql-kv-response queues indicate foreground traffic is being throttled. The elastic-cpu queue queuing is often acceptable as it correctly prioritizes foreground work.

  7. Check clock offset. If clock_offset_meannanos shows any node above 250ms (50% of the default 500ms max-offset), clock skew is widening uncertainty intervals and causing readwithinuncertainty restarts. These restarts add latency that can push queries past client timeouts.

  8. Compare query latency against the client timeout. If all cluster-side metrics look healthy, the issue is likely an application timeout that is too aggressive for the query’s normal execution time. Check sql_service_latency and txn_durations P99 against the configured application timeout or statement_timeout.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ranges_unavailableNonzero means active data unavailabilityAny nonzero sustained value
KV exec latencyKV batch execution time, isolates storage from SQLP99 rising without SQL plan changes
round_trip_latencyInter-node RPC health, catches network and CPU issuesAny pair above 5x baseline
storage_l0_sublevelsMost predictive storage degradation signalAbove 10 sustained, above 20 critical
Admission queue depth and waitWhether internal flow control is throttlingSustained queue depth with growing wait times
Raft log commit latencyWAL fsync path, isolates disk I/O from other storageAbove 50ms on SSDs
storage_wal_fsync_latencyDisk write path healthP99 above 50ms on local SSDs
Lease transfer rateLiveness churn, each transfer adds latencyAbove 10x baseline without operational cause
clock_offset_meannanosClock skew drives uncertainty restarts and self-terminationAbove 250ms on any node
sql_service_latencyEnd-to-end query time as seen by clientsP99 exceeding application timeout
txn_restarts by causeContention or clock issues adding latencyreadwithinuncertainty any nonzero rate

Fixes

Storage compaction debt

If L0 sublevels are climbing, compaction cannot keep up with writes. Reduce write pressure immediately: pause bulk operations (IMPORT, RESTORE, heavy batch inserts), reduce client connection count, or tighten admission control settings. Check whether a specific store is affected versus cluster-wide. If L0 is elevated but decreasing, compaction is catching up.

For deeper compaction issues, see the related guides on L0 sublevels and compaction death spirals.

CPU saturation and admission throttling

If admission control queues are deep and CPU is above 70-80%, the node is at capacity. Admission control is protecting the system by adding latency, but that latency is what pushes requests past their deadlines. Short-term: reduce workload, add nodes, or rebalance hot ranges. Check for hot ranges causing CPU asymmetry where one leaseholder node is saturated while others idle.

Go GC pressure above 15% of CPU time is a secondary concern. Sustained GC pauses approaching the liveness heartbeat interval risk cascading into liveness failure.

Network issues

If round_trip_latency is elevated for specific node pairs, check for NIC errors, packet loss, or bandwidth saturation. CockroachDB multiplexes inter-node traffic over a limited number of gRPC connections per node pair, so Raft heartbeats compete with bulk data transfers. In multi-region deployments, cross-region latency is a fixed cost on every cross-region write.

A TCP handshake that lingers rather than failing immediately with “connection refused” will also produce context deadline exceeded instead of a clean connection error. Check firewall rules and host reachability between nodes.

Node liveness and Raft failures

If a node is flapping liveness, it creates oscillating availability. Each liveness loss triggers lease redistribution, and each recovery triggers lease transfers back. The resulting churn produces intermittent timeouts. The underlying cause is usually resource starvation: GC pauses, disk stalls, or CPU saturation preventing heartbeat processing.

Check Go GC pause durations (sys_gc_pause_ns). Sustained pauses exceeding several hundred milliseconds risk missing liveness heartbeats. WAL fsync latency spikes or disk stall detection can also prevent heartbeat renewal.

Clock skew

Fix NTP on all affected nodes. Check chronyc tracking or ntpstat. Clock correction may be gradual (slew) rather than immediate. If a node has self-terminated due to clock offset exceeding the max-offset threshold, verify the clock is corrected before restarting, or it will crash-loop.

The readwithinuncertainty restart cause in txn_restarts is nearly diagnostic of clock skew. It does not occur in meaningful quantities for other reasons.

Application timeout too short

If all cluster metrics are healthy, the client timeout is too short for the query’s execution time. Either increase the timeout, optimize the query (indexes, plan quality), or reduce the data the query scans. Check whether statistics are stale after a data load, which can cause the optimizer to choose poor plans that are much slower than the baseline.

Prevention

  • Monitor L0 sublevels proactively. This metric gives 10 to 30 minutes of warning before write stalls. It is the single most common gap in CockroachDB monitoring.
  • Set realistic client timeouts. Compare sql_service_latency and txn_durations P99 against application-side deadlines. A timeout shorter than the database’s normal P99 guarantees intermittent failures under load.
  • Use /health?ready=1 for load balancer checks. Plain TCP health checks route traffic to nodes that are draining, write-stalled, or GC-thrashing.
  • Monitor clock offset continuously. A node self-terminating at 400ms offset is avoidable with proactive monitoring.
  • Track admission control queue depth as a capacity signal. Regular queuing means zero burst headroom.
  • Instrument client-side retries. CockroachDB’s internal retry counter does not capture application-side reconnect-and-retry loops. These silent retries amplify any database issue.

How Netdata helps

Correlating the right signals turns a generic “context deadline exceeded” into a localized diagnosis. Netdata surfaces these per-second signals:

  • KV exec latency and round-trip latency side by side, to distinguish storage-layer slowness from network-layer delay.
  • L0 sublevels with admission control queue depth, so compaction debt and the resulting throttling appear as a single correlated event.
  • Node liveness status with lease transfer rate, catching the liveness churn pattern that produces intermittent timeouts.
  • Clock offset with readwithinuncertainty restart rate, making clock skew visible before self-termination.
  • WAL fsync latency alongside write stall counters, isolating the disk write path from other storage signals.
  • SQL service latency P99 against error rate, showing whether timeout errors correlate with latency spikes or appear independently.

For more, see CockroachDB monitoring with Netdata.