CockroachDB ReadWithinUncertaintyInterval restarts: the near-diagnostic signal of clock skew

When CockroachDB reports readwithinuncertainty as a transaction restart cause, you have a clock synchronization problem. This restart cause does not appear in meaningful quantities for any other reason. Any sustained nonzero rate, even well below the self-termination threshold, indicates that NTP or the underlying clock source is not keeping node clocks aligned.

The error stems from CockroachDB’s Hybrid Logical Clock (HLC) design. Each transaction receives a timestamp from its gateway node’s HLC, which combines physical wall-clock time with a logical counter. When a read on one node encounters a write from a transaction that started on a different node, and the two nodes’ clocks are not aligned, the database cannot determine which transaction began first. CockroachDB conservatively restarts the reading transaction at a higher timestamp. Under SERIALIZABLE isolation, this restart requires client-side retry logic and adds directly to tail latency.

Other restart causes (writetooold, txnpush) point to contention or application-level conflicts. readwithinuncertainty points to infrastructure. If you see it, investigate NTP and clock sources before looking at queries, schema, or workload patterns.

How uncertainty restarts work

Because physical clocks differ between nodes, CockroachDB defines an uncertainty interval for every read: [reader_timestamp, reader_timestamp + max_offset]. The max_offset defaults to 500ms, set via the --max-offset node start flag.

Any write with a timestamp falling inside this interval is ambiguous. The reading transaction cannot tell whether the write happened before or after it began. CockroachDB resolves this conservatively by restarting the reading transaction at a timestamp above the uncertain write. This is the ReadWithinUncertaintyIntervalError.

At moderate clock offsets, the uncertainty interval is wide enough that concurrent transactions on different nodes frequently overlap. The result is a persistent tail-latency tax: a fraction of reads must restart, each adding the cost of a full transaction retry. At higher offsets, the restart rate climbs and latency degrades further.

If a node detects its clock is skewed beyond 80% of --max-offset (400ms at the default 500ms), it self-terminates to prevent data corruption from clock divergence. This creates an availability event. If multiple nodes share the same broken NTP infrastructure and drift simultaneously, the cluster can lose quorum.

flowchart TD
    A["NTP failure or VM migration"] --> B["Clock drift between nodes increases"]
    B --> C["More writes fall within uncertainty window"]
    C --> D["readwithinuncertainty restarts climb"]
    D --> E["SQL read P99 latency increases"]
    B --> F{"Drift exceeds 80% of max-offset?"}
    F -->|"Yes, sustained > 2 min"| G["Node self-terminates"]
    G --> H["Lease redistribution / quorum risk"]

Common causes

Quick checks

These are safe, read-only commands. Run them on the node showing elevated restarts, or on all nodes if the problem is cluster-wide.

# Check restart causes from the Prometheus metrics endpoint
curl -s http://localhost:8080/_status/vars | grep readwithinuncertainty

# Check all restart causes for comparison
curl -s http://localhost:8080/_status/vars | grep txn_restarts

# Check pairwise clock offset between this node and all peers
# Note: Prometheus metric names use hyphens, not underscores
curl -s http://localhost:8080/_status/vars | grep "clock-offset"

# Check chrony synchronization status (recommended NTP client)
chronyc tracking

# Check chrony NTP sources
chronyc sources

# Check node uptime <!-- TODO: verify sys_uptime metric name in current CockroachDB versions -->
curl -s http://localhost:8080/_status/vars | grep uptime

For a cluster-wide view of clock offsets, this admin-only SQL query returns gossip-level node information including remote clock offset:

SELECT * FROM crdb_internal.gossip_nodes;

How to diagnose it

1. Confirm the restart cause is specifically readwithinuncertainty. Total restart count is not enough. The txn_restarts metric exposes sub-metrics by cause. Only readwithinuncertainty indicates clock skew. writetooold indicates contention. txnpush indicates application conflicts. serializable indicates generic serialization pressure.

2. Check the clock offset metric. clock-offset-meannanos shows the observed offset between this node and each remote node. Compare against your --max-offset value. At default settings: below 100ms is acceptable, above 250ms is dangerous, and at or above 400ms the self-termination mechanism is about to fire.

3. Verify NTP health on all nodes, not just the one showing restarts. The offset is measured pairwise, so the drift could be on either side. Run chronyc tracking on every node. A node reporting “Not synchronised” or showing large correction values is the source.

4. Correlate with SQL read latency. readwithinuncertainty restarts add a full retry to the read path. Check whether the P99 spike on sql_service_latency aligns temporally with the restart rate increase. If they track together, clock skew is confirmed as the latency driver.

5. Check for recent VM migrations or cloud maintenance events. Cloud providers may migrate VMs, freezing the clock during the migration window. On resume, the clock is arbitrarily stale. Look for maintenance event timestamps that precede the offset spike.

6. Identify the drift pattern. A single node drifting points to local NTP failure or hardware issues. Multiple nodes drifting in the same direction points to shared NTP infrastructure. Multiple nodes drifting independently suggests a broader time synchronization architecture problem.

Metrics and signals to monitor

Fixes

Fix NTP configuration

The most common root cause is NTP not running, misconfigured, or pointing at an unreachable server. Cockroach Labs recommends chrony as the NTP client. ntpd has been superseded by chrony in most distributions. systemd-timesyncd is not recommended for multi-region topologies.

1. Ensure chrony is installed and running on every node.
2. Verify the NTP source is reachable with chronyc sources.
3. Confirm all nodes point to equivalently reliable NTP sources.
4. Monitor offset convergence with chronyc tracking. Chrony corrects gradually (slew), so the offset may take minutes to normalize. Do not force a step correction (jump) unless you understand the risk of a sudden timestamp discontinuity.

Disable VM live migration

Live VM migration suspends the VM, stopping its clock. On resume, the clock is arbitrarily stale.

• GCE: Set host maintenance policy to TERMINATE. GCE does not guarantee uninterrupted clock during live migrations.
• Azure: Use dedicated hosts or configure maintenance policies to minimize live migrations. Azure does not guarantee uninterrupted clock during live migrations.
• AWS: Use the Amazon Time Sync Service at 169.254.169.123.

Reduce –max-offset (with caution)

Reducing --max-offset tightens the uncertainty interval, which reduces the rate of uncertainty restarts when clocks are well-synchronized. Cockroach Labs states it is generally safe to reduce to 250ms from the default 500ms if clock synchronization is well-managed.

--max-offset cannot be changed online. Changing it requires shutting down all nodes and bringing them back up with the new flag. A rolling restart is insufficient: nodes self-terminate if they encounter a peer with a different --max-offset.

This is a cluster-wide change with downtime. Do not attempt it during an active incident. Fix the underlying clock synchronization first, verify stability, then consider tightening the bound.

Application-level mitigations

These reduce the impact of uncertainty restarts but do not address the root cause:

• Implement client-side retry logic for error code 40001. Under SERIALIZABLE isolation, this is required for correctness regardless of clock skew. Without it, restarts surface as application-visible errors.
• Batch statements within a transaction. Sending all statements in a single batch reduces round trips subject to restart.
• Use historical reads for read-only queries. SELECT ... AS OF SYSTEM TIME with a timestamp slightly in the past eliminates uncertainty restarts for those reads by moving outside the uncertainty window.

Prevention

• Monitor clock offset proactively. Alert on any node exceeding 50% of --max-offset (250ms at default). The fatal self-termination threshold is 80% (400ms at default). Ideal offset is below 10ms.
• Alert on readwithinuncertainty restarts specifically. Any sustained nonzero rate is a signal. Set the threshold to zero, with a cold-start suppression window.
• Break restart causes into separate alerts. Total retry rate is a noisy aggregate. readwithinuncertainty means infrastructure. writetooold means schema. txnpush means application behavior.
• Use chrony on every node. Verify it is running, synced, and pointing to reachable NTP sources on a regular schedule. Do not treat NTP as set-and-forget.
• Disable VM live migration on cloud instances. Use TERMINATE policies where uninterrupted clock is not available.
• Test NTP failure scenarios. Inject clock drift in a staging environment to verify monitoring catches it before the self-termination threshold fires.

How Netdata helps

• Per-second clock offset visibility. Netdata collects clock-offset-meannanos at per-second resolution, catching drift trends that coarser scrape intervals miss.
• Restart cause breakdown. Netdata surfaces txn_restarts sub-metrics by cause, letting you distinguish readwithinuncertainty (clock skew) from writetooold (contention) without manual endpoint queries.
• Latency correlation. Correlating readwithinuncertainty restarts against sql_service_latency P99 confirms the tail-latency impact before users report it.
• Node liveness early warning. Correlating clock offset with node liveness status and ranges_underreplicated gives advance warning when drift approaches the self-termination threshold.

Netdata’s CockroachDB monitoring brings these signals together with per-second metrics.

Related guides

• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum
• CockroachDB replica unavailable: lost quorum and stuck Raft groups
• CockroachDB result is ambiguous: ambiguous commit errors and how to handle them