CockroachDB clock_offset_meannanos high: catching clock drift before self-termination

clock_offset_meannanos measures the mean clock offset between a CockroachDB node and its peers. When it climbs, you are on a path that ends in silent performance degradation from widened read uncertainty windows, or a node self-terminating to preserve data consistency.

The thresholds are unforgiving. CockroachDB uses a default --max-offset of 500ms. A node self-terminates when its mean offset exceeds 80% of that value (400ms) relative to a majority of peers. But a constant 200ms offset, stable and below any threshold, still doubles the uncertainty interval for every read. Transactions silently restart more often, P99 read latency creeps up, and nobody suspects the clock.

What this means

CockroachDB uses Hybrid Logical Clocks (HLC) to order transactions. Each HLC timestamp combines physical wall-clock time with a logical counter. When a node reads data, it uses the clock offset to define an uncertainty window. If a write might have happened within that window, the read retries at a higher timestamp to guarantee serializable correctness.

Larger clock offsets widen the uncertainty window. More reads fall within it. The readwithinuncertainty transaction restart cause is nearly diagnostic of clock skew: it does not occur in meaningful quantities for any other reason.

One limitation of the metric: clock_offset_meannanos is the mean of signed offsets between this node and its peers. Positive and negative offsets cancel out. If one peer is +300ms ahead and another is -300ms behind, the mean reports near zero. The companion metric clock_offset_stddevnanos detects this: a high standard deviation with a low mean means outliers are being averaged away.

At 80% of max-offset (400ms at the default 500ms), CockroachDB’s safety mechanism fires. The node logs a clock synchronization error and self-terminates. It will not rejoin until the clock is corrected. If multiple nodes share the same NTP infrastructure and drift together, quorum loss across ranges follows.

flowchart TD
    A["NTP drift or VM clock stall"] --> B["clock_offset_meannanos rises"]
    B --> C["Uncertainty interval widens"]
    C --> D["readwithinuncertainty restarts climb"]
    D --> E["SQL read P99 increases"]
    B --> F{"Offset > 400ms?"}
    F -->|Below threshold| G["Silent: 200ms doubles uncertainty window"]
    F -->|Above threshold| H["Node self-terminates"]
    H --> I["Multiple nodes drift? Quorum loss"]

Common causes

Quick checks

Run these on the node exhibiting drift or from a host with access to the cluster’s HTTP endpoint.

# Check chrony synchronization status on the affected node
chronyc tracking

# Check clock offset metrics from the local node
curl -s http://localhost:8080/_status/vars | grep clock_offset

# Check readwithinuncertainty restart rate (nearly diagnostic for clock skew)
curl -s http://localhost:8080/_status/vars | grep readwithinuncertainty

# Check NTP sources and their reachability
chronyc sources -v

# Verify node liveness status across the cluster
curl -s http://localhost:8080/_status/nodes | python3 -c "
import json, sys
for n in json.load(sys.stdin)['nodes']:
    print(f'Node {n[\"desc\"][\"node_id\"]}: liveness={n.get(\"liveness\",{}).get(\"liveness\",\"UNKNOWN\")}')"

How to diagnose it

1. Identify which nodes are drifting. Pull clock_offset_meannanos and clock_offset_stddevnanos for every node. The metric is per remote node, so each node reports its offset relative to every peer. Look for the outlier, not just the aggregate. If node A reports high offset to node B, but node B reports normal offset to everyone else, node A is the problem.

2. Check NTP status on the affected node. Run chronyc tracking. If “Leap status” is not “Normal” or “Last offset” is large, the clock is not synchronized. Run chronyc sources -v to verify NTP source reachability.

3. Correlate with readwithinuncertainty restarts. Pull the restart breakdown from the txn_restarts metric family and look for the readwithinuncertainty cause. Any sustained nonzero rate is a clock synchronization signal, even if clock_offset_meannanos has not yet reached the 250ms alerting threshold.

4. Check for recent VM migrations. If nodes run on virtualized infrastructure, check the hypervisor migration log. VM live migration stalls the guest clock during the copy phase. After resumption, the clock jumps and offset metrics spike.

5. Determine if the problem is shared infrastructure. If multiple nodes drift simultaneously, the root cause is likely shared NTP infrastructure or a cloud time service. Check NTP server reachability from all affected nodes, not just one.

Metrics and signals to monitor

Fixes

Fix NTP on the affected node

The immediate priority is restoring clock synchronization. If the NTP daemon has stopped, restart it. If it is running but cannot reach its sources, resolve the network or firewall issue blocking NTP traffic.

Use chrony, not ntpd or systemd-timesyncd. Cockroach Labs recommends chrony. ntpd is not included in RHEL 8+ or installed by default on recent Ubuntu. systemd-timesyncd lacks the slew-based correction precision CockroachDB requires.

After fixing NTP, clock correction is gradual (slew), not instantaneous. Monitor clock_offset_meannanos as it converges. Do not force a step change with chronyc makestep unless you understand the consequences: a sudden clock jump can cause a burst of readwithinuncertainty restarts before stabilizing.

Restart a self-terminated node safely

A node that self-terminated from clock skew will crash-loop if restarted before the clock is corrected. Before restarting:

1. Verify chronyc tracking shows normal synchronization on the host.
2. Confirm clock_offset_meannanos is converging toward baseline (under 100ms).
3. Restart the CockroachDB process.

Cloud-specific clock configuration

Virtualized environments introduce clock stalls that bare metal does not. Common hardening steps:

• AWS: Use the Amazon Time Sync Service (169.254.169.123). It implements leap second smearing. AWS also exposes a PTP hardware clock at /dev/ptp0 on supported instance types.
• GCP: Disable live migration by setting the host maintenance policy to TERMINATE. GCE does not provide an uninterrupted clock to guest VMs during live migration.
• Azure: Disable the Hyper-V time synchronization integration service. Use Google Public NTP or Amazon Time Sync Service instead, which both smear.

For VMs subject to live migration (vMotion), CockroachDB supports the --clock-device flag (Linux only) to bind the clock to a PTP hardware clock device. This prevents stale-clock reads after migration suspension. When using --clock-device, do not enable the server.clock.forward_jump_check_enabled cluster setting, as forward jumps are expected with PTP devices.

Adjusting max-offset (use with caution)

--max-offset is a node-start flag, not a cluster setting. Lowering it tightens the uncertainty window (better read performance, fewer restarts) but increases the risk of self-termination from minor clock drift. Cockroach Labs recommends lowering max-offset to 250ms for multi-region clusters using global tables, where tighter uncertainty bounds improve follower read freshness.

Raising max-offset reduces self-termination risk but widens the uncertainty interval, causing more read restarts and longer stale-read windows. This trades availability for correctness headroom and is rarely the right answer.

Changing --max-offset requires a rolling restart of all nodes. All nodes should converge to the same value.

Prevention

• Alert on clock_offset_meannanos at 250ms (50% of max-offset). By the time offset reaches 400ms, self-termination is imminent.
• Alert on readwithinuncertainty restarts at any sustained nonzero rate. This catches clock skew that has not yet reached the offset threshold but is already degrading performance.
• Monitor clock_offset_stddevnanos alongside the mean. High standard deviation reveals outlier masking that the mean hides.
• Use chrony on all nodes. Configure all nodes to use the same leap-second-smearing NTP sources.
• Ensure all NTP sources implement leap second smearing. Google Public NTP and Amazon Time Sync Service smear. The default NTP pool does not. Leap seconds have historically caused mass CockroachDB node crashes.
• Disable VM features that stall the guest clock. GCE live migration and Azure Hyper-V time synchronization both cause clock stalls. Use PTP clock devices where vMotion is unavoidable.
• Gate clock offset alerts on node uptime greater than 10 minutes. Brief post-restart clock corrections cause transient spikes that should not page.

How Netdata helps

Netdata collects CockroachDB metrics at per-second resolution, which matters here: a node can drift tens of milliseconds in a single 15-30 second Prometheus scrape gap. Key capabilities for this failure mode:

• Per-second clock_offset_meannanos and clock_offset_stddevnanos collection catches drift faster than typical scrape intervals.
• ML anomaly detection on clock offset trends identifies gradual drift before it crosses static thresholds. A node creeping from 5ms to 50ms over an hour is anomalous even if neither value triggers a threshold.
• Correlation between clock_offset_meannanos and readwithinuncertainty restarts in a single view confirms clock skew as the root cause of latency degradation without manual cross-referencing.
• Node liveness changes displayed alongside clock metrics show the full failure path from drift to self-termination.

Related guides

• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum
• CockroachDB replica unavailable: lost quorum and stuck Raft groups
• CockroachDB result is ambiguous: ambiguous commit errors and how to handle them