CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Multiple CockroachDB nodes crashed overnight. The logs show “clock synchronization error: this node is more than 500ms away from at least half of the known nodes.” You restart them, and they crash again. Some ranges are now unavailable. The cluster is losing quorum.

A shared NTP failure caused multiple nodes to drift past CockroachDB’s self-termination threshold in quick succession. Single-node clock skew is bad but recoverable. Multi-node skew from a shared NTP source can take down quorum faster than the cluster can heal.

The cascade has three phases:

1. NTP drift widens the uncertainty interval, increasing read restarts and tail latency.
2. One or more nodes cross 80% of max-offset and self-terminate.
3. If enough nodes die before NTP is fixed, ranges lose quorum and become unavailable.

What this means

CockroachDB uses Hybrid Logical Clocks (HLC) to order transactions. HLC combines wall-clock time with a logical counter, but depends on nodes having reasonably synchronized physical clocks. The --max-offset flag (default 500ms) sets the maximum allowed clock difference between any two nodes.

When a node detects its clock is more than 80% of max-offset away from at least half of the known nodes, it self-terminates immediately. With the default max-offset of 500ms, this triggers at 400ms. The fatal log message reports the configured max-offset value:

clock synchronization error: this node is more than 500ms away from at least half of the known nodes

CockroachDB would rather kill a node than risk serving inconsistent data. The self-termination is non-negotiable: the node will not restart successfully until the clock is corrected.

The cascade happens when multiple nodes share the same NTP source and that source drifts. Nodes rarely drift at identical rates. Different hardware clocks, different VM hypervisors, and asymmetric NTP reachability cause uneven drift. The first node to cross the 80% threshold self-terminates. Others follow in quick succession. If enough die before the cluster can rebalance, ranges lose quorum.

clock_offset_meannanos measures inter-node offset, not absolute time accuracy. If all nodes share a drifting NTP source and drift at the same rate, the inter-node offset stays low while absolute time drifts. The cascade triggers when drift rates diverge enough that inter-node offsets exceed the threshold. The metric may look deceptively healthy in the early stages of a shared-cause drift event, then spike rapidly as drift rates diverge.

flowchart TD
    A["Shared NTP source fails or drifts"] --> B["Nodes drift at uneven rates"]
    B --> C["Uncertainty widens, restarts climb"]
    B --> D["First node exceeds 80% max-offset"]
    D --> E["Node self-terminates"]
    E --> F["More nodes cross threshold"]
    F --> G["Additional self-terminations"]
    G --> H{"Quorum lost?"}
    H -- Yes --> I["ranges_unavailable spikes"]
    H -- No --> J["Cluster heals leases and replicas"]

Common causes

Quick checks

Safe, read-only commands. Run on any surviving node.

# Check current clock offset from Prometheus metrics
curl -s http://localhost:8080/_status/vars | grep clock_offset

# Check range availability
curl -s http://localhost:8080/_status/vars | grep ranges_unavailable

# Check under-replication
curl -s http://localhost:8080/_status/vars | grep ranges_underreplicated

# Check transaction restart causes (look for readwithinuncertainty)
curl -s http://localhost:8080/_status/vars | grep txn_restarts

# Check node health readiness (returns 503 when draining or quorum lost)
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health?ready=1

Run on each affected node (requires shell access):

# Check chrony status (recommended NTP client)
chronyc tracking

# List NTP sources and their state
chronyc sources

# Check ntpd status (if using ntpd instead of chrony)
ntpstat

How to diagnose it

1. Confirm the failure is clock-related. Grep logs for the fatal message:

   grep "clock synchronization error" /path/to/cockroach-data/logs/cockroach.log
   

   If multiple nodes show this within minutes of each other, you have a cascade. The log message includes the configured max-offset value, so if you see a number other than 500ms, the cluster uses a non-default setting.

2. Check which nodes self-terminated. List node status from a surviving node:

   curl -s http://localhost:8080/_status/nodes | python3 -c "
   import json, sys
   for n in json.load(sys.stdin)['nodes']:
       print(f'Node {n[\"desc\"][\"node_id\"]}: liveness={n.get(\"liveness\",{}).get(\"liveness\",\"UNKNOWN\")}')"
   

   Nodes that self-terminated will show as dead or not-live.

3. Check NTP on all surviving nodes. The NTP problem may still be active and affecting remaining nodes:

   for node in node1 node2 node3; do
       echo "=== $node ==="
       ssh $node "chronyc tracking"
   done
   

   Look for: offset values, reachability of NTP sources, whether the clock is being corrected (slew) or is static.

4. Assess quorum damage. Check if ranges are unavailable:

   curl -s http://localhost:8080/_status/vars | grep ranges_unavailable
   

   Nonzero means some ranges cannot serve reads or writes. If system ranges (meta, liveness) are affected, impact is cluster-wide.

5. Identify the NTP root cause. Common patterns:

   • All nodes point to the same internal NTP server that is down
   • Firewall rule change blocked NTP traffic (UDP 123)
   • VM live migration left clocks stale
   • Cloud provider time sync service regressed
   • Leap second caused a jump that non-smearing NTP sources handled poorly

6. Check for shared-cause drift. Even if clock_offset_meannanos looks acceptable, all nodes may be drifting together at similar rates. Compare node time against a reliable external reference, or against a cluster in a different failure domain. The inter-node metric will not reveal synchronized absolute drift until rates diverge.

Metrics and signals to monitor

Fixes

Fix NTP first

Before restarting any self-terminated node, fix the NTP source on all nodes. A node with an uncorrected clock will crash-loop on restart.

If using chrony (recommended):

# Verify the NTP source is reachable
chronyc sources

# Check current offset and correction status
chronyc tracking

# If the daemon stopped, restart it
sudo systemctl restart chronyd

Clock correction may be gradual (slew mode). Chrony adjusts the clock slowly to avoid backward jumps. Monitor offset convergence with chronyc tracking over several minutes.

Do not force a step change with chronyc makestep on a running CockroachDB node. A backward clock jump can cause data consistency issues. Let chrony slew the correction instead. Only consider makestep if the node’s CockroachDB process is stopped and will not restart until the clock is corrected.

Verify before restarting self-terminated nodes

After fixing NTP, verify each node’s clock is within acceptable range before restarting the CockroachDB process:

# Confirm offset is small and stable
chronyc tracking

Look for the “System time” line showing a small offset relative to the NTP source. If the offset is still large, wait for chrony to slew it down. Only then restart the CockroachDB process.

Recover from quorum loss

If enough nodes self-terminated that ranges lost quorum, the cluster needs those nodes back to restore quorum.

1. Fix NTP on all dead nodes first.
2. Verify clocks are corrected on each.
3. Restart dead nodes in sequence, verifying each rejoins cleanly before starting the next.
4. Monitor ranges_unavailable as it converges to zero.
5. Monitor ranges_underreplicated as the cluster up-replicates.

If the liveness range itself lost quorum, the entire cluster may be frozen. All nodes must be restarted after NTP is fixed.

Consider max-offset changes with version caveats

Tightening --max-offset reduces the uncertainty window (fewer read restarts, better tail latency) but increases self-termination risk if NTP is imperfect.

Before v22.2, changing --max-offset required a full cluster stop-the-world restart. A rolling restart with different values caused nodes to self-terminate on detecting mismatched max-offset values. In v22.2 and later, nodes can run with different --max-offset values in the same cluster, enabling rolling restarts. The effective cluster max-offset is the minimum across all nodes.

Cockroach Labs recommends 250ms max-offset for new multi-region clusters using SQL abstractions.

Prevention

• Diversify NTP sources. Do not point all nodes at a single internal NTP server with no fallback. Use multiple sources including external references. All nodes in the same region must have identical NTP configuration, but configure redundancy within that config.
• Use leap-second-smearing NTP sources. Google Public NTP (time.google.com) and Amazon Time Sync Service (169.254.169.123) implement leap second smearing. Azure native time sync does not smear. The EC2 PTP hardware clock does not smear; use the Amazon Time Sync Service endpoint instead.
• Monitor clock_offset_meannanos proactively. The upstream ClockOffsetNearMax Prometheus alert fires at 300ms sustained for 5 minutes. Treat 400ms (80% of the 500ms default) as imminent self-termination risk.
• Watch readwithinuncertainty restarts. Any sustained nonzero rate is a clock signal. It does not appear in quantity for any other reason.
• Set GCP VMs to terminate on maintenance. GCP VMs default to live migrate, which suspends the VM and stalls its clock. Set the host maintenance policy to TERMINATE so the VM restarts with a fresh clock instead.
• Use cloud-specific time services. Each cloud provider offers a time sync service optimized for their hypervisor. Use it as the primary NTP source, supplemented by external references.
• Enable forward jump detection. The server.clock.forward_jump_check_enabled cluster setting logs a warning when the CockroachDB clock jumps forward, indicating it was running stale (for example after vMotion).

How Netdata helps

• Per-second clock_offset_meannanos collection catches drift trends minutes before the 80% threshold fires, giving operators time to fix NTP before nodes self-terminate.
• readwithinuncertainty restart rate provides a corroborating signal that is nearly diagnostic of clock skew, visible alongside offset metrics in the same dashboard.
• Node liveness tracking with per-second resolution shows exactly when each node self-terminated and whether the deaths cluster in time (cascade) or are independent events.
• ranges_unavailable and ranges_underreplicated appear alongside clock metrics, so you can assess quorum damage without switching tools during an incident.

Netdata’s CockroachDB monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum