CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

You see this fatal log line on a CockroachDB node:

clock synchronization error: this node is more than 500ms away from at least half of the known nodes

The process exits immediately. If the node is managed by systemd, Kubernetes, or a process supervisor, it restarts and crashes again. The crash-loop continues until the clock problem is fixed.

CockroachDB uses Hybrid Logical Clocks (HLC) to enforce serializable consistency across distributed transactions. HLC combines physical wall-clock time with a logical counter. Every node must agree on time within a bounded window, controlled by --max-offset (default 500ms). When a node detects its clock has drifted beyond 80% of --max-offset relative to a majority of peers, it calls log.Fatal and exits rather than risk serving stale reads or assigning timestamps that conflict with committed transactions.

The self-termination trigger fires at roughly 400ms with the default 500ms --max-offset. If you have lowered --max-offset, the trigger drops proportionally. A node that crashes from this error will not recover by restarting; if the clock is still wrong, it crashes again immediately. Fix the clock first, then restart.

flowchart TD
    A["NTP failure or VM migration"] --> B["Clock drifts from peers"]
    B --> C{"Offset > 80% of max-offset?"}
    C -->|No| D["Uncertainty window widens"]
    D --> E["readwithinuncertainty restarts climb"]
    E --> F["Tail latency degrades silently"]
    C -->|Yes| G["Node calls log.Fatal"]
    G --> H["Process exits"]
    H --> I{"Clock fixed?"}
    I -->|No| J["Crash-loop on restart"]
    I -->|Yes| K["Node rejoins cluster"]

A single-node cluster or cockroach demo may also emit this message if the machine was suspended (laptop sleep, VM pause). The clock jumps by the sleep duration and triggers the self-termination check.

Common causes

Quick checks

All read-only and safe to run during an incident.

# Chrony synchronization status
chronyc tracking

# NTP sources and reachability
chronyc sources

# Chrony daemon status
systemctl status chronyd

# CockroachDB clock offset (nanoseconds; divide by 1,000,000 for ms)
curl -s http://localhost:8080/_status/vars | grep clock_offset_meannanos

# Transaction restarts caused by clock uncertainty
curl -s http://localhost:8080/_status/vars | grep -i readwithinuncertainty

# Node liveness status
<!-- TODO: verify the exact JSON field path for liveness in /status/nodes. The liveness_status field name and structure vary by CockroachDB version. -->
curl -s http://localhost:8080/_status/nodes | python3 -c "
import json, sys
for n in json.load(sys.stdin).get('nodes', []):
    nid = n['desc']['node_id']
    ls = n.get('liveness_status', 'unknown')
    print(f'Node {nid}: liveness_status={ls}')"

# Check for conflicting time sync services
systemctl status chronyd systemd-timesyncd ntpd 2>/dev/null | grep -E "Active:|Loaded:"

If chronyc tracking shows Leap status : Not synchronized or the system offset is in the hundreds of milliseconds, the node’s clock is not being corrected and the crash will repeat.

How to diagnose

1. Identify affected nodes. Check logs on all nodes for the fatal error. A single crashed node points to a local problem (NTP stopped, hardware clock, VM migration). Multiple crashed or drifting nodes points to shared NTP infrastructure.

2. Check NTP status on each affected node. Run chronyc tracking. Look at System time and Last offset. If chrony reports Not synchronized, the node has no valid time source.

3. Verify NTP source reachability. Run chronyc sources. Each source should show a reachability value of 377 (octal, meaning all recent probes succeeded). A value of 0 means the source is unreachable.

4. Check the clock offset metric. On surviving nodes, scrape clock_offset_meannanos for the affected node. Values are in nanoseconds; divide by 1,000,000 for milliseconds. Anything above 250ms is dangerous. Above 400ms triggers self-termination.

5. Check readwithinuncertainty restarts. This transaction restart cause is nearly diagnostic for clock skew. If the rate was elevated before the crash, drift was already degrading performance before the node terminated. Scrape the txn_restarts histogram on surviving nodes and filter for readwithinuncertainty.

6. Review recent infrastructure events. Check for VM live migrations, network changes blocking NTP traffic, or cloud platform maintenance. A crash immediately after a migration event strongly suggests a migration-induced clock freeze.

7. Check for conflicting time sync services. Running both chrony and systemd-timesyncd (or ntpd) on the same host causes conflicts as they fight over clock adjustments. Only one time sync daemon should be active per node.

Metrics and signals to monitor

Fixes

NTP daemon not running

Start and enable chrony:

sudo systemctl start chronyd
sudo systemctl enable chronyd

# Verify synchronization begins
chronyc tracking

Chrony may take several minutes to slew the clock back into sync. Monitor chronyc tracking until System time converges toward zero. Do not restart CockroachDB until the offset is well below the self-termination threshold.

NTP server unreachable

If chrony is running but cannot reach any NTP source, check firewall rules, DNS resolution, and network routing. Cloud environments may block outbound NTP (UDP 123) or require using the cloud provider’s internal time service. Use cloud-specific NTP sources when available: they are lower latency and more reliable than public pools from within a VPC.

VM live migration

VM live migration (vMotion on VMware, live migration on GCE or Azure) freezes the guest clock during migration. When the VM resumes, its clock is arbitrarily stale. The node detects the jump and self-terminates.

For GCE and Azure VMs, disable live migration. Set the host maintenance behavior to TERMINATE rather than migrate. The VM will be restarted rather than live-migrated during maintenance, which is the correct tradeoff for clock safety.

Leap second smearing mismatch

Leap seconds cause time to jump or smear depending on the NTP source. Google and Amazon time services apply leap smearing, gradually adjusting over 24 hours. The default NTP pool does not smear; it steps the clock. If nodes use different smearing strategies, their clocks will diverge by up to a second during a leap second event, triggering self-termination.

All nodes must use time sources with identical leap smearing behavior. Do not mix Google/Amazon sources with the standard NTP pool.

Changing –max-offset

The --max-offset flag cannot be changed with a rolling restart. Nodes with different --max-offset values will detect each other as exceeding the threshold and self-terminate. Changing it requires a full cluster shutdown and restart with the new value on every node simultaneously.

For multi-region clusters, lowering --max-offset to 250ms can tighten the uncertainty window and reduce read restart overhead, but increases sensitivity: the self-termination threshold drops to approximately 200ms. Only do this if your NTP infrastructure can reliably keep all nodes within that bound.

Single-node development clusters

If you are running a single-node cluster or cockroach demo and see this error, check whether the machine was suspended or slept. A laptop waking from sleep will have a clock that jumped by the sleep duration. Fix the clock and restart the process.

Prevention

• Monitor clock_offset_meannanos proactively. Alert at 250ms (250,000,000 ns). The upstream CockroachDB alerting rules fire a ClockOffsetNearMax alert at 300ms. Do not wait for the self-termination log line.

• Use chrony, not ntpd. Chrony is the recommended NTP client for CockroachDB. ntpd handles network interruptions and VM clock jumps poorly. Ensure only one time sync service is running per node.

• Disable VM live migration on GCE and Azure. Set the host maintenance behavior to terminate.

• Standardize NTP sources across all nodes. All nodes should use the same time sources with the same leap second smearing strategy. Use cloud provider internal time services where available for lower latency.

• Run periodic NTP health checks. Verify that chrony reports synchronized status on every node. A node that silently loses its NTP source will drift until it crashes. The clock offset metric is a physical measurement, not workload-shaped, so it is reliable as a direct alerting signal.

• Gate clock alerts on uptime. A brief clock offset spike is normal after NTP slew correction following a restart. Use a 10-minute uptime gate to suppress false positives during node recovery.

How Netdata helps

• Per-second clock offset monitoring. Netdata collects clock_offset_meannanos at 1-second resolution, catching drift trends before they reach the self-termination threshold. Standard 15-30 second scrape intervals can miss rapid drift events entirely.

• Correlation with readwithinuncertainty restarts. When clock offset rises, Netdata surfaces the corresponding increase in uncertainty-driven transaction restarts, confirming that drift is already impacting application latency before the node dies.

• Node-level liveness tracking. Netdata shows node liveness transitions alongside clock metrics, making it clear whether a node has self-terminated and whether offset was the cause.

• System-level NTP visibility. Netdata collects chrony and system time metrics at the OS level, providing the infrastructure context needed to distinguish a database-side issue from an NTP-side issue.

Netdata’s database monitoring brings these signals together with per-second metrics.

Related guides

• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum
• CockroachDB replica unavailable: lost quorum and stuck Raft groups
• CockroachDB result is ambiguous: ambiguous commit errors and how to handle them
• How CockroachDB actually works in production: a mental model for operators