$ guides / cockroachdb / cockroachdb-clock-synchronization-error-500ms ▌

Operations Guides

CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

You see this fatal log line on a CockroachDB node:

clock synchronization error: this node is more than 500ms away from at least half of the known nodes

The process exits immediately. If the node is managed by systemd, Kubernetes, or a process supervisor, it restarts and crashes again. The crash-loop continues until the clock problem is fixed.

CockroachDB uses Hybrid Logical Clocks (HLC) to enforce serializable consistency across distributed transactions. HLC combines physical wall-clock time with a logical counter. Every node must agree on time within a bounded window, controlled by --max-offset (default 500ms). When a node detects its clock has drifted beyond 80% of --max-offset relative to a majority of peers, it calls log.Fatal and exits rather than risk serving stale reads or assigning timestamps that conflict with committed transactions.

The self-termination trigger fires at roughly 400ms with the default 500ms --max-offset. If you have lowered --max-offset, the trigger drops proportionally. A node that crashes from this error will not recover by restarting; if the clock is still wrong, it crashes again immediately. Fix the clock first, then restart.

flowchart TD
    A["NTP failure or VM migration"] --> B["Clock drifts from peers"]
    B --> C{"Offset > 80% of max-offset?"}
    C -->|No| D["Uncertainty window widens"]
    D --> E["readwithinuncertainty restarts climb"]
    E --> F["Tail latency degrades silently"]
    C -->|Yes| G["Node calls log.Fatal"]
    G --> H["Process exits"]
    H --> I{"Clock fixed?"}
    I -->|No| J["Crash-loop on restart"]
    I -->|Yes| K["Node rejoins cluster"]

A single-node cluster or cockroach demo may also emit this message if the machine was suspended (laptop sleep, VM pause). A single-node cluster compares against itself, so the check still fires; the clock jumps by the sleep duration and triggers the self-termination check.

Common causes

Cause	What it looks like	First thing to check
NTP daemon stopped or not installed	Node crash-loops; `chronyc tracking` reports no sync source	`systemctl status chronyd`
VM live migration	Node crashes immediately after a hypervisor migration event; clock jumps by seconds or minutes	Check hypervisor event logs
NTP server unreachable	Multiple nodes drift simultaneously if they share an NTP source; offset grows gradually	`chronyc sources` to see reachability
Hardware clock failure	Single node drifts while others stay synchronized; drift is rapid and non-linear	`chronyc tracking` shows large offset
Leap second smearing mismatch	Nodes crash around a leap second event; some NTP sources smear, others step	Check NTP source compatibility
Multiple time sync services running	chrony and systemd-timesyncd both active, fighting over the clock	`systemctl status chronyd systemd-timesyncd ntpd`

Quick checks

All read-only and safe to run during an incident.

# Chrony synchronization status
chronyc tracking

# NTP sources and reachability
chronyc sources

# Chrony daemon status
systemctl status chronyd

# CockroachDB clock offset (nanoseconds; divide by 1,000,000 for ms)
curl -s http://localhost:8080/_status/vars | grep clock_offset_meannanos

# Transaction restarts caused by clock uncertainty
curl -s http://localhost:8080/_status/vars | grep -i readwithinuncertainty

# Node liveness status (JSON field paths vary by version; print any liveness-related fields)
curl -s http://localhost:8080/_status/nodes | python3 -c "
import json, sys
for n in json.load(sys.stdin).get('nodes', []):
    nid = n.get('desc', {}).get('node_id', '?')
    liveness = {k: v for k, v in n.items() if 'live' in k.lower() or 'status' in k.lower()}
    print(f'Node {nid}: {liveness}')"

# Check for conflicting time sync services
systemctl status chronyd systemd-timesyncd ntpd 2>/dev/null | grep -E "Active:|Loaded:"

If chronyc tracking shows Leap status : Not synchronized or the system offset is in the hundreds of milliseconds, the node’s clock is not being corrected and the crash will repeat.

How to diagnose

Identify affected nodes. Check logs on all nodes for the fatal error. A single crashed node points to a local problem (NTP stopped, hardware clock, VM migration). Multiple crashed or drifting nodes points to shared NTP infrastructure.
Check NTP status on each affected node. Run chronyc tracking. Look at System time and Last offset. If chrony reports Not synchronized, the node has no valid time source.
Verify NTP source reachability. Run chronyc sources. Each source should show a reachability value of 377 (octal, meaning all recent probes succeeded). A value of 0 means the source is unreachable.
Check the clock offset metric. On surviving nodes, scrape clock_offset_meannanos for the affected node. Values are in nanoseconds; divide by 1,000,000 for milliseconds. Anything above 250ms is dangerous. Above 400ms triggers self-termination.
Check readwithinuncertainty restarts. This transaction restart cause is nearly diagnostic for clock skew. If the rate was elevated before the crash, drift was already degrading performance before the node terminated. Scrape the txn_restarts histogram on surviving nodes and filter for readwithinuncertainty.
Review recent infrastructure events. Check for VM live migrations, network changes blocking NTP traffic, or cloud platform maintenance. A crash immediately after a migration event strongly suggests a migration-induced clock freeze.
Check for conflicting time sync services. Running both chrony and systemd-timesyncd (or ntpd) on the same host causes conflicts as they fight over clock adjustments. Only one time sync daemon should be active per node.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`clock_offset_meannanos`	Direct measure of clock drift between nodes; the metric that drives self-termination	Any node above 250ms (250,000,000 ns)
`txn_restarts` (readwithinuncertainty cause)	Transactions retrying due to clock uncertainty interval; nearly diagnostic for clock skew	Any sustained nonzero rate
`sql_service_latency` P99	Uncertainty restarts inflate tail read latency	P99 rising without SQL or storage explanation
Node liveness status	Self-terminated nodes show as not-live or dead	Unexpected node death
`ranges_underreplicated`	Dead nodes leave ranges under-replicated	Count rising after node crash

Fixes

NTP daemon not running

Start and enable chrony:

sudo systemctl start chronyd
sudo systemctl enable chronyd

# Verify synchronization begins
chronyc tracking

Chrony may take several minutes to slew the clock back into sync. Monitor chronyc tracking until System time converges toward zero. Do not restart CockroachDB until the offset is well below the self-termination threshold.

NTP server unreachable

If chrony is running but cannot reach any NTP source, check firewall rules, DNS resolution, and network routing. Cloud environments may block outbound NTP (UDP 123) or require using the cloud provider’s internal time service. Use cloud-specific NTP sources when available: they are lower latency and more reliable than public pools from within a VPC.

VM live migration

VM live migration (vMotion on VMware, live migration on GCE or Azure) freezes the guest clock during migration. When the VM resumes, its clock is arbitrarily stale. The node detects the jump and self-terminates.

For VMware vMotion, use the --clock-device flag to point CockroachDB at a PTP hardware clock device, which continues running during VM suspension and keeps the node’s clock accurate across the migration. For GCE and Azure VMs, disable live migration. Set the host maintenance behavior to TERMINATE rather than migrate. The VM will be restarted rather than live-migrated during maintenance, which is the correct tradeoff for clock safety.

Leap second smearing mismatch

Leap seconds cause time to jump or smear depending on the NTP source. Google and Amazon time services apply leap smearing, gradually adjusting over 24 hours. The default NTP pool does not smear; it steps the clock. If nodes use different smearing strategies, their clocks will diverge by up to a second during a leap second event, triggering self-termination.

All nodes must use time sources with identical leap smearing behavior. Do not mix Google/Amazon sources with the standard NTP pool.

Changing –max-offset

The --max-offset flag cannot be changed with a rolling restart. Nodes with different --max-offset values will detect each other as exceeding the threshold and self-terminate. Changing it requires a full cluster shutdown and restart with the new value on every node simultaneously.

Cockroach Labs recommends lowering --max-offset to 250ms for multi-region clusters. This tightens the uncertainty window and reduces read restart overhead, but increases sensitivity: the self-termination threshold drops to approximately 200ms. Only do this if your NTP infrastructure can reliably keep all nodes within that bound.

Single-node development clusters

If you are running a single-node cluster or cockroach demo and see this error, check whether the machine was suspended or slept. A laptop waking from sleep will have a clock that jumped by the sleep duration. Fix the clock and restart the process.

Prevention

Monitor clock_offset_meannanos proactively. Alert at 250ms (250,000,000 ns). The upstream CockroachDB alerting rules fire a ClockOffsetNearMax alert at 300ms. Do not wait for the self-termination log line.
Use chrony, not ntpd. Chrony is the recommended NTP client for CockroachDB. ntpd handles network interruptions and VM clock jumps poorly. Ensure only one time sync service is running per node.
Disable VM live migration on GCE and Azure. Set the host maintenance behavior to terminate.
Standardize NTP sources across all nodes. All nodes should use the same time sources with the same leap second smearing strategy. Use cloud provider internal time services where available for lower latency.
Run periodic NTP health checks. Verify that chrony reports synchronized status on every node. A node that silently loses its NTP source will drift until it crashes. The clock offset metric is a physical measurement, not workload-shaped, so it is reliable as a direct alerting signal.
Gate clock alerts on uptime. A brief clock offset spike is normal after NTP slew correction following a restart. Use a 10-minute uptime gate to suppress false positives during node recovery.

How Netdata helps

Per-second clock offset monitoring. Netdata collects clock_offset_meannanos at 1-second resolution, catching drift trends before they reach the self-termination threshold. Standard 15-30 second scrape intervals can miss rapid drift events entirely.
Correlation with readwithinuncertainty restarts. When clock offset rises, Netdata surfaces the corresponding increase in uncertainty-driven transaction restarts, confirming that drift is already impacting application latency before the node dies.
Node-level liveness tracking. Netdata shows node liveness transitions alongside clock metrics, making it clear whether a node has self-terminated and whether offset was the cause.
System-level NTP visibility. Netdata collects chrony and system time metrics at the OS level, providing the infrastructure context needed to distinguish a database-side issue from an NTP-side issue.

Netdata’s database monitoring brings these signals together with per-second metrics.

The Netdata solution

CockroachDB monitoring with Netdata

Netdata monitors CockroachDB with per-second metrics and automatic dashboards. Watch LSM compaction, Raft liveness, clock skew, hot ranges, and intent buildup so the distributed-systems failure modes in these runbooks surface early.

See CockroachDB monitoring → Start monitoring free