Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn

The MON_CLOCK_SKEW health check fires when the leader monitor detects clock drift beyond mon_clock_drift_allowed (default 0.05 seconds, 50 milliseconds) on any monitor in the quorum. It raises HEALTH_WARN, not HEALTH_ERR, and on its own it does not stop client I/O. But it is the precursor to a class of failure that does: repeated Paxos elections, slow map distribution, and, if the skew grows, monitor quorum loss.

A single transient warning is usually NTP convergence after boot, after a hypervisor live migration, or under brief network pressure. Sustained or recurring skew is almost always an NTP/chrony problem on the MON host: a stopped daemon, an unreachable source, an undersized polling interval, or a virtualized monitor on an overloaded hypervisor. Paxos uses wall-clock time for lease management and internal ordering, so even sub-second drift degrades election stability.

What this means

Monitors maintain cluster maps (OSD map, MON map, PG map, MDS map, CRUSH map) via Paxos consensus. A majority quorum must agree on every map update. The leader periodically checks the clocks of its peers. Under normal conditions the check runs on mon_timecheck_interval (default 300 seconds). Once skew is detected, it tightens to mon_timecheck_skew_interval (default 30 seconds).

When the leader sees any peer’s clock beyond mon_clock_drift_allowed, it raises MON_CLOCK_SKEW. The health detail output names the offender and shows the magnitude and network latency to it:

mon.<id> addr <ip>:6789/0 clock skew <value>s > max 0.05s (latency <value>s)

The 50ms threshold must stay well below the monitor lease interval. Upstream Ceph documentation is explicit: do not raise mon_clock_drift_allowed without testing, and even then prefer fixing the underlying time source. The documentation states plainly that you should run NTP on bare metal, because VM-virtualized clocks are not suitable for steady timekeeping in monitor workloads.

The downstream symptom is election churn. Paxos leadership and lease renewals are time-bound. When clocks disagree, leases appear to expire early or late from the perspective of different monitors. The result is monitors calling for elections that should not be needed, freezing map updates for the duration of each round.

flowchart TD
    A[NTP/chrony stopped or source unreachable] --> B[MON clock drifts]
    C[Hypervisor overloaded or live migration] --> B
    B --> D[Leader detects skew over 50ms]
    D --> E[MON_CLOCK_SKEW fires, HEALTH_WARN]
    E --> F[Paxos leases misfire on skewed MON]
    F --> G[Election epoch increments]
    G --> H[Map distribution stalls, CLI commands slow]
    H --> I[Possible quorum loss if skew grows]

Common causes

Quick checks

These are read-only and safe to run on any cluster with read credentials.

# Cluster-level clock-skew status from the monitor leader
ceph time-sync-status

# Cluster health detail, filtered to clock-related lines
ceph health detail | grep -i clock

# Election epoch and quorum leader
ceph quorum_status -f json | jq '{epoch: .election_epoch, leader: .quorum_leader_name, quorum: .quorum_names}'

# Monitor stats
ceph mon stat

# Per-MON Paxos and election counters (run on the MON host)
ceph daemon mon.$(hostname -s) perf dump | jq '{paxos: .paxos, mon: .mon}'

# Monitor store size (large stores slow elections)
du -sh /var/lib/ceph/mon/ceph-$(hostname -s)/store.db

# Time daemon status on the MON host
chronyc tracking
chronyc sources -v

# Generic timedatectl status (covers timesyncd too)
timedatectl status
timedatectl timesync-status

How to diagnose it

1. Confirm the warning is current and identify the offender. ceph health detail names the skewed monitor and shows the magnitude. If only one monitor is named, the other two are the reference. If multiple are named, the cluster has no common time base and the underlying NTP failure is broader.

2. Establish whether this is transient or structural. The first occurrence after boot, after a live migration, or after a network blip may clear within minutes as NTP reconverges. Use ceph_health_detail{name="MON_CLOCK_SKEW"} sustained for more than 60 seconds as the operational signal; anything shorter is likely convergence noise.

3. Check the time daemon on every MON host, not just the leader. The leader reports the skew; the offender is usually elsewhere. On each monitor, run chronyc tracking and look at System time and Last offset. If the daemon reports Not synchronised, the source is unreachable or the daemon is misconfigured.

4. Correlate skew with election churn. Pull Paxos counters from each monitor with ceph daemon mon.<id> perf dump. The relevant fields are under paxos (commit latency, accept latency) and mon (election_call, election_win, election_lose). Election counts incrementing more than once per hour while skew is active confirm the cascade.

5. Distinguish host-level from cluster-level. If only one MON host drifts, the problem is local (daemon, hardware clock, hypervisor pressure). If all MONs drift together, the NTP source itself is wrong or unreachable everywhere (broken upstream, restrictive firewall, shared dependency).

6. Rule out the monitor store. A large or uncompacted RocksDB store slows Paxos rounds and can mimic election instability. Check du -sh /var/lib/ceph/mon/ceph-*/store.db on every MON. Healthy clusters keep this well under 5GB. Run ceph daemon mon.<id> compact during a maintenance window if a single store is bloated.

7. In Rook or containerized deployments, check the host, not the pod. Monitor containers inherit the host clock. Running chrony inside the MON pod is a common but ineffective workaround; the host kernel clock is what matters. The fix lives on the Kubernetes node, not in the MON container.

Metrics and signals to monitor

Fixes

Restart and reconfigure the time daemon

If chronyd is stopped or wedged, restarting it is the first action. Verify the source list is sensible before restarting so the daemon converges on a real upstream:

# Inspect current sources
chronyc sources -v

# Restart chrony
sudo systemctl restart chronyd
sudo systemctl enable chronyd

# Watch convergence
watch -n 2 chronyc tracking

If the source list is empty or only points to unreachable hosts, add reachable servers and restart. On AWS use 169.254.169.123; on GCP use metadata.google.internal (the metadata server also serves NTP on GCP instances). For bare metal, prefer multiple low-stratum sources from the NTP Pool or your network’s stratum-2 servers.

Replace systemd-timesyncd with chrony

systemd-timesyncd is fine for general hosts but its clock discipline is too loose for monitors. Community reports from Proxmox, ServerFault, and Rook issues consistently show that switching to chrony resolves persistent MON_CLOCK_SKEW that timesyncd could not. Disable timesyncd before enabling chronyd to avoid two daemons fighting:

sudo systemctl disable --now systemd-timesyncd
sudo systemctl enable --now chronyd

Move virtualized MONs to bare metal

The Ceph documentation is explicit: run NTP on bare metal; VM-virtualized clocks are not suitable for steady timekeeping. If your monitors are VMs and skew recurs, the durable fix is to deploy at least the leader and one peer on bare metal. If that is not possible, reduce the surface for drift:

• Pin MON VMs to dedicated cores to reduce scheduling jitter.
• Disable the balloon driver and memory over-commit on MON VMs.
• Avoid live migration of MON VMs. If migration is required, schedule it during a maintenance window and expect brief skew warnings.
• Install the hypervisor’s guest integration (VMware Tools, qemu-guest-agent) and any host timekeeping integration it exposes.

Fix SELinux denials

On SELinux-enforced hosts, denials can prevent chronyd from synchronizing. The supported fix is to configure the correct policy, not to disable SELinux. Inspect denials and apply the recommended policy fix:

# Inspect denials
sudo ausearch -m AVC -ts recent | grep chrony
sudo sealert -a /var/log/audit/audit.log

Do not raise mon_clock_drift_allowed

The 0.05s value is tight because Paxos leases are time-bound; raising it lets drift get worse before Ceph warns, which moves the failure closer to quorum loss rather than further from it. Red Hat and upstream Ceph guidance both warn against changing this value without testing, and testing in this context means deliberately inducing skew on a non-production cluster and observing election behavior.

If, after fixing timekeeping, residual warnings persist from a single host with known small drift (for example, false positives at 0.011s during NTP convergence), the right action is to wait for convergence and watch the trend, not to widen the threshold.

Reduce election churn while you fix the cause

If skew is producing election storms and CLI commands are slow, you can stop the worst monitor temporarily to restore stable quorum while you fix the underlying clock:

# Stop the offending MON daemon on its host
sudo systemctl stop ceph-mon@<id>

Removing one monitor from a 3-MON cluster leaves a 2-of-3 majority, so quorum holds. This is a temporary measure, not a fix; a 3-MON cluster has no fault tolerance with one monitor down. Restart the monitor as soon as its clock is stable.

Prevention

• Run chrony on every MON host. Prefer chrony over ntpd and over systemd-timesyncd. Configure at least three independent upstream sources and monitor chronyc tracking offset as a time-series metric.
• Run MONs on bare metal when possible. If you must virtualize, pin the VM, disable balloon and over-commit, and never live-migrate MON VMs.
• Monitor skew as a time series, not just a health check. Track ceph_health_detail{name="MON_CLOCK_SKEW"} with a 60-second sustain for tickets. Treat any individual monitor losing quorum as a separate, more urgent signal.
• Monitor election epoch rate. A flat or slowly growing epoch is healthy. Spikes coincide with user-visible slowness in ceph -s, ceph status, and map distribution.
• Track MON store size. Large stores amplify election latency and make the cluster more sensitive to skew. Alert if any MON store exceeds 5GB or grows faster than 100MB/day.
• Track chrony offset on MON hosts directly. Netdata’s chrony collector exposes the system offset and source reachability as per-second metrics. This catches drift before it crosses the 50ms threshold that triggers Ceph.
• In Rook/Ceph, configure time on the Kubernetes nodes, not in the MON pods. Verify with chronyc tracking on each node that runs a MON pod.

How Netdata helps

• Per-second ceph_health_detail{name="MON_CLOCK_SKEW"} distinguishes transient post-boot convergence (clears in seconds) from structural drift (sustained for minutes). The 60-second sustain aligns with the operational threshold.
• Correlate MON_CLOCK_SKEW with ceph_mon_quorum_status to confirm whether skew has graduated to quorum loss. Two signals, one chart, immediate context.
• Netdata’s chrony collector exposes system offset, last offset, and source reachability per-second on every MON host, so you see drift building before it crosses 50ms and triggers Ceph.
• ML anomaly detection on the election epoch counter surfaces election churn that correlates with skew events, even when no static threshold would have fired.
• Host-level CPU steal and scheduling metrics on MON hosts identify hypervisor-driven drift in virtualized deployments, distinguishing a Ceph problem from a VM placement problem.
• Filesystem metrics on the MON data directory flag bloated stores that amplify election sensitivity to skew.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitor quorum lost: the cluster can no longer update its maps
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio