CockroachDB monitoring checklist: the signals every production cluster needs

CockroachDB layers SQL execution on a replicated KV store backed by Pebble LSM trees, Raft consensus, and MVCC concurrency control. Each subsystem has distinct failure modes, and interactions between them create cascades that single-signal monitoring cannot catch. A cluster can show healthy CPU, adequate disk space, and sub-millisecond SQL latency while L0 sublevels climb toward write stalls or clock offset drifts toward the self-termination threshold.

This checklist defines four monitoring maturity levels (survival, operational, mature, expert) and maps every production signal to its level. Use it to audit current coverage, identify gaps before they cause incidents, and plan what to add as your cluster grows.

The four monitoring levels

The levels are cumulative. Each builds on the one below it.

flowchart BT
    L1["Level 1: Survival
5 signals: is it broken?"]
    L2["Level 2: Operational
16 more: is it healthy?"]
    L3["Level 3: Mature
20 more: leading indicators"]
    L4["Level 4: Expert
18 more: deep internals"]
    L1 --> L2 --> L3 --> L4

• Survival: is the cluster fundamentally broken? If you have nothing else, have these.
• Operational: is the cluster healthy, not just alive? A team running CockroachDB in production should have all of these.
• Mature: leading indicators and storage internals that give 10 to 30 minutes of warning before degradation becomes an outage.
• Expert: deep internals and per-range analysis. These are the signals operators add after repeated incidents, when a failure mode was invisible until too late.

Level 1: survival

Use /health?ready=1 for load balancer health checks. It returns 503 when a node is draining or quorum is lost. Plain TCP checks will route traffic to functionally impaired nodes.

The liveness mechanism is evolving across versions, transitioning toward store-level liveness and lease-based failure detection. Check the release notes for your deployment version to understand the exact heartbeat and expiry behavior.

Level 2: operational

Everything in Level 1, plus:

Level 3: mature

Everything in Level 2, plus leading indicators:

Level 4: expert

Everything in Level 3, plus signals operators add after repeated incidents:

How to alert on these signals

Not every signal should trigger a page. The playbook distinguishes PAGE (wake someone up) from TICKET (investigate during business hours).

Gate on sustained duration. Most conditions require sustained presence (typically over 5 minutes) before alerting. Transient spikes from range splits, lease transfers, or brief I/O contention are normal operating noise.

Gate on node uptime. Suppress most performance alerts during the first 10 minutes after process start (sys_uptime). Cache warmup takes 10 to 30 minutes. Statistics collection after restart may produce suboptimal query plans temporarily.

Binary signals page immediately. ranges_unavailable is zero under normal conditions. Any nonzero value sustained over 5 minutes is live user impact. storage_disk_stalled is similarly binary: nonzero means the node may self-terminate.

Workload-shaped signals require context. CPU at 80 percent alone is not page-worthy. CPU at 80 percent plus liveness flapping plus elevated lease transfers is a cascade. Latency thresholds are workload-dependent: establish baselines per statement fingerprint and alert on deviation (over 2x rolling P99 is a reasonable starting point), not on absolute values.

Distinguish restart causes. Alarm on txn_restarts breakdown, not the aggregate. readwithinuncertainty appearing for the first time is a clock problem. writetooold climbing is a schema or contention problem. txnpush spiking is an application transaction design problem. Each routes to a different team.

Treat recovery activity with suspicion. Snapshot and rebalance storms during cluster healing compete with foreground traffic. Recovery I/O can degrade the cluster further. Monitor recovery rate and foreground latency together.

What most teams miss

These gaps appear most often in incident reviews:

• L0 sublevel count uninstrumented until write stalls. The most common and damaging gap. L0 gives 10 to 30 minutes of warning before stalls, and teams consistently waste that window because the metric was never collected.
• Clock offset not monitored proactively. NTP is treated as set-and-forget. The first sign of trouble is a node self-terminating at 3 a.m. Meanwhile, readwithinuncertainty restarts have been degrading tail latency for days.
• Aggregate latency instead of per-statement-fingerprint latency. A new slow query among fast queries barely moves P99 but kills specific endpoints. Per-fingerprint tracking catches plan regressions and missing indexes.
• Transaction restart causes not distinguished. Total retry rate is alarmed without breakdown, wasting diagnosis time because the three main causes need completely different responses.
• MVCC garbage and protected timestamps unmonitored. Data gets deleted but nobody checks whether GC reclaims space. Protected timestamps from stalled CDC or hung backups silently prevent GC. Disk fills with dead data.
• Range count not tracked as a scaling dimension. Operators monitor data volume and query throughput but miss that 200,000 ranges per node behaves fundamentally differently from 20,000 because of per-range Raft processing overhead.
• TCP health checks instead of /health?ready=1. Load balancers route traffic to draining, write-stalled, or GC-thrashing nodes because the TCP check succeeds.
• Backup duration trend not tracked. Backup succeeds so the check is green. But duration grew from 20 minutes to 3 hours. Eventually it exceeds the backup interval, creating overlapping backups or gaps.
• Admission control ignored as a capacity signal. Regular AC queuing means zero burst headroom. The system is maintaining throughput by adding latency, not by having spare capacity.
• Cluster averages trusted over per-node data. One hot leaseholder or overloaded region hides under healthy global metrics. Always monitor per-node.

How Netdata helps

• Per-second collection from CockroachDB’s /_status/vars endpoint means brief write stalls and lease transfers are visible, not hidden by 15 to 30 second scrape gaps.
• ML anomaly detection on storage_l0_sublevels, read amplification, and write stalls catches slow drift toward compaction death spirals before static thresholds fire.
• Cross-signal correlation links clock offset spikes to readwithinuncertainty restart rates, or L0 growth to admission control queue depth, so you see the cascade instead of isolated symptoms.
• Per-node dashboards prevent hot ranges and single-node degradation from hiding under cluster averages.
• Cold-start awareness uses sys_uptime to suppress false positives during the warmup window after node restarts.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
• CockroachDB Pebble write stalls: when the storage engine refuses writes
• CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
• CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum
• CockroachDB replica unavailable: lost quorum and stuck Raft groups
• How CockroachDB actually works in production: a mental model for operators