CockroachDB layers SQL execution on a replicated KV store backed by Pebble LSM trees, Raft consensus, and MVCC concurrency control. Each subsystem has distinct failure modes, and interactions between them create cascades that single-signal monitoring cannot catch. A cluster can show healthy CPU, adequate disk space, and sub-millisecond SQL latency while L0 sublevels climb toward write stalls or clock offset drifts toward the self-termination threshold.
This checklist defines four monitoring maturity levels (survival, operational, mature, expert) and maps every production signal to its level. Use it to audit current coverage, identify gaps before they cause incidents, and plan what to add as your cluster grows.
The four monitoring levels
The levels are cumulative. Each builds on the one below it.
flowchart BT
L1["Level 1: Survival
5 signals: is it broken?"]
L2["Level 2: Operational
16 more: is it healthy?"]
L3["Level 3: Mature
20 more: leading indicators"]
L4["Level 4: Expert
18 more: deep internals"]
L1 --> L2 --> L3 --> L4- Survival: is the cluster fundamentally broken? If you have nothing else, have these.
- Operational: is the cluster healthy, not just alive? A team running CockroachDB in production should have all of these.
- Mature: leading indicators and storage internals that give 10 to 30 minutes of warning before degradation becomes an outage.
- Expert: deep internals and per-range analysis. These are the signals operators add after repeated incidents, when a failure mode was invisible until too late.
Level 1: survival
| Signal | Metric / source | What it tells you | Alert posture |
|---|---|---|---|
| Node liveness | /_status/nodes, /health?ready=1 | Whether the cluster considers each node alive. Loss triggers lease redistribution. | PAGE: unexpected not-live transition sustained over 5 min, node not draining |
| Range unavailability | ranges_unavailable | Ranges with no leaseholder or lost Raft quorum. Any nonzero value means some keyspace is unreadable or unwritable. | PAGE: any nonzero value sustained over 5 min |
| Disk space | capacity_available | Free space per store. CockroachDB recommends keeping below 60 percent utilization (at least 40 percent free). Running out prevents compaction and triggers a storage death spiral. | PAGE: under 10 percent available and decreasing. TICKET: under 30 percent |
| SQL connectivity | Synthetic probe: cockroach sql -e "SELECT 1" | Whether clients can connect and execute queries through the full pgwire path (TCP, TLS, SQL execution). | PAGE: probe failure sustained over 2 min |
| Certificate expiration | security.certificate.expiration.* metrics from /_status/vars, openssl x509 -enddate | Time until TLS certificates expire. Expiry breaks inter-node communication or client access. | TICKET: under 30 days. PAGE: expired or under 24 hours with no replacement staged |
Use /health?ready=1 for load balancer health checks. It returns 503 when a node is draining or quorum is lost. Plain TCP checks will route traffic to functionally impaired nodes.
The liveness mechanism is evolving across versions, transitioning toward store-level liveness and lease-based failure detection. Check the release notes for your deployment version to understand the exact heartbeat and expiry behavior.
Level 2: operational
Everything in Level 1, plus:
| Signal | Metric / source | What it tells you |
|---|---|---|
| SQL statement latency (P50, P99) | sql_service_latency | Client-visible query performance. P99 is where SLO violations live. |
| Transaction throughput | sql_select_count, sql_insert_count, sql_update_count, sql_delete_count | Workload volume and composition. Sudden drops need correlation with error rate to distinguish traffic decrease from system failure. |
| Transaction restart/retry rate | txn_restarts (by cause) | Contention level. writetooold means schema contention, readwithinuncertainty means clock skew, txnpush means application conflicts. Each requires a different response. |
| SQL error rate | sql_failure_count | Active failures visible to applications. Watch for XX000 (internal errors) and 53200 (resource exhaustion). |
| CPU utilization | sys_cpu_user_ns, sys_cpu_sys_ns | Capacity headroom. Per-range Raft ticking is a hidden CPU multiplier: 50,000 ranges costs significantly more CPU than 5,000 at identical query load. |
| Memory RSS | sys_rss | Proximity to memory limits. CGo allocations (Pebble block cache) grow independently of the Go heap. |
| LSM L0 sublevel count | storage_l0_sublevels | The single most predictive storage signal. 0 to 5 is healthy. 10 to 20 means active degradation. Over 20 means write stalls are imminent or active. |
| Active client connections | sql_conns | Connection pressure. Each connection consumes a goroutine and memory. Over 1,000 to 2,000 per node suggests client pooling is misconfigured. |
| Under-replicated range count | ranges_underreplicated | Replication safety margin. Sustained under-replication means the cluster cannot heal fast enough. |
| Clock offset | clock_offset_meannanos | Time sync health. Over 80 percent of --max-offset (default 500ms, so over 400ms) triggers self-termination. Even steady 200ms offset doubles the uncertainty interval and causes persistent read restarts. |
| Inter-node RPC latency | round_trip_latency | Network health as CockroachDB experiences it. Includes kernel scheduling delay, so CPU-saturated nodes show elevated RPC latency even with a healthy network. |
| Disk I/O utilization and latency | OS: iostat -xz 1, /proc/diskstats | Whether storage is the bottleneck. On cloud volumes, watch against provisioned IOPS caps, not just percentage utilization. |
| WAL fsync latency | storage_wal_fsync_latency | Most direct measure of write-path health. WAL fsync is on the critical path of every write. Over 50ms on SSDs signals I/O saturation. |
| Job status | crdb_internal.jobs, per-type Prometheus metrics | Whether backups, schema changes, and other long-running jobs are progressing or stuck. Track backup duration trend, not just success. |
| Admission control queue depth | admission.wait_durations.*, admission.io.overload | Whether internal flow control is throttling. The store-write queue begins shaping traffic at approximately 5 L0 sublevels. |
| Node uptime | sys_uptime | Cold-start gating signal. Suppress most performance alerts during the first 10 minutes after process start. Block cache warmup takes 10 to 30 minutes. |
Level 3: mature
Everything in Level 2, plus leading indicators:
| Signal | Metric / source | What it tells you |
|---|---|---|
| LSM read amplification | rocksdb_read_amplification | SSTable files consulted per read. Name retained from RocksDB era. 10 to 15 is normal. Over 25 indicates compaction debt. |
| Compaction throughput and backlog | rocksdb_compactions, storage_marked_for_compaction_files, storage_l0_num_files | Whether background storage maintenance keeps pace with writes. Upward backlog trend over hours means approaching write capacity. |
| Write stall count | storage_write_stalls | Pebble refusing writes because LSM state is unsafe. Rate over 1 per second sustained for over 1 minute means foreground writes are materially impaired. |
| Pebble block cache hit rate | rocksdb_block_cache_hits, rocksdb_block_cache_misses | Cache efficiency. Drop below 90 percent impacts read-heavy workloads. Sudden drop after restart is expected (cold cache). |
| SQL memory budget utilization | sql_mem_root_current | Memory available for query execution. Over 90 percent risks error 53200. Per-node, not per-query: one large query can starve all others. |
| Go GC pause duration and frequency | sys_gc_pause_ns, sys_gc_count | Pauses approaching the liveness heartbeat interval (default 3s) risk node death. GC CPU over 15 percent indicates memory pressure. |
| Goroutine count | sys_goroutines | Concurrency level. Monotonic growth without workload increase means leak. Over 100,000 is critical. |
| Range count per node | ranges, leases_count | Per-node overhead from Raft ticking. Over 50,000 ranges per node creates significant CPU cost independent of query load. |
| Raft snapshot rate | range_snapshots_generated, range_snapshots_applied_initial | Replication health. High rate beyond rebalancing means nodes cannot keep up with log application. Each snapshot transfers up to 512 MiB. |
| Lease transfer rate | leases_transfers_success | System churn. Each transfer creates a brief unavailability window. Elevated rate without operational cause indicates instability. |
| Intent count and bytes | intentcount, intentbytes | Unresolved write intents from uncommitted transactions. Growing count means abandoned transactions or intent resolution falling behind. |
| MVCC garbage bytes | MVCC metrics via /_status/vars | Dead data awaiting GC. Grows silently and eventually consumes disk. Protected timestamps from CDC or backups can block GC entirely. |
| KV read/write latency | exec_latency | Storage layer latency isolated from SQL planning overhead. Rising without SQL changes means storage degradation. |
| Raft log commit latency | raft.process.logcommit.latency | WAL fsync path specifically. Over 50ms on SSDs is a strong I/O saturation signal. |
| Changefeed lag (if using CDC) | changefeed_max_behind_nanos | CDC consumer freshness. A stalled changefeed creates protected timestamps that silently prevent MVCC GC, causing unbounded disk growth. |
| Protected timestamp count and age | spanconfig_kvsubscriber_protected_record_count, jobs_changefeed_protected_age_sec | Records preventing MVCC GC. Age over 24 hours or count growing without active backup or CDC means a stalled operation is blocking cleanup. |
| File descriptor usage | sys_fd_open, /proc/<pid>/fd | FD exhaustion prevents new connections and SSTable opens. Production should have ulimit of at least 35,000 to 65,000. |
| Composite pattern alerting | Multi-signal correlations | Detects cascades that individual thresholds miss. Example: L0 sublevels climbing plus write stalls plus lease transfers equals a compaction death spiral. |
| Per-table and per-index statistics | crdb_internal queries (diagnostic use only) | Size, row count, and query patterns for capacity planning. Do not use crdb_internal tables for automated monitoring pipelines; they are unsupported and may change between versions. |
| Network throughput between nodes | OS: /proc/net/dev | Inter-node bandwidth saturation. CockroachDB multiplexes all traffic on a single port, so Raft heartbeats cannot be prioritized over data transfer. |
Level 4: expert
Everything in Level 3, plus signals operators add after repeated incidents:
| Signal | What it tells you |
|---|---|
| Raft proposal drop rate | Dropped proposals mean silently lost writes (retried, but immediate latency impact). Rarely watched. |
| Per-range request rate distribution | Detects hot ranges before they become obvious from cluster-level metrics. A range at 10x average QPS bottlenecks its leaseholder while other nodes idle. |
| Intent resolution throughput | Separate from intent count. Tells you whether cleanup is keeping pace during an intent accumulation cascade. |
| Lease preference violation count | Whether the allocator can satisfy zone config lease preferences. Critical for multi-region latency SLOs. |
| Raft entry cache hit rate | Cache misses cause disk reads for log entries, a hidden latency source. |
| Compaction debt by LSM level | Shows where in the LSM tree pressure is building, not just the aggregate. |
Goroutine profiling (debug/pprof) | Stack traces for all goroutines. Detects where goroutines are stuck (I/O, lock contention) before count becomes critical. |
| KV write batch size distribution | Changes in write patterns that precede L0 pressure or compaction issues. |
| SQL plan cache hit rate | Cache misses mean re-planning, which is CPU-intensive. |
| Cross-range transaction percentage | Higher percentage means more distributed coordination overhead and latency. |
| Time-series data retention pressure | CockroachDB’s internal time-series data can itself become a storage burden on large clusters. |
| Node decommission progress rate | Ensures decommissions complete before patience runs out. Stalled decommissions block maintenance. |
| Admission control token exhaustion rate | How frequently work is being delayed, broken down by queue type. Foreground queue exhaustion is more concerning than elastic queue. |
| Closed timestamp lag | Affects follower read freshness in multi-region setups. |
| Queue processor error counts | Split, merge, replicate, and GC queue failures indicate internal scheduling problems. |
| Disk stall detection metrics | storage_disk_stalled (nonzero means the node may self-terminate) and storage_disk_slow (incrementing means slow disk operations). |
| Non-gateway SQL activity | Connections arriving at nodes not designated as application gateways. May indicate unauthorized direct access or misconfigured routing. |
| Bulk data export activity | Unexpected EXPORT or BACKUP operations. May indicate data exfiltration. |
How to alert on these signals
Not every signal should trigger a page. The playbook distinguishes PAGE (wake someone up) from TICKET (investigate during business hours).
Gate on sustained duration. Most conditions require sustained presence (typically over 5 minutes) before alerting. Transient spikes from range splits, lease transfers, or brief I/O contention are normal operating noise.
Gate on node uptime. Suppress most performance alerts during the first 10 minutes after process start (sys_uptime). Cache warmup takes 10 to 30 minutes. Statistics collection after restart may produce suboptimal query plans temporarily.
Binary signals page immediately. ranges_unavailable is zero under normal conditions. Any nonzero value sustained over 5 minutes is live user impact. storage_disk_stalled is similarly binary: nonzero means the node may self-terminate.
Workload-shaped signals require context. CPU at 80 percent alone is not page-worthy. CPU at 80 percent plus liveness flapping plus elevated lease transfers is a cascade. Latency thresholds are workload-dependent: establish baselines per statement fingerprint and alert on deviation (over 2x rolling P99 is a reasonable starting point), not on absolute values.
Distinguish restart causes. Alarm on txn_restarts breakdown, not the aggregate. readwithinuncertainty appearing for the first time is a clock problem. writetooold climbing is a schema or contention problem. txnpush spiking is an application transaction design problem. Each routes to a different team.
Treat recovery activity with suspicion. Snapshot and rebalance storms during cluster healing compete with foreground traffic. Recovery I/O can degrade the cluster further. Monitor recovery rate and foreground latency together.
What most teams miss
These gaps appear most often in incident reviews:
- L0 sublevel count uninstrumented until write stalls. The most common and damaging gap. L0 gives 10 to 30 minutes of warning before stalls, and teams consistently waste that window because the metric was never collected.
- Clock offset not monitored proactively. NTP is treated as set-and-forget. The first sign of trouble is a node self-terminating at 3 a.m. Meanwhile,
readwithinuncertaintyrestarts have been degrading tail latency for days. - Aggregate latency instead of per-statement-fingerprint latency. A new slow query among fast queries barely moves P99 but kills specific endpoints. Per-fingerprint tracking catches plan regressions and missing indexes.
- Transaction restart causes not distinguished. Total retry rate is alarmed without breakdown, wasting diagnosis time because the three main causes need completely different responses.
- MVCC garbage and protected timestamps unmonitored. Data gets deleted but nobody checks whether GC reclaims space. Protected timestamps from stalled CDC or hung backups silently prevent GC. Disk fills with dead data.
- Range count not tracked as a scaling dimension. Operators monitor data volume and query throughput but miss that 200,000 ranges per node behaves fundamentally differently from 20,000 because of per-range Raft processing overhead.
- TCP health checks instead of
/health?ready=1. Load balancers route traffic to draining, write-stalled, or GC-thrashing nodes because the TCP check succeeds. - Backup duration trend not tracked. Backup succeeds so the check is green. But duration grew from 20 minutes to 3 hours. Eventually it exceeds the backup interval, creating overlapping backups or gaps.
- Admission control ignored as a capacity signal. Regular AC queuing means zero burst headroom. The system is maintaining throughput by adding latency, not by having spare capacity.
- Cluster averages trusted over per-node data. One hot leaseholder or overloaded region hides under healthy global metrics. Always monitor per-node.
How Netdata helps
- Per-second collection from CockroachDB’s
/_status/varsendpoint means brief write stalls and lease transfers are visible, not hidden by 15 to 30 second scrape gaps. - ML anomaly detection on
storage_l0_sublevels, read amplification, and write stalls catches slow drift toward compaction death spirals before static thresholds fire. - Cross-signal correlation links clock offset spikes to
readwithinuncertaintyrestart rates, or L0 growth to admission control queue depth, so you see the cascade instead of isolated symptoms. - Per-node dashboards prevent hot ranges and single-node degradation from hiding under cluster averages.
- Cold-start awareness uses
sys_uptimeto suppress false positives during the warmup window after node restarts.
Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.
Related guides
- CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
- CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
- CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
- CockroachDB monitoring maturity model: from survival to expert
- CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping
- CockroachDB Pebble write stalls: when the storage engine refuses writes
- CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability
- CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum
- CockroachDB replica unavailable: lost quorum and stuck Raft groups
- How CockroachDB actually works in production: a mental model for operators