CockroachDB admission control throttling: queue depth, store-write, and capacity headroom

When admission control starts queuing requests, p99 latency climbs while throughput stays flat or degrades. The cluster hasn’t crashed, disks aren’t full, and CPU may not be saturated. An internal flow-control system decided the node or store is at capacity and started holding work back to protect itself.

Admission control (v21.2+, enabled by default since v22.1) regulates work through five queues: kv, sql-kv-response, sql-sql-response, elastic-cpu, and store-write. Each gates a different class of work with distinct triggers. Knowing which queue is deep and why is the difference between a five-minute diagnosis and a multi-hour investigation into “why is the database slow.”

Queuing is a feature, not a bug. It prevents cascading overload. But sustained queuing means the system is running at its limit with zero burst headroom. Throughput is maintained by adding latency to every request.

What this means

Admission control is a token and slot-based system. The kv queue uses slots. The sql-kv-response, sql-sql-response, elastic-cpu, and store-write queues use tokens. When tokens or slots are exhausted, incoming work waits until capacity frees up. That wait time adds directly to user-visible latency.

The store-write queue is directly tied to Pebble LSM L0 health. Admission control begins shaping regular (foreground) traffic at 5 L0 sublevels and elastic (background) traffic at 1 sublevel. The store-write token bucket recalculates capacity periodically based on L0 compaction progress. When compaction cannot keep up with ingestion, the store-write queue deepens, deliberately slowing writes to prevent L0 from spiraling.

The admission.io.overload metric is a 1-normalized float. A value at or above 1.0 indicates the IO admission control system considers the store overloaded with respect to L0 compaction health. The L0 sub-level overload threshold is 20 (cluster setting admission.l0_sub_level_count_overload_threshold), and the file count overload threshold is 1000 (admission.l0_file_count_overload_threshold). A sustained L0 sublevel count above 10 typically indicates overload well before the hard threshold of 20 is hit. Do not wait for the threshold to be reached before investigating.

The elastic-cpu queue handles lower-priority work: backfills, schema changes, statistics collection. Queuing here is often acceptable. It means the system is correctly prioritizing foreground queries over background work. The queues to worry about are kv, sql-kv-response, and sql-sql-response. Sustained wait times on foreground queues means users are experiencing that latency directly.

Memory is not admission-controlled. CockroachDB explicitly omits memory from admission control because it is non-preemptible. Slowing KV processing to reduce memory pressure can make things worse. Memory pressure must be handled through SQL memory budgets, --max-sql-memory, and Go GC tuning.

flowchart TD
    A["Admission control queuing detected"] --> B{"Which queue has depth?"}
    B -->|"store-write"| C["Check L0 sublevels and admission.io.overload"]
    B -->|"elastic-cpu"| D["Likely acceptable"]
    B -->|"kv / sql-kv-response / sql-sql-response"| E["Corroborate with SQL latency"]
    C --> F{"L0 sublevels > 5?"}
    F -->|"Yes, rising"| G["Compaction falling behind: reduce write load or add I/O"]
    F -->|"No"| H["Check provisioned bandwidth setting"]
    D --> I["Check foreground queues for corroborating waits"]
    E --> J["Capacity ceiling: scale out or reduce load"]
    G --> K["Zero burst headroom confirmed"]
    H --> K
    I --> K
    J --> K

Common causes

Quick checks

All commands below are read-only and safe for production.

# Check all admission control metrics
curl -s http://localhost:8080/_status/vars | grep admission

# Check IO overload (1-normalized; >= 1.0 means overloaded)
curl -s http://localhost:8080/_status/vars | grep admission_io_overload

# Check L0 sublevels per store (primary trigger for store-write queue)
curl -s http://localhost:8080/_status/vars | grep storage_l0_sublevels

# Check for write stalls (late-stage LSM distress)
curl -s http://localhost:8080/_status/vars | grep storage_write_stalls

# Check WAL fsync latency (write-path health)
curl -s http://localhost:8080/_status/vars | grep 'storage_wal_fsync'

# Check CPU utilization trend
curl -s http://localhost:8080/_status/vars | grep -E 'sys.cpu.(user|sys).ns'

# Check disk I/O utilization and latency
iostat -xz 1 3

For running jobs (admin-only):

SELECT job_id, job_type, status, running_status, fraction_completed
FROM crdb_internal.jobs
WHERE status NOT IN ('succeeded','canceled')
ORDER BY created DESC;

How to diagnose it

1. Identify which queues are deep. Pull all admission metrics and look at wait durations per queue. The metric names follow the pattern admission.wait_durations.<queue> as histograms. For the store-write queue specifically, look at admission.wait_durations.kv-stores and admission.wait_durations.elastic-stores histograms.

2. Classify the queuing. If only elastic-cpu is deep, the system is working correctly: background work is being deprioritized. If foreground queues (kv, sql-kv-response, sql-sql-response) have sustained wait times above 10ms over a 5-minute window, the system is at capacity and users are feeling it.

3. If store-write is deep, trace to LSM health. Check storage_l0_sublevels per store. The store-write queue begins shaping regular traffic at 5 sublevels and elastic traffic at 1. If L0 is above 10, compaction is not keeping up. If above 20, write stalls are imminent or active. Check admission.io.overload: a value at or above 1.0 confirms IO-driven throttling.

4. Correlate with SQL latency. Check sql_service_latency p99. If admission control wait time maps to latency increases, the queuing is directly causing user-visible impact. Admission wait duration adds directly to query latency.

5. Check for concurrent operations. Bulk operations (IMPORT, RESTORE, schema change backfills, backups) generate load that can push a cluster past its capacity. Check crdb_internal.jobs for running operations that correlate with the queuing onset.

6. Check disk I/O. If store-write is deep and L0 is elevated, the root cause is likely disk I/O bandwidth. Check iostat for utilization and await latency. Check WAL fsync latency: p99 above 50ms on SSDs is a strong I/O saturation signal. If compaction throughput equals sustained disk write bandwidth, there is zero headroom.

7. Check CPU. If elastic-cpu and kv queues are both deep, CPU may be saturated. CockroachDB recommends keeping sustained CPU below 60-70%. Check range count per node: high range counts multiply Raft ticking CPU overhead. A node with 50,000 ranges needs significantly more CPU just for Raft ticking than one with 5,000.

Metrics and signals to monitor

Fixes

Reduce write load

If store-write is deep and L0 is climbing, the most immediate fix is to reduce write rate. Pause bulk operations, schema changes, and backups. If the workload itself is the cause, consider whether the cluster is sized correctly for sustained write throughput.

Tradeoff: Pausing bulk operations delays their completion. Schema changes resume from where they left off, but the table remains in the intermediate state longer.

Add disk I/O capacity

If compaction throughput is disk-limited, the options are constrained. On cloud volumes (EBS gp3, PD), increase provisioned IOPS and throughput. On local SSDs, the only option is to reduce write amplification or add nodes to spread the load.

Tradeoff: Cloud volume upgrades cost more and may require a volume modification with brief performance impact.

Check provisioned bandwidth setting

The cluster setting kvadmission.store.provisioned_bandwidth controls disk bandwidth-based admission control. If set too low for the actual hardware, the store-write queue will throttle prematurely. If set too high, admission control will not engage when it should.

Tradeoff: This setting should be validated carefully before relying on it in production.

Scale out the cluster

If foreground queues are consistently deep during normal peak traffic and no single cause explains it, the cluster is undersized. Adding nodes distributes range ownership (reducing per-node Raft CPU overhead) and spreads write load across more disk devices.

Tradeoff: Adding nodes triggers replica rebalancing, which itself generates I/O and network load for hours or days depending on data volume.

Address hot ranges

If one node shows admission queuing while others are idle, a hot range is funneling all writes through a single leaseholder. Check per-range QPS via crdb_internal.ranges or the DB Console Hot Ranges page. Short-term: ALTER TABLE ... SPLIT AT. Long-term: redesign sequential primary keys to use hash-prefixed or UUID-based patterns.

Prevention

• Monitor L0 sublevel count proactively. It is the single most predictive storage signal, giving 10-30 minutes of warning before write stalls. The store-write queue responds to L0 growth at 5 sublevels. If you wait until L0 is above 20, you are already in distress.

• Track admission queue wait times as a capacity signal. Regular foreground queuing means zero burst headroom. The system is maintaining throughput by adding latency. Alert on sustained foreground queue waits above 10ms over a 5-minute window.

• Size disk I/O with at least 2x headroom over sustained write ingestion. Compaction throughput must exceed ingestion rate to prevent L0 debt accumulation. If they are equal, any burst will tip the system into admission throttling.

• Monitor disk space. CockroachDB requires at least 20% free space for compaction. Below 15%, compaction may fail to run, creating a death spiral that no amount of admission control can prevent.

• Version awareness for v26.2+. The unit of measurement for admission control duration metrics changed from microseconds to nanoseconds in v26.2. Affected metrics include admission.granter.slots_exhausted_duration.kv, admission.granter.io_tokens_exhausted_duration.kv, and admission.elastic_cpu.nanos_exhausted_duration, among others. Existing dashboards and alert thresholds will show values 1000x higher after upgrade. Update thresholds accordingly.

How Netdata helps

• Per-second admission metrics. Netdata collects admission wait durations and admission.io.overload at per-second resolution, catching burst throttling events that coarser scrape intervals miss.

• Correlation across queues and LSM health. When store-write queue depth rises, Netdata correlates it with storage_l0_sublevels, storage_write_stalls, WAL fsync latency, and disk I/O utilization on the same timeline.

• Foreground vs. elastic queue separation. Each admission queue is surfaced independently, so you can distinguish acceptable elastic-cpu queuing from capacity-limiting foreground throttling without manual metric filtering.

• Anomaly detection on admission signals. ML-based anomaly detection flags unusual admission control behavior before sustained queuing becomes a user-visible latency problem.

• Per-node and per-store granularity. Admission control operates per-node and per-store. Netdata’s per-node dashboards reveal asymmetry that cluster-level aggregates hide, such as one store hitting store-write throttling while others are healthy.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles