CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth

When intentcount and intentbytes climb and refuse to come down, your cluster is accumulating unresolved write intents from transactions that never committed or rolled back. Every subsequent transaction touching those keys must stop, resolve the intent, then proceed. At scale, the cluster spends more CPU and I/O resolving old intents than executing new work.

This is the intent accumulation cascade. It is one of CockroachDB’s subtler failure modes because the cluster looks healthy on infrastructure metrics: nodes are live, ranges are available, disk I/O is within bounds. The damage shows up in transaction latency and throughput, not in availability signals.

Intent counts normally track active transaction volume, so a steady climb can look like harmless growth from increased load. The difference is the trajectory. Intents from healthy transactions resolve within seconds as the coordinator commits or rolls back. Abandoned intents from crashed clients or stuck transactions persist for minutes or hours, and they accumulate.

What this means

CockroachDB uses MVCC write intents to represent uncommitted writes. When a transaction writes a key, it stores a provisional value (an intent) with a pointer to the transaction record. Other transactions that read or write the same key must resolve that intent before proceeding. Resolution means checking the transaction record’s status (PENDING, STAGING, COMMITTED, or ABORTED) and potentially issuing a PushTxn request to force the transaction to a terminal state.

Under normal conditions, the transaction coordinator resolves its own intents at commit or rollback time. The problem arises when the coordinator disappears: the application crashes mid-transaction, a network timeout severs the connection, or a long-running batch job holds intents for hours. Those intents remain in storage, and resolution falls to whichever transaction encounters them next.

This creates a tax on every operation touching affected keys. The more intents accumulate, the more work each new transaction does cleaning up after abandoned predecessors. At scale, this tax dominates real work.

flowchart TD
    A[Abandoned transaction leaves write intents] --> B[Intents persist in storage across keys]
    B --> C[New transactions encounter unresolved intents]
    C --> D[Each must resolve: check txn record, issue PushTxn if needed]
    D --> E[Per-operation latency increases]
    E --> F[Retry rate climbs as txns hit contention]
    F --> G[Cluster-wide P99 latency rises, throughput drops]

Common causes

Quick checks

These are read-only diagnostics safe to run during an active incident.

# Check intent count and bytes per store
curl -s http://localhost:8080/_status/vars | grep 'intent'

-- Find sessions with active queries, oldest first
SELECT * FROM [SHOW CLUSTER SESSIONS] WHERE active_queries != '' ORDER BY start DESC;

-- List all active transactions across the cluster
SHOW CLUSTER TRANSACTIONS;

-- Find ranges with the most unresolved intents (admin-only, version-sensitive)
<!-- TODO: verify column names in crdb_internal.ranges for target versions -->
SELECT range_id, start_key_pretty, lease_holder, intent_count
FROM crdb_internal.ranges
WHERE intent_count > 0
ORDER BY intent_count DESC
LIMIT 20;

# Check transaction restart rate by cause
curl -s http://localhost:8080/_status/vars | grep 'txn_restarts'

-- Inspect recent contention events (admin-only, expensive cluster-wide RPC)
SELECT * FROM crdb_internal.transaction_contention_events
ORDER BY contention_time DESC
LIMIT 50;

# Check transaction commit vs abort rate
<!-- TODO: verify exact metric names for commit/abort counters in target versions -->
curl -s http://localhost:8080/_status/vars | grep -E 'sql_txn_commit_count|sql_txn_abort_count'

crdb_internal tables are unsupported and may change between versions. Use them for manual diagnosis, not automated monitoring pipelines.

How to diagnose it

1. Confirm intent growth is abnormal. Pull intentcount and intentbytes over the last several hours. Normal intent counts track active transaction volume. If count is growing while active connections are flat or declining, you have abandoned intents.

2. Identify the source transactions. Run SHOW CLUSTER SESSIONS and look for sessions with old start timestamps. Cross-reference with SHOW CLUSTER TRANSACTIONS to find transactions in PENDING state running far longer than your application’s expected transaction duration.

3. Locate where intents are concentrated. Query crdb_internal.ranges for ranges with high intent_count. If intents cluster on specific tables or key ranges, the source is likely a transaction operating on those keys.

4. Check whether resolution is keeping up. If intentcount is rising but the intent resolution rate is also high, the source is still actively generating intents. If intentcount is rising and resolution rate is low or flat, the cleanup mechanism is overwhelmed or blocked.

5. Correlate with latency impact. Check txn_durations P99 and txn_restarts with the txnpush cause. Rising transaction latency with elevated txnpush restarts confirms that intent resolution is the bottleneck.

Metrics and signals to monitor

Fixes

Cancel the source session

If SHOW CLUSTER SESSIONS identifies the abandoned or long-running transaction:

-- Cancel the session generating the intents
CANCEL SESSION <session_id>;

This forces the transaction to abort, which triggers cleanup of its write intents. The session’s in-flight queries are terminated immediately.

For finer granularity, CANCEL QUERY <query_id> cancels a specific statement without closing the entire session.

Warning: CANCEL SESSION is disruptive. The client application will receive a connection error. Use it when the alternative (letting the transaction run indefinitely) is worse.

Address long-running batch jobs

Large INSERT INTO ... SELECT, bulk UPDATE, or data migration transactions can generate millions of intents across wide key ranges. If the job is legitimate but running too long:

• Break it into smaller batches with explicit transaction boundaries.
• Add LIMIT clauses and loop in the application with commit per batch.
• Consider IMPORT for bulk loads, which bypasses the normal transaction path.

Check cluster settings for intent cleanup

The cluster setting kv.transaction.abandoned_intents_cleanup_interval controls how aggressively the background process cleans up abandoned intents. If set too conservatively, abandoned intents persist longer than necessary.

-- Check current setting
SHOW CLUSTER SETTING kv.transaction.abandoned_intents_cleanup_interval;

The MVCC GC queue also handles intent cleanup as a last resort, running periodically over ranges. This is slower than on-access resolution but will eventually clean up intents that no active transaction encounters.

Reduce per-transaction intent footprint

For workloads where individual transactions write to many keys, the kv.transaction.max_intents_bytes setting limits the memory budget for tracking point intents per transaction.

When a transaction exceeds this budget, point intent resolution switches to range intent resolution. Range resolution requires scanning entire key ranges to find and resolve intents, which is dramatically more expensive and can itself cause performance degradation.

The setting kv.transaction.reject_over_max_intents_budget.enabled acts as a circuit breaker. When enabled, a transaction exceeding the intent tracking budget is rejected with error 53400 (“configuration limit exceeded”) rather than silently degrading to range resolution.

Prevention

• Set idle transaction timeouts. Configure idle_in_transaction_session_timeout at the cluster or session level to automatically abort transactions that sit idle without committing. This catches application bugs where a transaction is opened but never closed.

• Ensure applications send ROLLBACK on exit. Application crash handlers and connection pool eviction callbacks should explicitly roll back open transactions before releasing connections to the pool.

• Monitor intentcount proactively. Alert on monotonic growth over a 1-hour window, or when intentbytes exceeds 1% of total data size. Do not wait for latency to degrade before investigating.

• Watch the intent-to-transaction ratio. If intentcount diverges from active transaction count, abandoned intents are accumulating. Healthy systems show intent count that rises and falls with transaction volume.

• Batch large writes deliberately. Application-level chunking with explicit commit boundaries prevents any single transaction from generating millions of intents that must be tracked and resolved.

How Netdata helps

• Per-second intent metrics. Netdata collects intentcount and intentbytes at per-second resolution, making monotonic growth visible faster than 15-30 second scrape intervals that may miss short resolution stalls.

• Correlation with transaction latency. Overlaying intentcount against txn_durations P99 and txn_restarts (by cause) in a single view confirms whether latency degradation tracks intent accumulation or has a different root cause.

• Anomaly detection on intent trajectory. ML-based anomaly flags can distinguish normal intent fluctuation (correlated with transaction volume) from abnormal accumulation (growing independent of workload).

• Store-level granularity. Per-store intent metrics let you identify whether accumulation is cluster-wide or concentrated on specific nodes, narrowing the search for the source transaction.

• Composite signal alerting. Alerting when intent growth coincides with rising transaction restarts and declining commit rate catches the cascade signature early, before throughput collapses.

Netdata’s CockroachDB monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs
• CockroachDB monitoring maturity model: from survival to expert
• CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping