CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload

When a CockroachDB node dies, every client connected to it reconnects at the same time. Without jittered backoff in client connection pools, hundreds or thousands of new connections land on the surviving nodes within seconds. The survivors are not overwhelmed by additional query load. They are overwhelmed by connection overhead: per-connection goroutines, session memory allocations, TLS handshakes, and SQL planner initialization.

The signature is a 2-3x spike in sql_conns on surviving nodes, with a simultaneous jump in sys_goroutines and sql_mem_root_current. Latency rises across all nodes, not just those receiving the flood. The storm usually self-resolves as pools stabilize. But if memory pressure reaches OOM, one node failure can cascade into a multi-node outage.

Why it happens

CockroachDB does not include a built-in connection pooler. Each client connection gets a dedicated goroutine and per-session memory. Admission control manages SQL request execution, KV operations, and storage writes, but does not regulate incoming client connections. The only server-side limit is server.max_connections_per_gateway, which rejects non-superuser connections beyond the configured threshold with PostgreSQL error code 53300 (“sorry, too many clients already”). The default is effectively unlimited — no cap is applied unless the setting is explicitly configured.

When a node fails, the cluster redistributes its range leases within seconds. From the application perspective, the load balancer detects the dead node and routes new connections to survivors. If every application instance reconnects simultaneously without staggered backoff, the connection establishment cost hits all survivors at once. Each new connection requires a TCP handshake, TLS negotiation, a goroutine for the pgwire session, SQL memory allocation for session state, and authentication setup.

At scale, this overhead alone can push memory and CPU past sustainable limits on the surviving nodes, even though the actual query workload has only increased by a fraction (the dead node’s share divided among the survivors).

flowchart TD
    A["Node N crashes or drains"] --> B["Load balancer detects failure"]
    B --> C["All clients reconnect simultaneously"]
    C --> D["sql_conns spike 2-3x on survivors"]
    D --> E["goroutines + sql_mem spike"]
    E --> F{"Memory exceeds limit?"}
    F -->|"No"| G["Storm self-resolves
as pools stabilize"]
    F -->|"Yes"| H["Node OOM-killed"]
    H --> I["Second node failure
triggers next wave"]
    I --> C

Common causes

Quick checks

# Check current connection count per node
curl -s http://localhost:8080/_status/vars | grep '^sql_conns'

# Check rate of new connection establishment
curl -s http://localhost:8080/_status/vars | grep 'sql_new_conns'

# Check goroutine count (should correlate with connections)
curl -s http://localhost:8080/_status/vars | grep 'sys_goroutines'

# Check SQL memory utilization
curl -s http://localhost:8080/_status/vars | grep 'sql_mem_root_current'

# Check process RSS vs. memory limit
curl -s http://localhost:8080/_status/vars | grep 'sys_rss'

# Check SQL latency during the event
curl -s http://localhost:8080/_status/vars | grep 'sql_service_latency'

# Check node liveness status
curl -s http://localhost:8080/_status/nodes | python3 -c "
import json, sys
for n in json.load(sys.stdin)['nodes']:
    print(f'Node {n[\"desc\"][\"node_id\"]}: liveness={n.get(\"liveness\",{}).get(\"liveness\",\"UNKNOWN\")}')"

# Check total SQL failure count (query crdb_internal.node_statement_statistics for error code breakdown)
curl -s http://localhost:8080/_status/vars | grep 'sql_failure_count'

# Check active session distribution across nodes (requires auth: --certs-dir or --insecure)
cockroach sql -e "SELECT node_id, count(*) FROM [SHOW CLUSTER SESSIONS] GROUP BY node_id ORDER BY count(*) DESC;"

How to diagnose it

1. Confirm the timeline. Correlate the sql_conns spike with a node liveness transition or restart. The connection storm signature is a sharp jump at the exact moment a node goes down, not a gradual ramp.

2. Check whether connections are stabilizing or still climbing. If sql_conns plateaus and then slowly decreases, the storm is self-resolving. If it keeps climbing, client pools are creating connections faster than they close them.

3. Compare memory growth to query load. If sql_mem_root_current and sys_rss are climbing much faster than SQL throughput, the overhead is connection establishment, not query execution. This confirms the stampede pattern rather than a genuine workload increase.

4. Check for OOM precursors. Query SELECT key, count, last_error_code FROM crdb_internal.node_statement_statistics WHERE last_error_code IS NOT NULL ORDER BY count DESC LIMIT 20; and look for error code 53200 (resource exhaustion). If these appear, the node is rejecting queries due to memory pressure and may be OOM-killed soon.

5. Identify which clients are reconnecting. Use SHOW CLUSTER SESSIONS to see session distribution across nodes and application instances. If one application instance is opening hundreds of connections, its pool is misconfigured.

6. Check whether the dead node is crash-looping. Each restart attempt may trigger another wave of connection churn as clients detect the port as open, attempt to connect, then fail again.

Metrics and signals to monitor

Fixes

Immediate: ride out the storm

The connection storm is usually self-resolving. Client pools reach their max size, idle connections time out, and the system stabilizes within minutes. If memory is not approaching limits, the best action is to wait and monitor.

If memory pressure is critical and OOM is imminent on a surviving node, identifying and throttling the most aggressive client is a last resort.

Client-side pool tuning (primary fix)

The root cause is almost always client-side pool configuration. CockroachDB’s connection pooling guidance recommends:

• Max lifetime: 5-30 minutes. Connections should be recycled regularly to distribute load.
• Connection jitter: 10% of max lifetime. This is the critical anti-stampede setting. If max lifetime is 30 minutes, jitter should be 3 minutes. This ensures connections are recycled at different times rather than in lockstep.

Without jitter, all connections created at the same time (such as during application startup or after a failover) expire simultaneously, creating a secondary stampede.

Reconnection backoff

Client libraries must use exponential backoff with randomization on connection failure. When a node dies, reconnection attempts should be spread over time, not fire all at once. Without randomized jitter on retry intervals, clients that detected the failure simultaneously will retry simultaneously, perpetuating the stampede. CockroachDB’s official docs address connection pool jitter but do not publish specific reconnection backoff recommendations; the guidance below is general client-side best practice.

Check your client library or connection pool configuration for:

• Exponential backoff on connect failure (not a fixed retry interval)
• Randomized jitter on each retry to desynchronize clients
• A maximum retry delay to prevent excessively long waits

Graceful drain configuration

For planned maintenance, ensure the drain timeout gives clients time to reconnect gradually:

• server.shutdown.connections.timeout controls how long the draining node waits for SQL connections to close. The default is 0s, which terminates connections immediately.
• The client pool’s max lifetime must be shorter than server.shutdown.connections.timeout, or connections will be forcefully terminated during drain rather than recycled gracefully. This forces all clients to reconnect simultaneously to surviving nodes.

On CockroachDB Dedicated, the total drain timeout is bounded by the pod’s terminationGracePeriodSeconds, which is 300s (5 minutes) and not user-configurable. Plan rolling restarts accordingly.

Server-side connection limits

Set server.max_connections_per_gateway to prevent unbounded connection acceptance. This is a safety net, not a primary control. When the limit is exceeded, new non-superuser connections receive error 53300. Size the limit based on the cluster’s vCPU count: CockroachDB documentation recommends active connections not exceed 4x the number of vCPUs.

Prevention

• Use PgBouncer or an equivalent pooler in transaction mode. This caps the number of connections to CockroachDB regardless of how many application instances are running. Transaction mode is safe with CockroachDB. Do not use statement mode, which breaks multi-statement transactions.

• Configure connection jitter on all client pools. A jitter of 10% of max lifetime is the recommended setting. This is the single most effective prevention measure for stampedes.

• Use /health?ready=1 for load balancer health checks. This endpoint returns 503 when a node is draining or has lost quorum. Plain TCP health checks will route traffic to nodes that are technically listening but functionally impaired.

• Tune drain timeouts for rolling maintenance. Set server.shutdown.connections.timeout to a value longer than your client pool’s max lifetime.

• Monitor sql_conns with per-second granularity. The storm is sharp and short. At 15-30 second scrape intervals, the peak may be invisible. Per-second monitoring catches the spike and its resolution.

How Netdata helps

• Per-second sql_conns and sql_new_conns collection captures the connection spike that slower scrapers miss entirely. The storm may last only 30-60 seconds.
• Correlation of sql_conns with sys_goroutines, sql_mem_root_current, and sys_rss on a single timeline confirms whether memory growth is connection-driven rather than query-driven.
• Node liveness transitions overlaid with connection metrics show the causal relationship between node loss and the reconnect stampede.
• Latency correlation across all nodes distinguishes a connection storm (latency up everywhere, connections spiked) from a hot range (latency up on one node, connections stable).

Netdata’s database monitoring brings these signals together with per-second metrics and anomaly detection.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles