CockroachDB too many connections: sql_conns, pooling, and goroutine pressure

CockroachDB has no hard built-in connection limit comparable to PostgreSQL’s max_connections. Connections accumulate until something else breaks: goroutine pressure, memory exhaustion, or file descriptor limits.

The sql_conns gauge tracks active pgwire (SQL) connections per node. With properly configured client-side pooling, this number stays modest. When it climbs past 1000-2000 per node, you almost certainly have a pooling problem upstream.

Idle connections are relatively cheap because CockroachDB uses goroutines, not OS processes, per connection. The danger is the cascade: each connection consumes a goroutine (initial stack ), memory for session state, and a file descriptor. At scale, connection pressure degrades into goroutine scheduling overhead, SQL memory budget contention, or outright FD exhaustion. A node that cannot accept new file descriptors cannot accept new connections, new SSTable opens, or new inter-node gRPC connections.

What this means

Each pgwire connection is serviced by a dedicated goroutine. CockroachDB does not include a built-in server-side connection pooler. Connection management is the client’s responsibility, whether via application-side pools (HikariCP, pgxpool, SQLAlchemy) or external poolers like PgBouncer.

There is no single threshold where “too many connections” fires as an error. You see escalating symptoms as the node hits related resource limits:

• Goroutine pressure: Each connection adds goroutines for protocol handling, query execution, and session state. As goroutine count climbs past 30,000-50,000, Go runtime scheduling overhead increases, and CPU spent on goroutine management eats into the budget for SQL execution, Raft processing, and compaction.
• Memory pressure: The SQL memory budget (--max-sql-memory) is shared across all connections on a node. A flood of connections running queries competes for the same pool. When the pool is exhausted, queries spill to disk or fail with error code 53200 (out_of_memory).
• File descriptor exhaustion: Each connection consumes one FD. SSTable handles and inter-node gRPC connections also consume FDs. If ulimit -n is too low, FD exhaustion hits before any other limit, producing “too many open files” errors.

flowchart TD
    A[Client connections arrive] --> B{sql_conns rising?}
    B -->|Yes, high per node| C[Check client pool config]
    B -->|Sudden 2-3x spike| D[Check for node failover]
    C --> E{Pool misconfigured?}
    E -->|Yes| F[Fix pool sizing and backoff]
    E -->|No| G[Check for connection leak]
    D --> H[Verify backoff with jitter]
    B -->|No| I[Check FD limits]
    I --> J{FD usage near limit?}
    J -->|Yes| K[Raise ulimit -n]
    J -->|No| L[Check goroutine count]
    L --> M{Goroutines 2x baseline?}
    M -->|Yes| N[Profile goroutine stacks]
    M -->|No| O[Check SQL memory budget]

Common causes

Quick checks

All cockroach sql commands assume either --insecure or valid --certs-dir. Adjust for your cluster’s security mode.

# Current SQL connections per node
curl -s http://localhost:8080/_status/vars | grep sql_conns

# Goroutine count (each connection adds goroutines)
curl -s http://localhost:8080/_status/vars | grep sys_goroutines

# Active sessions with details (requires admin privileges)
cockroach sql -e "SHOW CLUSTER SESSIONS;" --host=localhost:26257 --insecure

# Connection failure rate
curl -s http://localhost:8080/_status/vars | grep sql_conn_failures

# File descriptor usage vs soft limit
ls /proc/$(pgrep -x cockroach)/fd | wc -l
cat /proc/$(pgrep -x cockroach)/limits | grep "open files"

# SQL memory budget utilization
curl -s http://localhost:8080/_status/vars | grep sql_mem_root_current

# Quick goroutine profile (first line shows count)
curl -s http://localhost:8080/debug/pprof/goroutine?debug=1 | head -1

How to diagnose

1. Establish the baseline. Check sql_conns during normal operation. A well-pooled deployment typically shows 50-300 connections per node. Anything consistently above 1000 suggests a pooling problem.

2. Check for monotonic growth. If sql_conns grows over hours and never drops during low-traffic periods, the application is opening connections without closing them.

3. Correlate with failover events. If the spike coincides with a node restart, crash, or drain, all clients connected to the lost node reconnected to the survivors simultaneously.

4. Check FD headroom. Compare open FDs against the soft limit. CockroachDB recommends a production ulimit -n of at least 35,000. If you are running with the default 1024 found on many Linux distributions, FD exhaustion is your real problem.

5. Check goroutine count. Normal range is 5,000-30,000 depending on cluster size and connections. Above 100,000 warrants investigation. Use the pprof goroutine endpoint to identify where goroutines are stuck.

6. Check SQL memory budget. If sql_mem_root_current is climbing alongside sql_conns, connections are running queries that compete for the shared memory pool. Error code 53200 confirms the budget is exhausted.

7. Verify load balancer distribution. If one node has dramatically more connections than others, check whether your LB uses plain TCP health checks (which route to impaired nodes) instead of CockroachDB’s /health?ready=1 endpoint.

Metrics and signals to monitor

Fixes

Fix client-side connection pooling

The most common root cause. If sql_conns regularly exceeds 1000 per node, applications are either not pooling or their pools are oversized.

Application-side pools (HikariCP, pgxpool, SQLAlchemy): set maxLifetime to 5-30 minutes to recycle connections. A common mistake is sizing the pool for peak burst rather than steady-state throughput. Active connections compete for CPU, memory, and SQL execution resources.

External pooler (PgBouncer): transaction pooling mode is safe with CockroachDB. Session mode works but wastes connections by holding them between transactions. Statement mode breaks multi-statement transactions and should not be used.

Fix failover reconnection storms

When a node dies, all clients reconnect simultaneously. Without exponential backoff with jitter, the surviving nodes absorb the entire connection load in a burst.

Ensure client libraries use exponential backoff with jitter on reconnect. The initial reconnect attempt should be delayed by a random small interval, not immediate. This spreads the reconnection burst across seconds rather than milliseconds.

If using a load balancer, verify it uses CockroachDB’s /health?ready=1 endpoint for health checks. This endpoint returns 503 when the node is draining or has lost quorum. Plain TCP checks route to impaired nodes, making failover behavior worse.

Fix connection leaks

If sql_conns grows monotonically and never drops, the application is not closing connections. Check for:

• Missing defer rows.Close() or equivalent in application code
• Connection pool implementations that open without recycling
• Long-lived sessions from interactive tools (SQL clients, notebooks) left open

Use SHOW CLUSTER SESSIONS to identify which clients hold connections.

Warning: CANCEL SESSION <id> kills any active query on that session. Use it to clear abandoned sessions, not as routine maintenance.

Fix file descriptor limits

If FD exhaustion is the binding constraint, raise the limit. This requires a CockroachDB process restart to take effect.

# Check current limits
cat /proc/$(pgrep -x cockroach)/limits | grep "open files"

# Production recommendation: at least 35000
# Set via systemd LimitNOFILE in the [Service] section:
# LimitNOFILE=65536

See CockroachDB file descriptor exhaustion for a deeper treatment of FD limits, SSTable handle growth, and the exhaustion cascade.

Prevention

• Instrument sql_conns per node. Alert on more than 2x expected peak or steady growth. Do not page on connection count alone; page only when combined with FD exhaustion or connection rejection errors.
• Set ulimit -n to at least 35,000 on all nodes before production deployment.
• Use PgBouncer in transaction mode as a connection funnel. This decouples client connection counts from server-side connection counts.
• Verify client retry behavior under simulated node failure. If all clients reconnect instantly without backoff, you will hit connection storms on every failover.
• Monitor goroutine count alongside sql_conns. If goroutines track roughly 1:1 with connections, the growth pattern tells you whether the problem is new connections or stuck connections.
• Use /health?ready=1 for load balancer checks, not plain TCP checks.

How Netdata helps

Netdata’s CockroachDB collector captures sql_conns and sys_goroutines at per-second resolution, which matters for detecting failover storms that 15-30s scrape intervals miss. Key correlations available on a single timeline:

• sql_conns and sys_goroutines rising together confirms connection-driven goroutine pressure.
• sql_conns spiking while sql_mem_root_current follows indicates the connection flood is consuming the shared SQL memory pool.
• OS-level FD collectors alongside CockroachDB metrics make FD exhaustion visible before it becomes connection rejection errors.
• The anomaly engine flags slow sql_conns leaks that fall below static threshold alerts.

See CockroachDB monitoring with Netdata for collector setup and dashboard configuration.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats