CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit

When CockroachDB hits its file descriptor limit, multiple subsystems fail at once. New SQL client connections are refused. SSTable file opens fail with I/O errors. Inter-node gRPC connections fail, destabilizing Raft consensus. The node may crash, stall, or refuse to start. In the logs you will see “too many open files” errors and, if the limit is below the startup minimum, the message “hard open file descriptor limit of X is under the minimum required Y”.

The symptoms look like network problems (connections refused), disk problems (I/O errors), or database internals (crashes, instability). The actual cause is a shared process-level resource budget that four independent subsystems compete for simultaneously: SSTable handles, client connections, inter-node gRPC, and WAL files.

What this means

CockroachDB opens file descriptors for four categories of I/O:

1. SSTable handles. Pebble opens SSTable files during reads, memtable flushes, and compaction. Each open SSTable file consumes one FD. A store with many L0 files and active compaction can hold thousands of handles open simultaneously.
2. Client connections. Each pgwire (SQL) client connection consumes one FD. Without client-side connection pooling, a burst of application connections directly inflates FD usage.
3. Inter-node gRPC connections. CockroachDB maintains gRPC connections to every other node for Raft heartbeats, log replication, DistSQL data transfer, and internal RPCs. Each connection pair consumes FDs on both endpoints.
4. WAL files. Write-Ahead Log files consume FDs during active writes.

At startup, CockroachDB reads the process hard RLIMIT_NOFILE value and raises its soft limit to match. It then divides the budget: approximately 10,000 FDs per store and 5,000 for networking. For a 3-store node, that means a hard limit of 35,000. If the limit is too small to satisfy this allocation, CockroachDB allocates 256 FDs for networking and splits the remainder across stores. If the total is below 1,956 for a single-store node (1,700 per store plus 256 for networking), CockroachDB refuses to start.

The critical detail: CockroachDB uses the hard limit, not the soft limit. Setting only the soft limit has no effect. You must raise the hard limit (ulimit -Hn or equivalent).

flowchart TD
    LIMIT["Process FD limit
(hard RLIMIT_NOFILE)"]
    LIMIT --> SST["SSTable handles"]
    LIMIT --> CONN["Client connections"]
    LIMIT --> RPC["Inter-node gRPC"]
    LIMIT --> WAL["WAL files"]

    SST --> EXH["FD budget exhausted"]
    CONN --> EXH
    RPC --> EXH
    WAL --> EXH

    EXH --> FAIL["Reads fail, connections refused,
Raft liveness loss, cluster instability"]

When the FD budget is exhausted during runtime, every consumer fails. Pebble cannot open SSTables, so reads fail and compaction stalls. New SQL connections are refused. Inter-node gRPC fails, which can cascade into Raft liveness loss and range unavailability. The node may not crash immediately, but it becomes functionally non-responsive while the FD limit is saturated.

Common causes

Quick checks

All commands are read-only and safe to run at any time.

# Check current process FD count against limits
ls /proc/$(pgrep -x cockroach)/fd | wc -l
cat /proc/$(pgrep -x cockroach)/limits | grep "open files"

# Check both soft and hard limits (run as the cockroach process owner)
ulimit -Sn
ulimit -Hn

# Check CockroachDB FD metrics if exposed by your version
curl -s http://localhost:8080/_status/vars | grep -E 'sys_fd'

# Check active SQL connections (each consumes one FD)
curl -s http://localhost:8080/_status/vars | grep sql_conns

# Check L0 file count as a proxy for SSTable FD pressure
curl -s http://localhost:8080/_status/vars | grep storage_l0_num_files

# Check system-wide FD limit (must be at least 10x per-process limit)
cat /proc/sys/fs/file-max

How to diagnose it

1. Confirm FD exhaustion is the root cause. Check ls /proc/$(pgrep -x cockroach)/fd | wc -l against the limit in /proc/$(pgrep -x cockroach)/limits. If the count is at or near the “open files” soft limit, FD exhaustion is confirmed. If you see “too many open files” in the CockroachDB logs, this is the same problem.

2. Identify the dominant FD consumer. Connection-driven exhaustion correlates with sql_conns spikes. SSTable-driven exhaustion correlates with storage_l0_num_files growth. Compaction-driven exhaustion is transient and spikes during flush or compaction jobs. Check which metric moved before FD exhaustion hit.

3. Verify the limit is adequate for your topology. A single-store node needs at least 15,000 (10,000 per store + 5,000 networking). A 3-store node needs 35,000. Production deployments should run 35,000-65,000 depending on store count.

4. Check whether the limit is set where CockroachDB reads it. For systemd-managed deployments, PAM limits in /etc/security/limits.conf do not apply. The limit must be set via LimitNOFILE in the service unit. This is the most common misconfiguration: operators set the limit in the wrong place, the cockroach process inherits the default (often 1,024 or 4,096), and the node hits exhaustion under load.

5. Distinguish acute from chronic. A connection storm after failover is acute: it resolves when clients back off. SSTable accumulation from compaction debt is chronic: it worsens over days until the node hits the wall.

Metrics and signals to monitor

The sys_fd_open and sys_fd_softlimit gauges are standard runtime metrics (sys.fd.open, sys.fd.softlimit), exported with the other process runtime metrics at /_status/vars. The DB Console Storage Dashboard includes a File Descriptors graph comparing open count against limit per node. For automated monitoring, OS-level /proc/<pid>/fd counting is equally reliable and works across versions.

Fixes

Raise the FD limit

The primary fix. CockroachDB reads the hard RLIMIT_NOFILE at startup and raises its soft limit to match.

For systemd-managed deployments, set LimitNOFILE in the [Service] section of the cockroach service unit:

[Service]
LimitNOFILE=35000

Then reload and restart: systemctl daemon-reload && systemctl restart cockroach. This restart causes a brief node outage. PAM limits in /etc/security/limits.conf do not apply to systemd services.

For non-systemd deployments, set both soft and hard limits before starting the process:

ulimit -n 35000

Or configure them persistently in /etc/security/limits.conf and ensure the process is started from a PAM session that applies them.

Also set the system-wide limit: /proc/sys/fs/file-max should be at least 10x the per-process limit. For a per-process limit of 35,000, set file-max to at least 350,000.

This change requires a process restart to take effect.

Address Pebble compaction FD spikes

Pebble compaction can temporarily open a large number of SSTable handles simultaneously. A community-documented case shows a node failing because a compaction job tried to open nearly 15,000 L0 SSTable files at once, hitting “too many open files” despite the system-wide limit being set extremely high. The per-process ulimit was the bottleneck, not file-max.

If your FD limit is adequate for steady-state reads and connections but the node still fails during compaction, the limit is too low for burst conditions. Raise it. There is no cluster setting to cap per-store FD usage. The allocation is determined entirely by the process FD limit.

The OPTIONS file in the store directory contains a max_open_files setting, but editing it manually has no effect. CockroachDB regenerates this file on each start. Do not attempt to tune FD allocation through the OPTIONS file.

Reduce connection pressure

If sql_conns is the dominant FD consumer, the problem is client-side. Ensure applications use connection pooling. PgBouncer in transaction mode is safe with CockroachDB. Without pooling, every application instance opens its own connections, and a failover event causes all clients to reconnect simultaneously to surviving nodes, spiking FD usage.

After a node failure, surviving nodes absorb the dead node’s connections. If those nodes were already running at 60-70% of their FD limit, the failover pushes them past 100%. Connection pooling with exponential backoff and jitter prevents this cascade.

Reduce SSTable count

If compaction debt is driving SSTable accumulation, the root cause is disk I/O capacity, not FD limits. Address the compaction backlog first. Monitor storage_l0_sublevels and compaction throughput. If L0 sublevels are sustained above 10, compaction is falling behind and SSTable file count will continue growing, consuming more FDs.

See the related guides on L0 sublevels and compaction death spirals for detailed remediation.

Prevention

• Set hard RLIMIT_NOFILE to 35,000 or higher. CockroachDB reads the hard limit at startup and divides it across stores and networking. Each store gets its own 10,000 FD allocation from the shared pool. A 3-store node needs 35,000, not 15,000.
• Use LimitNOFILE for systemd deployments. PAM limits.conf does not apply to systemd services. This is the most common cause of under-configured FD limits.
• Set /proc/sys/fs/file-max to at least 10x the per-process limit. The system-wide limit must accommodate all processes including monitoring agents and SSH sessions.
• Monitor FD usage at 80% of the soft limit. This threshold gives lead time before exhaustion.
• Ensure client-side connection pooling with backoff. PgBouncer in transaction mode prevents connection storms from consuming FDs on surviving nodes during failover.
• Track L0 sublevels and SSTable file count. Compaction debt silently inflates FD usage as SSTables accumulate in L0.
• Verify per-process limits on co-located nodes. When running multiple CockroachDB nodes on a single machine, do not rely on the DB Console File Descriptors graph for per-process headroom: each co-located process reports the same machine-level limit value, so per-node lines on the graph do not show the combined pressure on the host. Verify the per-process limit and FD count directly with /proc/<pid>/limits and /proc/<pid>/fd.

How Netdata helps

• Netdata collects per-process FD usage from /proc/<pid>/fd at per-second resolution, catching transient spikes during compaction bursts that 15-30 second scrape intervals miss.
• Correlating FD count with sql_conns distinguishes connection-driven exhaustion from SSTable-driven exhaustion in seconds rather than minutes.
• Correlating with storage_l0_sublevels and storage_l0_num_files identifies compaction-driven FD pressure before the node hits the wall.
• ML anomaly detection flags unusual FD growth patterns before the 80% threshold fires, giving lead time during slow SSTable accumulation.
• Per-node dashboards reveal FD asymmetry across the cluster, showing which node is approaching its limit while others have headroom.

See CockroachDB monitoring with Netdata for integrated dashboards covering these signals.

Related guides

• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles
• CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth
• CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls
• CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls
• CockroachDB monitoring checklist: the signals every production cluster needs