RabbitMQ Erlang process exhaustion: proc_used near the process limit

Your monitoring shows proc_used climbing toward proc_total on a RabbitMQ node. The default limit is 1,048,576 Erlang processes, and when the count hits the ceiling, the BEAM VM terminates with a system_limit error. There is no graceful degradation: the node crashes, taking every connection, channel, and non-replicated queue on it down.

Process exhaustion is almost never the root cause. Every connection, channel, and queue in RabbitMQ is at least one Erlang process, so when the process table fills, something upstream is creating these objects faster than they are being destroyed. The usual suspects are channel-per-message publishers, leaked temporary queues (auto-delete or exclusive queues whose owning connections never cleanly close), and connection proliferation. Raising the +P limit buys time; it does not fix anything.

This guide covers how to confirm the symptom, identify which object type is driving the growth, and stop the leak.

What this means

The Erlang VM maintains a fixed-size process table, set at VM boot by the +P flag (RabbitMQ exposes this as RABBITMQ_MAX_NUMBER_OF_PROCESSES). The default is about 1 million. Every AMQP object maps onto processes:

• One process per TCP connection, handling framing, heartbeats, and channel multiplexing.
• One process per channel. A single connection with 100 channels spawns 101 processes.
• One or more processes per queue. Classic queues are a single process; quorum queues run a Raft state machine process per member.
• Internal processes for message stores, indices, Mnesia, management stats, and plugins.

Process count grows linearly with connections + channels + queues plus internal overhead. A healthy node with 500 connections, 5 channels each, and 200 queues sits around 3,000 processes, which is 0.3% of the default limit. If you are anywhere near the limit, something has gone wrong by orders of magnitude.

flowchart TD
  A[proc_used climbing toward proc_total] --> B{Which object count is growing?}
  B --> C[Channels growing: channel-per-message anti-pattern or channel leak]
  B --> D[Queues growing: leaked auto-delete or exclusive queues]
  B --> E[Connections growing: connection storm or missing pooling]
  C --> F[Fix client code: reuse channels, close them after use]
  D --> G[Fix lifecycle: ensure owning connections close cleanly]
  E --> H[Fix reconnect logic: backoff, pooling, one connection per instance]
  F --> I[proc_used stabilises]
  G --> I
  H --> I
  A -.stop-gap only.-> J[Raise +P via RABBITMQ_MAX_NUMBER_OF_PROCESSES]

Common causes

Quick checks

All of these are read-only and safe to run on a production node. The examples use the default guest credentials, which RabbitMQ only accepts from localhost.

# 1. Current process usage vs limit
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, proc_used, proc_total, ratio: (.proc_used / .proc_total)}'

# 2. Object totals: which of connections, channels, queues is large?
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.object_totals'

# 3. Churn rates: created vs closed/deleted tells you leak vs balanced churn
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.churn_rates'

# 4. CLI view of process status
rabbitmq-diagnostics status | grep -A3 -i process

# 5. Confirm the VM-level view directly
rabbitmqctl eval 'erlang:system_info(process_count).'
rabbitmqctl eval 'erlang:system_info(process_limit).'

Interpretation shortcuts:

• Channels » connections: channels are the problem. Healthy ratios are typically 1-20 channels per connection. A ratio above 100, or one that grows week over week, means channels are not being closed.
• Queues in the tens of thousands or more: queue proliferation. Also check Mnesia pressure; huge queue counts make node restarts and management API calls slow.
• Connections growing: check FD usage too, since connections consume both. See the related guide on file descriptor exhaustion.

How to diagnose it

1. Confirm the trend, not just the level. Sample proc_used a few times over 10-15 minutes. A stable high value after a known big workload is different from a monotonic climb. Only a growing value is an emergency.

2. Identify the object class. Compare object_totals connections, channels, and queues against your known baseline. The one that has diverged is your leak source. Correlate with churn rates: created persistently exceeding closed/deleted for 30+ minutes confirms a leak rather than bursty-but-balanced churn.

3. If channels: find the offending connections. Run rabbitmqctl list_channels and aggregate by connection. One connection holding thousands of channels points at a specific client instance. The management API also shows channel counts per connection under GET /api/connections. Listing channels is expensive on a busy node; do it once or twice, not in a tight loop.

4. If queues: find what is creating them. List queues and look for naming patterns: server-generated names, reply-to queues, per-request or per-session queues. Check whether they are auto-delete or exclusive and whether their owning connections still exist. Temporary queues are cleaned up when the owning connection closes; if connections die ungracefully, cleanup can lag or fail.

5. If connections: find the source peers. Run rabbitmqctl list_connections peer_host user_provided_name and aggregate by peer_host. A single application instance opening thousands of connections is a pooling bug. Many instances each opening a few connections per minute is a reconnect-loop problem; see RabbitMQ connection storm.

6. If none of the object counts explain it: inspect the process list directly. The top processes by heap often reveal the culprit subsystem:

# Top 20 processes by heap size (read-only, but non-trivial on a huge process table)
rabbitmqctl eval 'lists:sublist(lists:reverse(lists:keysort(2, [{P,element(2,erlang:process_info(P,total_heap_size))} || P <- erlang:processes()])), 20).'

On a node with hundreds of thousands of processes, iterating the full process table is itself heavy. Run this once, during off-peak if possible.

7. Check what the processes are doing while you plan the fix. High memory per process, or a single internal process with a huge mailbox, means a secondary bottleneck may trigger memory alarms before the process limit is actually hit.

Metrics and signals to monitor

proc_used does not always drop immediately after the offending connections or channels close: channel and queue termination is asynchronous, and memory release lags further. Do not conclude the fix failed because the number did not fall in the first minute.

Fixes

Stop the leak in application code

This is the only real fix. The patterns to correct:

• Reuse channels. A channel per message, or a channel per thread that is never closed, is the most common cause. Channels are lightweight but not free; each is a process. See the channel leak and churn guide for the full anti-pattern breakdown.
• One connection per application instance, pooled. Not one per operation, not one per thread.
• Clean up temporary queues. If you use reply-queue or per-session queue patterns, make sure queues are auto-delete or exclusive where appropriate, and that connections close cleanly so the broker can reap them.

Tradeoff: application fixes take a release cycle. That is why the stop-gap below exists.

Raise the process limit (stop-gap only)

RabbitMQ exposes the Erlang +P flag via the RABBITMQ_MAX_NUMBER_OF_PROCESSES environment variable (or directly through RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS). The official runtime tuning guide suggests adjusting it only for environments with genuinely high concurrency, such as hundreds of thousands of concurrent connections or queues.

Be honest about what this does: it moves the cliff. Each additional process costs memory for its heap and metadata, so a higher limit also raises the memory floor. If the leak is still running, you will hit the new ceiling later, after a longer, more memory-pressured decline that may trip memory alarms first. A limit change also requires a node restart, which means a connection storm on rejoin; plan for that.

Emergency relief during an active incident

• Close offending connections from the broker side. The management API allows force-closing individual connections. Closing a connection destroys its channels and triggers cleanup of its exclusive and auto-delete queues, reclaiming processes in one shot. This is disruptive to the affected client; identify the right connections first.
• Delete leaked queues. Deleting an orphaned queue destroys its process. Verify the queue is genuinely orphaned (no consumers, no publishes, transient) before deleting.
• Do not restart the node as a first move. A restart clears the process table but every durable queue reloads, every client reconnects at once, and if the leaking application is still running, the table starts filling again immediately. Restart only after the leak source is identified and stopped or isolated.

Prevention

• Alert on the ratio, not the crash. Alert at proc_used / proc_total > 0.8 sustained. The VM gives you no warning at the limit itself; the first symptom of crossing it is a dead node.
• Alert on the leading indicators. Queue created-minus-deleted trend positive for 30+ minutes, channel churn gap, and channels-per-connection ratio drift all fire days or weeks before the process limit does.
• Baseline object counts. Know your normal connection, channel, and queue counts. Process exhaustion creeps; you catch it by noticing that channels went from 5,000 to 60,000 over a month, not by watching the process table.
• Load-test client libraries before rollout. Channel-per-message and missing connection pooling show up instantly in churn rates under load, long before production traffic amplifies them.
• Keep queue counts sane. Beyond processes, very large queue counts slow node restarts (index loading and Mnesia sync) and the management API. If your architecture genuinely needs hundreds of thousands of queues, treat +P, memory, and restart time as a designed capacity question, not an accident.

How Netdata helps

• Netdata’s RabbitMQ collector charts proc_used against proc_total, so the exhaustion ratio is visible as a trend, not a single alarming sample.
• Object totals for connections, channels, and queues are collected alongside process usage, letting you see which object class is driving the growth without logging into the node.
• Churn rates (created vs closed/deleted) are charted per object type, which separates a genuine leak from balanced churn during deployments.
• Correlating process growth against memory usage shows whether process heaps will trigger the memory alarm before the process limit is reached, which changes which incident you are actually fighting.
• Per-second collection catches the steep ramps of connection storms that coarser management polling smooths over.

Related guides

• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic
• RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse