$ guides / rabbitmq / rabbitmq-channel-leak-churn ▌

Operations Guides

RabbitMQ channel leak and churn: the channel-per-message anti-pattern

Your connection count is flat and queue depths look normal, but broker memory is creeping up and the Erlang process count keeps climbing. That is the classic shape of a RabbitMQ channel leak: a slow resource drain that hides behind a stable connection count until the node runs out of processes or memory.

Channels are where all AMQP work happens: publishing, consuming, acknowledging. Each channel is a separate Erlang process on the broker. They are lightweight but not free. When a client opens a new channel per message instead of reusing long-lived channels, or recreates a channel on every exception, the broker pays for setup and teardown on every operation. If channels open faster than they close, the leak accumulates Erlang processes and memory indefinitely.

This article covers how to tell channel churn (high open and close rates, balanced) from a channel leak (opens consistently outpacing closes), how to find the offending client, and how to fix the application pattern that causes it.

What this means

RabbitMQ multiplexes many channels over a single TCP connection. The intended pattern is a small number of long-lived channels per connection: most applications can use a single-digit number of channels per connection, commonly one channel per thread. A connection with hundreds or thousands of channels, or a broker whose channel open rate never settles, signals a client-side problem.

There are two distinct failure modes, and they need different responses:

Channel churn: channels created and channels closed are both consistently high. The total channel count stays roughly stable, but the broker burns CPU and scheduler time on constant setup and teardown. Each channel open is a network round trip, so the client also slows itself down.
Channel leak: the channel creation rate is consistently higher than the close rate. The total channel count grows without bound. Each leaked channel is an Erlang process with a heap, so proc_used and memory grow until the node degrades or crashes.

flowchart TD
    A[channel_created rate high?] -->|no| B[Healthy baseline]
    A -->|yes| C{channel_closed rate
tracks created?}
    C -->|yes, total channel count stable| D[Channel churn:
channel-per-message anti-pattern]
    C -->|no, opens outpace closes| E[Channel leak:
growing channel count,
proc_used and memory climbing]
    D --> F[Find connection with
high channel count or churn]
    E --> F
    F --> G[Fix client: reuse long-lived channels,
one channel per thread]

A useful first heuristic from the field: a channels-per-connection ratio above 100 suggests channels are not being closed properly. Most healthy applications sit well below that.

Common causes

Cause	What it looks like	First thing to check
Channel-per-message anti-pattern	High `channel_created` and `channel_closed` rates tracking each other; publish throughput lower than expected	`churn_rates` in `/api/overview`; inspect publisher code for channel creation inside the publish path
Client recreates channel on every exception	Churn spikes correlate with bursts of `channel_error` log entries; closed channels replaced immediately	Broker log for channel exceptions (PRECONDITION_FAILED, ACCESS_REFUSED)
Channel leak: opened but never closed	Channel count grows monotonically; one or few connections hold hundreds or thousands of channels	`rabbitmqctl list_connections name channels`, sorted by channel count
Declaring topology per operation	Channel churn plus elevated `queue_declared` churn; client re-declares queues/exchanges on every publish	`churn_rates.queue_declared` in `/api/overview`
Consumer channel closed with unacked messages	Sudden bursts of requeued messages; elevated `redeliver` rate after channel closes	Per-queue `redeliver` rate in `message_stats`
Short-lived runtime (PHP and similar)	High connection and channel churn together; both counts cycle with request volume	Check whether connection churn is also high; if so, the problem is at the connection level, not channels

Quick checks

All of these are read-only and safe to run during an incident. Note that list_connections, list_channels, and full /api/connections iteration are expensive on nodes with very many connections; run them once, not in a tight loop.

# 1. Channel churn rates: created vs closed
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.churn_rates | {channel_created, channel_closed}'

# 2. Total channels vs total connections
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.object_totals | {connections, channels}'

# 3. Per-connection channel count: find the fattest connections
rabbitmqctl list_connections name channels

# 4. Erlang process usage: is the leak approaching the process limit?
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, proc_used, proc_total}'

# 5. Memory trend
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_used, mem_limit}'

# 6. Channel errors in the broker log (recreate-on-exception pattern)
grep -i "channel_error\|PRECONDITION_FAILED\|ACCESS_REFUSED" /var/log/rabbitmq/rabbit@$(hostname).log | tail -20

# 7. Individual channel detail for a suspicious connection
rabbitmqctl list_channels connection name number transactional confirm consumer_count

Two things to look for immediately. In check 1, if channel_created is persistently higher than channel_closed, you have a leak; if they are both high and roughly equal, you have churn. In check 3, any connection holding hundreds of channels is your primary suspect, because most applications need a single-digit number per connection.

How to diagnose it

Confirm the symptom class. Pull churn_rates.channel_created and churn_rates.channel_closed a few times over several minutes. Opens consistently above closes means a leak. Both high and balanced means churn. Balanced and low means this article is not your problem.
Check whether the connection layer is also churning. Look at churn_rates.connection_created and connection_closed. If connection churn is high too, the client is recreating whole connections, and channels are just going along for the ride. Treat it as a connection problem first (see the connection storm guide linked below). Short-lived runtimes such as PHP legitimately produce high connection and channel churn unless a proxy or pooling layer is used; that is an architecture property, not a bug.
Attribute channels to connections. Run rabbitmqctl list_connections name channels and sort by channel count. A leak almost always concentrates: one or two connections accumulate channels while everyone else looks normal. Compare the count against your expected pattern (one channel per thread is a common healthy design).
Correlate with resource signals. For a leak, check proc_used / proc_total and mem_used / mem_limit. Each channel is an Erlang process, so a leak shows up directly in process count and in per-process heap memory. This is what turns a “weird channel count” into a real incident: the default process limit is large (1,048,576), but a fast leak with a heavy per-channel footprint can exhaust memory long before the process table.
Correlate churn with errors. If channels churn in bursts, grep the broker log for channel exceptions. A common pattern is a client that hits a protocol error (for example re-declaring a queue with different arguments, which raises PRECONDITION_FAILED), lets the channel die, and opens a fresh one on the next operation. The churn rate then tracks the error rate exactly.
Check the unacked side effect. When a consumer’s channel closes, all of its unacknowledged messages are requeued at once. If your churn involves consumer channels, look at per-queue redeliver rates: elevated redelivery after channel closes means your consumers are redoing work, and the churn is costing you throughput as well as broker resources.
Identify the client. Use the connection name (many client libraries let applications set a connection_name, visible as user_provided_name in listings) and peer host to find the application instance. The fix is in the client code, not on the broker.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`churn_rates.channel_created` vs `channel_closed`	Direct measure of churn vs leak	Created > closed sustained (leak); both high and equal (churn); created > 3x rolling baseline
`object_totals.channels` vs `object_totals.connections`	Total channel footprint relative to connections	Channels/connections ratio > 100
Per-connection channel count	Attributes the problem to a specific client	A single connection holding hundreds or thousands of channels
`proc_used / proc_total`	Each channel is an Erlang process; leaks consume the process table	Ratio > 0.8 sustained
`mem_used / mem_limit`	Leaked channels hold per-process heap memory	Slow monotonic growth uncorrelated with queue depth
Per-queue `redeliver` rate	Consumer channels closing requeue all unacked messages	Redeliver spikes coinciding with channel closes
Channel errors in broker log	Explains recreate-on-exception churn	PRECONDITION_FAILED or ACCESS_REFUSED bursts matching churn spikes
`churn_rates.connection_created/closed`	Separates channel-level from connection-level churn	Both high together means the problem is connections, not channels

One sampling caveat: brief lifecycle spikes between scrapes are invisible, whether you read rates from the management API or compute them from cumulative counters. Sample over at least a minute, and treat a single high reading as a prompt to re-sample, not a conclusion.

Fixes

Fix the channel-per-message pattern

The actual fix is in the application: create channels once and reuse them for the life of the connection. The canonical healthy shape is one connection per application process and one channel per thread (or per coroutine, depending on the client). Channels are cheap to keep open and expensive to create, since every open is a network round trip.

Concretely, in the client code:

Move channel creation out of the publish/consume hot path and into initialization.
Pool channels if the workload is concurrent; borrow and return rather than open and close.
Do not share a single channel instance across threads. The major client libraries state this explicitly; concurrent use on one channel causes framing errors that themselves trigger channel closures and more churn.
On channel exception, recreate the channel deliberately (with backoff and logging), not implicitly on every retry of the failed operation.

Contain an active leak before fixing the code

If a leak is growing right now and a code fix is not yet deployed:

Restart the offending application instances, not the broker. Closing the client connection destroys all of its channels and reclaims the Erlang processes and memory on the broker side. Restarting the broker is the wrong tool: it disrupts every healthy client and does nothing to fix the client bug.
Close the leaking connection from the broker side as a surgical alternative, via the management API or UI. The client will reconnect (if it has reconnect logic) with a clean channel slate. Be aware this requeues any unacknowledged messages on that connection’s consumer channels.
Set channel_max on the broker as a guardrail. This limits the number of channels per connection; a leaking client then fails fast with a negotiation error instead of silently consuming the node. Note the failure mode moves to the client: it will start seeing channel-open failures. The broker default is channel_max = 2047 per connection, so the guardrail is on but far above any healthy pattern; set it deliberately to a smaller value (0 means unlimited, which the documentation explicitly warns is dangerous).

Tradeoff to be aware of: closing a consumer connection with a large unacked backlog requeues all of those messages at once, which produces a delivery burst. With sane prefetch values this is manageable; with prefetch in the thousands, expect a visible redelivery spike.

Fix recreate-on-exception churn

If the churn traces to channel exceptions, fix the underlying error, not the retry loop. The two common broker-log signatures are PRECONDITION_FAILED (re-declaring a queue or exchange with mismatched arguments) and ACCESS_REFUSED (permissions). Align declarations across all application instances so they use identical queue arguments, and declare topology once at startup rather than on demand.

Prevention

Adopt a channel lifecycle convention across services: one connection per process, one long-lived channel per thread, topology declared once at startup. Put it in the client wrapper library so individual services cannot easily deviate.
Alert on the ratio, not just the totals. Total channel count alerts miss leaks hidden by growing connection counts. Alert on channels-per-connection ratio (warning above a value well under 100, tuned to your baseline) and on channel_created exceeding channel_closed sustained.
Alert on churn relative to baseline. Channel creation rate above 3x the rolling baseline sustained for a few minutes catches the anti-pattern at deployment time, when it is introduced, rather than weeks later when memory runs out.
Track proc_used / proc_total and mem_used as backstops. Even if the churn alerts miss a slow leak, the process and memory ratios will eventually catch it. A monotonic memory climb uncorrelated with queue depth should always trigger a channel/connection investigation.
Set channel_max deliberately so a future leaking client fails fast instead of draining the node.
Keep redeliver rates visible. Consumer channel churn shows up as redelivery before it shows up anywhere else.

How Netdata helps

Netdata’s RabbitMQ collector surfaces channel and connection churn rates, so you can see channel_created diverging from channel_closed (leak) or both spiking together (churn) on a per-second timeline.
Object totals for connections and channels are charted together, which makes the channels-per-connection ratio visible without manual calculation.
Erlang process usage and node memory are collected alongside AMQP objects, so you can correlate a rising channel count directly with proc_used and mem_used growth and confirm the leak’s blast radius.
Message statistics including redeliver rates sit on the same dashboards, letting you tie consumer channel churn to its requeue cost.
Because churn often starts at a deployment, per-second resolution helps you match the churn onset to the exact deploy window instead of discovering it from a memory alarm days later.