RabbitMQ binary heap bloat: reference-counted message bodies and force_gc

A consumer fleet finishes draining a queue. Queue depth drops to zero. And the RabbitMQ node’s memory graph stays high. It sits at two or three times what the message volume should account for, sometimes for hours, while dashboards keep telling you the broker is nearly out of memory. The instinct is to call it a leak. In most cases it is not one.

What you are looking at is a property of the Erlang VM’s memory model: message bodies live in a shared, reference-counted binary heap, and that heap is reclaimed lazily. The memory is real, it is attributed to the broker, but it is reclaimable on demand. Knowing how to prove that quickly is the difference between closing an incident in five minutes and chasing a memory leak that does not exist.

What the binary heap is and why it matters

Erlang processes each have a private heap for small terms, but large binary data (anything over 64 bytes) is stored differently. It goes into a shared, reference-counted binary heap. In RabbitMQ, message bodies are exactly this kind of data. A message published to five queues is not copied five times. One binary exists in the shared heap, and five queue processes hold references to it.

Reference counting is what makes RabbitMQ memory-efficient under fanout. It is also what makes post-drain memory graphs misleading: a binary is freed only when its reference count reaches zero, and a reference is dropped only when the process holding it runs garbage collection. Erlang GC is per-process and lazy. A process that handled a message body, finished with it, and then went idle will not collect again until it has done enough new work to trigger GC. Until then, it keeps the reference, and the binary stays allocated.

The result: binary memory tracks the high-water mark of what processes have touched, not what they currently need. After a queue drains, the messages are gone, but the binaries can linger behind stale references held by idle queue, channel, and connection processes.

How the retention happens, step by step

flowchart LR
  P[Publisher] -->|message body| B[Shared binary heap]
  B -->|refcount +1| Q[Queue process]
  B -->|refcount +1| C[Channel process]
  Q -->|deliver| C
  C -->|ack, consumer done| D[Message removed from queue]
  D -->|references still held by idle processes| E[Binary NOT freed]
  E -->|process GC eventually runs| F[refcount hits 0, binary freed]
  G[rabbitmq-diagnostics force_gc] -->|forces GC on all processes| F

The sequence is always the same:

A message body enters the shared binary heap on publish.
Every process that touches it (queue, channel, connection, message store) bumps the reference count.
The consumer acks, the queue drops the message, the workload is done.
Some referencing processes are now idle and will not GC on their own for a long time.
The binary memory chart stays elevated even though no queue holds any messages.

A binary heap at 2-3x what message volume would predict after a drain is an Erlang VM characteristic, not a RabbitMQ bug. A forced GC usually brings it down.

Confirming it in production

The diagnostic path has three steps. Run them in order.

# 1. See the memory breakdown by category
rabbitmq-diagnostics memory_breakdown

Look at the binary category. If binary dominates total memory (it is not unusual to see it at 50-60% or more of reported memory on a node that recently carried heavy traffic) while queue depth and unacked counts are near zero, lazy GC is the prime suspect.

# 2. Identify which processes hold the most binary references
rabbitmqctl eval 'recon:bin_leak(10).'

recon ships with RabbitMQ. bin_leak(10) returns the top 10 processes by number of binary references they would release if GC’d. These are almost always queue, channel, or connection processes that recently handled traffic and are now idle. If the output shows thousands of releasable references spread across ordinary messaging processes, that is lazy GC, not a leak.

# 3. Force GC across all Erlang processes on the node
rabbitmq-diagnostics force_gc

Now re-check memory. Two outcomes:

Binary memory drops sharply. Confirmed: it was lazy GC. The memory was always reclaimable. Nothing to fix, though you may want to adjust alerting (see below).
Binary memory stays high. The references are live. Something is genuinely holding message bodies: unacked messages pinned in memory, a consumer hoarding prefetch, or an actual leak. Pivot to the standard memory-pressure investigation: check messages_unacknowledged, per-queue memory, and consumer health.

On older RabbitMQ versions that lack force_gc as a dedicated command, the equivalent is:

# Force GC on all processes (older versions without force_gc)
rabbitmqctl eval '[garbage_collect(P) || P <- processes()].'

Safety note: force_gc is expensive. It garbage-collects every Erlang process on the node, which causes a CPU spike and a brief latency increase for in-flight operations. Use it as a diagnostic during an investigation, not as a cron job. If you find yourself wanting to run it on a schedule, the right fix is tuning (next section), not routine forced collection.

A related check that fools people in containerized deployments: RabbitMQ’s reported mem_used and the OS-level RSS are different things. The Erlang allocator can retain freed memory, so RSS stays high even when the VM’s own accounting drops. In containers, some monitoring stacks also misattribute kernel page cache to the RabbitMQ process, inflating apparent memory further (this was notably worse on cgroups v1 with older Kubernetes versions). When interpreting “memory is high,” be explicit about which number you are reading.

When it actually is a leak

Lazy GC is the common case, but not the only one. Treat binary memory as genuinely suspicious when:

force_gc does not reduce it. Live references mean live holders. The most frequent real cause is unacked message accumulation: unacked messages are pinned in memory and cannot be paged out in classic queues, so a stuck or slow consumer holds message bodies indefinitely.
It grows monotonically with no traffic pattern behind it. Correlate with queue depth, unacked count, and connection/channel count. A binary heap growing while all of those are flat deserves a real investigation.
You are on an affected Erlang version. There have been Erlang runtime bugs that pinned binary memory (for example, a file descriptor state leak on Windows in Erlang 23.2.7, addressed in Erlang 23.3.x and 24.2.x patch releases, that caused ever-increasing binary heap). If force_gc does nothing and the growth never plateaus, check your Erlang/OTP version against known issues.

Tuning: background_gc_enabled

RabbitMQ has a configuration option, background_gc_enabled, that periodically forces GC on Erlang processes in the waiting state. It exists precisely because lazy GC makes the binary heap look bloated.

The tradeoff, as documented in the RabbitMQ configuration reference: keeping background GC enabled can reduce median RAM usage by the binary heap; disabling it can reduce latency for client operations. It was enabled by default in 3.6.5 and earlier, and disabled by default starting in 3.6.7. Operators who upgraded across that boundary saw binary heap retention increase with no other change. If your node idles at a higher memory plateau than an older deployment did, and everything else checks out, this default change is a likely explanation. The option is still present in RabbitMQ 4.x and configurable in rabbitmq.conf; it still defaults to off.

Do not reach for this toggle to make dashboards prettier. Enable it when binary heap retention is pushing you toward the memory watermark and the extra GC work is acceptable; leave it disabled when client latency matters more than headline memory.

Signals to watch in production

Signal	Why it matters	Warning sign
`binary` in `rabbitmq-diagnostics memory_breakdown`	Direct view of reference-counted message body memory	Large share of total memory with near-zero queue depth
`mem_used / mem_limit` (Management API)	How close the node is to the memory alarm	Ratio elevated but stable; paging not active
`messages_unacknowledged`	Unacked messages pin binary memory and cannot be paged	High unacked alongside high binary memory (real use, not lazy GC)
Binary memory before/after `force_gc`	The definitive lazy-GC test	No drop after force_gc means live references
`recon:bin_leak(10)` output	Which processes hold stale binary references	Thousands of releasable references on idle processes
RSS vs `mem_used`	Allocator retention makes OS memory look worse than VM reality	RSS » mem_used with stable VM accounting

The operational takeaway: alert on mem_used / mem_limit trending toward the watermark with queue depth or unacked growth behind it. Do not page on binary memory alone after a known drain event. The first is a real problem forming; the second is usually the VM being lazy.

How Netdata helps

Netdata’s RabbitMQ collector tracks mem_used against mem_limit per node, so you can see the post-drain plateau as a stable flat line rather than mistaking it for ongoing growth.
Queue depth (messages_ready and messages_unacknowledged) on the same dashboard as memory lets you spot the signature instantly: depth at zero, memory flat and high. That combination is the lazy-GC pattern.
Per-queue memory and unacked charts help you confirm, before running force_gc, that no queue is actually holding the bodies the binary heap seems to account for.
Because Netdata samples every second, you can watch the memory curve drop in real time when you run force_gc, which makes the diagnostic conclusive instead of inferred.
Correlating node memory with connection and channel counts rules out the look-alike causes (connection/channel proliferation) that also keep memory elevated after traffic subsides.