RabbitMQ unacknowledged messages growing: the pinned-memory leak behind most alarms

The queue depth chart looks fine. messages_ready is flat or low. Then the memory alarm fires and every publisher in the cluster freezes. When you dig in, the culprit is almost always the same: messages_unacknowledged had been climbing for hours, quietly pinning RAM that RabbitMQ cannot page out.

Watching total queue depth while ignoring unacknowledged messages as a separate signal is the most common RabbitMQ memory monitoring mistake. A consumer holding 10,000 unacked messages with no acks for 5 minutes is the top precursor to memory alarms, and it is invisible if you only chart messages_ready.

This article covers why unacked messages behave differently from ready messages, how to find the offending consumers, and how to fix the underlying causes without making things worse.

What this means

A message moves to the messages_unacknowledged state the moment the broker delivers it to a consumer, and stays there until the consumer acks it. Until then, the broker must be able to redeliver that message instantly if the consumer dies. For classic queues, unacknowledged messages are not candidates for paging to disk. They are pinned in RAM.

Ready messages are different. When memory pressure crosses the paging threshold (vm_memory_high_watermark_paging_ratio, default 0.5 of the watermark), classic queues page ready messages out to disk and reclaim RAM. Unacked messages get no such relief. Every unacked message contributes directly to mem_used, and nothing short of an ack or a channel close releases it.

Two consequences follow:

1. The ceiling is prefetch x active consumers. A consumer’s basic.qos prefetch count caps how many unacked messages it can hold on a channel. In a healthy system, messages_unacknowledged should never exceed prefetch multiplied by the number of active consumers. If it does, something is wrong: a dead consumer whose channel is still open, or consumers hoarding messages they are not processing.

2. A dead channel requeues everything at once. When a consumer’s channel or connection dies, all of its unacked messages requeue simultaneously. If one consumer was holding 10,000 unacked messages, they all land back on the queue in a single burst, messages_ready spikes vertically, and the memory that was pinned stays pinned because the messages go right back into RAM for redelivery.

flowchart TD
  A[Consumer receives messages] --> B{Acks sent?}
  B -- yes --> C[Memory released]
  B -- no, consumer stuck --> D[Unacked grows, pinned in RAM]
  D --> E[mem_used approaches mem_limit]
  E --> F[Memory alarm: all publishers blocked]
  B -- channel dies --> G[All unacked requeue at once]
  G --> H[messages_ready spikes, memory stays high]

Common causes

Quick checks

All read-only. Safe to run during an incident.

# Global unacked total
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.queue_totals'

# Per-queue ready, unacked, and consumer count (pipe to sort if you want worst first)
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers

# Find queues with unacked but zero consumers
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | select(.messages_unacknowledged > 0 and .consumers == 0) | {name, messages_unacknowledged}'

# Per-consumer detail: connection, channel, queue, prefetch, ack mode
rabbitmqctl list_consumers

# Ack rate vs deliver rate (divergence confirms the leak)
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.message_stats | {deliver_get, ack, redeliver}'

# How close memory is to the alarm
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, mem_used, mem_limit, mem_alarm, ratio: (.mem_used / .mem_limit)}'

The guest user only authenticates over localhost by default. From a remote host, use a dedicated monitoring user.

How to diagnose it

1. Confirm unacked is the driver. Compare queue_totals.messages_unacknowledged against queue_totals.messages_ready. If unacked dominates, you are dealing with consumer behavior, not a producer flood.
2. Locate the queues. Sort per-queue output by messages_unacknowledged. The problem is usually concentrated in one or two queues.
3. Check consumer presence and effectiveness. For each hot queue: is consumers > 0? Is consumer_utilisation low (below 0.5) while there is a backlog? Consumers attached but ineffective points at slow processing or a downstream dependency.
4. Compare against the prefetch ceiling. Compute prefetch x active consumers. If unacked equals or exceeds it, consumers are at capacity and not acking. If unacked exceeds it, suspect a dead consumer whose channel never closed.
5. Check the ack/deliver divergence. Deliver rate positive with ack rate zero means messages are flowing out and never coming back. This is the stuck-consumer signature.
6. Check the redeliver rate. High redeliver with stable unacked is a poison message loop, not a stuck consumer.
7. Inspect the consumer application. RabbitMQ cannot see why a consumer is not acking. Check consumer-side logs for exceptions, deadlocks, or a dead downstream dependency.

Metrics and signals to monitor

Fixes

Stuck or failed consumers

Fix the consumer application: resolve the downstream dependency, the deadlock, or the exception swallowing acks. If a consumer process is dead but its TCP connection is still open (zombie), closing the connection via the management API forces the broker to requeue its unacked messages so live consumers can pick them up. This requeue happens all at once and can spike messages_ready; expect a burst of redeliveries.

Prefetch too high

Lower the consumer basic.qos prefetch. High prefetch lets one consumer hoard thousands of messages in RAM while others idle. Right-size it so a consumer holds only what it can process in a few seconds. Per-consumer prefetch is the correct scope; global QoS prefetch is deprecated.

Genuinely slow processing

If consumers are acking but too slowly, the fix is consumer capacity: more consumer instances, faster processing, or both. Adding consumers only helps if the bottleneck is not downstream of them.

Emergency relief

Do not restart the broker as a first move. Restarting loses in-memory state and can make recovery worse. If the memory alarm is already firing and you need immediate relief, closing a zombie connection releases its pinned messages back to the queue, where paging can once again move them to disk. Purging a queue also frees memory, but purge is destructive: it permanently deletes every ready message on that queue. Only purge a queue whose messages you can afford to lose, and only as a last resort.

Prevention

• Alert on unacked growth separately from ready growth. Rate-of-change alerts on messages_unacknowledged, not just absolute queue depth.
• Alert on the ack/deliver divergence. Deliver positive with ack zero for more than a few minutes is a stuck consumer, every time.
• Cap prefetch deliberately. Treat prefetch as a memory budget: prefetch x consumers x average message size is the RAM you are committing to pinned unacked messages.
• Use the delivery acknowledgement timeout. Since RabbitMQ 3.8, consumers that hold a message unacked for 30 minutes (default) have their channel closed and messages requeued. This bounds how long a stuck consumer can pin memory. Values below 5 minutes are not recommended. Since RabbitMQ 4.3 the timeout applies only to quorum queues.
• Watch consumer utilisation, not just consumer count. Connected consumers are not working consumers.

How Netdata helps

• Netdata charts messages_ready and messages_unacknowledged as separate per-queue series, so the pinned-memory leak is visible instead of hidden inside total depth.
• Publish, deliver_get, ack, and redeliver rates on the same dashboard make the deliver-vs-ack divergence a one-glance correlation.
• Per-queue consumer counts next to unacked counts surface the zombie-consumer pattern: unacked > 0, consumers = 0.
• Node-level mem_used / mem_limit alongside queue totals shows unacked growth translating directly into alarm risk before the alarm fires.
• Per-second collection catches the vertical requeue spike when a dead consumer’s channel closes and thousands of unacked messages land back on the queue at once.

Related guides

• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly
• RabbitMQ paging messages to disk: the paging ratio as an early memory warning
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ monitoring maturity model: from survival to expert