RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster

Your RabbitMQ cluster just stopped accepting messages. Publishers are connected but nothing flows. The broker log shows:

memory resource limit alarm set on node rabbit@node1.
*** Publishers will be blocked until this alarm clears ***

One node crossed its memory high watermark, and RabbitMQ responded by blocking every publisher on every node in the cluster. There is no gradual degradation: the transition from “fine” to “all ingestion halted” is a single threshold crossing.

The important operational facts: the broker is alive, consumers still work, and the alarm is a deliberate circuit breaker, not a crash. Do not restart the broker. Restarting throws away the diagnostic state (unacked messages, connection states) that tells you why memory grew, and it can make things worse when queues repopulate from disk into an already tight memory budget.

What this means

RabbitMQ tracks the Erlang VM’s memory usage against a high watermark (vm_memory_high_watermark). The default is 0.4 (40% of detected RAM) on RabbitMQ 3.x and 0.6 (60%) on RabbitMQ 4.x. When mem_used reaches mem_limit on any node, that node raises mem_alarm, and the alarm propagates cluster-wide: all publishing connections on all nodes are blocked.

Connection states tell you exactly what is happening:

• running: normal
• blocking: a resource alarm is active, but this connection has not tried to publish yet
• blocked: a resource alarm is active and this connection tried to publish; it is frozen
• flow: credit-based flow control, a different mechanism entirely (per-connection, transient, surgical)

Do not confuse flow with blocked. Flow control throttles one publisher outpacing a downstream queue. A resource alarm is cluster-wide: everything stops.

Two more properties matter for triage:

• Consumer-only connections are not blocked. Deliveries continue, so consumers can drain queues and relieve the pressure. This is your recovery path.
• There is no hysteresis. The alarm clears at the same threshold where it fired. If memory hovers at the watermark, the alarm can flap, blocking and unblocking publishers in cycles. The alarm also does not cap memory growth; it only stops ingestion. The node can still OOM if consumers do not drain.

flowchart TD
  A[Consumers slow, stuck, or dead] --> B[Ready and unacked messages grow]
  B --> C[mem_used rises toward mem_limit]
  C --> D[Paging to disk on classic queues]
  D --> C
  C --> E[mem_used crosses the watermark]
  E --> F[mem_alarm = true on one node]
  F --> G[Alarm propagates cluster-wide]
  G --> H[All publishers blocked on all nodes]
  H --> I[Publish rate drops to zero]

Common causes

The most common case is unacked accumulation. Queue depth dashboards look fine because operators watch messages_ready, while delivered-but-unacknowledged messages pin memory invisibly. Unacked messages also cannot be paged to disk in classic queues, because the broker must be able to redeliver them.

Quick checks

All read-only and safe to run during the incident.

# 1. Confirm the alarm and see which node raised it
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_alarm, disk_free_alarm}'

# 2. Check how far over the limit each node is
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_used, mem_limit, ratio: (.mem_used / .mem_limit)}'

# 3. Alternative: alarm state from the CLI
rabbitmqctl eval 'rabbit_alarm:get_alarms().'

# 4. Global queue totals: is unacked the driver?
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.queue_totals'

# 5. Find the heaviest queues by memory
curl -s -u guest:guest 'http://localhost:15672/api/queues?sort=memory&sort_reverse=true' | jq '.[] | {name, messages_ready, messages_unacknowledged, consumers, memory}'

# 6. Count blocked and blocking connections
curl -s -u guest:guest http://localhost:15672/api/connections | jq 'group_by(.state) | map({state: .[0].state, count: length})'

# 7. See what is consuming memory on the node
rabbitmq-diagnostics memory_breakdown

Iterating /api/connections is expensive on deployments with many connections, and there is no aggregate blocked-connection metric. Run check 6 once for diagnosis, not in a loop.

How to diagnose it

1. Confirm the scope. Check mem_alarm on every node (check 1). Any node with mem_alarm: true blocks the whole cluster. Also check disk_free_alarm, because a disk alarm produces the same publisher-blocked symptom with a different fix.

2. Identify the memory driver. Look at global queue_totals (check 4). If messages_unacknowledged is large and growing, consumers are the bottleneck. If messages_ready dominates, consumers are absent or too slow. If both are modest but memory is high, run rabbitmq-diagnostics memory_breakdown to see whether connections, queue processes, or the binary heap are responsible.

3. Find the offending queues. Sort queues by memory (check 5). Usually one or two queues hold most of the pressure. Note their consumers count.

4. Classify the consumer problem.

   • consumers: 0: nobody is subscribed. The queue grows until you restore consumers or purge it.
   • consumers > 0, unacked high, ack rate zero: consumers are connected but stuck. Check consumer application logs, downstream dependencies (databases, APIs), and prefetch settings.
   • Unacked pinned at exactly prefetch size times consumer count: consumers fetched everything and stopped processing. Classic stuck-consumer signature.

5. Check for zombie connections. A consumer process can crash while its TCP socket stays open; the broker keeps the channel and all its unacked messages. Look for connections from hosts whose application you know is down. Closing these connections requeues their unacked messages, which is what you want.

6. Check whether the alarm is flapping. If mem_used / mem_limit oscillates around 1.0, the alarm is firing and clearing repeatedly. Pressure is marginal: small relief may clear it, but it returns until the root cause is fixed.

Metrics and signals to monitor

Alert text matters. The alarm is cluster-wide, so “node X memory high” undersells it. The page should say “cluster publishing halted.”

Fixes

Stuck or dead consumers (unacked accumulation)

Fix or restart the consumer applications, not the broker. If consumer processes crashed but their connections are still open, close the zombie connections via the management API. Tradeoff: closing a connection requeues all its unacked messages in one burst back to the head of the queue, which briefly spikes messages_ready. That is correct behavior; the messages were never processed.

If the root cause is a downstream dependency (the consumer’s database or API is down), fixing that dependency unblocks the consumers, acks resume, memory drains, and the alarm clears on its own.

No consumers at all

Redeploy or scale up the consumer fleet. As emergency relief for non-critical queues, purging the queue frees memory immediately. Purging is destructive: every message in the queue is discarded. Confirm the queue is expendable before you purge, and never purge as a first response on queues with data you cannot replay.

Watermark too low for the workload

If the alarm fires repeatedly at memory levels your workload legitimately needs, raise vm_memory_high_watermark (or set an absolute limit). Tradeoff: the watermark is also your guardrail against the OOM killer. Raising it narrows the margin between “publishers blocked” and “node killed by the kernel,” and the alarm does not prevent memory growth past the watermark. Size the limit so the broker, plus OS and anything else on the host, fits in RAM with headroom. In containers, verify RabbitMQ sees the container memory limit rather than the host’s RAM; an absolute limit is safer there.

A related technique: rabbitmqctl set_vm_memory_high_watermark 0 sets the limit to zero, which raises the alarm and blocks all publishing. Operators sometimes use this as an emergency brake during migrations or upstream floods. Treat it as a controlled outage of ingestion.

Message flood or oversized messages

Throttle the upstream producer, and check whether average message size grew; a deployment that started sending larger payloads can move memory dramatically at the same message rate. Long term, large bodies belong in a payload-offload pattern, not in the broker.

Prevention

• Track mem_used / mem_limit with a warning at 0.7. The alarm is a cliff edge; the only warning you get is the ratio trending up. Paging activity is the secondary tell on classic queues.
• Monitor unacked separately from ready. Unacked growth is the number-one precursor to memory alarms and is invisible in a total-depth chart.
• Alert on ack rate divergence. Deliver > 0 with ack = 0 means consumers are connected but dead in practice. Catch this hours before the alarm.
• Set sane prefetch counts. A small number of consumers with huge prefetch can pin enormous memory in unacked messages.
• Size the watermark deliberately. Know your steady-state memory, size the watermark above it with real headroom, and use an absolute value in containers.
• Suppress alarm pages during warm-up. After a restart, memory spikes as queues repopulate. Gate the page on uptime > 10 minutes so cold starts do not page you.

How Netdata helps

• Netdata’s RabbitMQ collector pulls the Management API and charts mem_used against mem_limit per node, so you see the ratio approaching 1.0 before the alarm fires instead of discovering it at the cliff.
• messages_ready and messages_unacknowledged are charted as separate dimensions, making the unacked-accumulation pattern visible without custom queries.
• Publish, deliver, and ack rates on one dashboard let you spot the imbalance driving memory growth, and the publish rate dropping to zero confirms the block in the same view.
• Alarm state and per-queue depth correlation shortens the path from “publishing stopped” to “this specific queue with zero acks is the cause,” which is the whole diagnosis.
• Historical retention around the incident lets you reconstruct whether this was unacked buildup, a flood, or a slow leak, which determines the fix and the prevention work.

Related guides

• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ paging messages to disk: the paging ratio as an early memory warning
• RabbitMQ monitoring checklist: the signals every production broker needs
• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ monitoring maturity model: from survival to expert
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ queue backlog growing: messages_ready climbing and how to drain it
• RabbitMQ unacknowledged messages growing: the pinned-memory leak behind most alarms