RabbitMQ memory usage climbing toward the watermark: mem_used vs mem_limit

The memory alarm has not fired yet. Publishers are still running. But mem_used has been creeping up for hours and mem_used / mem_limit just crossed 0.7. This is the last quiet window you get: RabbitMQ’s memory failure mode is a cliff edge. When mem_used reaches mem_limit, the alarm fires and every publisher on every node in the cluster is blocked at once.

Two mistakes are common here. The first is treating mem_limit as raw RAM you should multiply by the watermark again. It is not; the watermark is already baked in. The second is dismissing a rising ratio because the alarm has not fired. By the time it fires, you have zero recovery headroom.

What this means

mem_used is the memory the node reports as in use. mem_limit is the effective limit computed from total RAM times vm_memory_high_watermark. The default watermark is 0.4 (40% of RAM) on RabbitMQ 3.x and 0.6 (60%) on 4.x. Because mem_limit already incorporates the watermark, the only number you need to watch is:

ratio = mem_used / mem_limit

• ratio < 0.5: comfortable headroom.
• 0.5 to 0.7: concerning. Paging is active or imminent.
• ratio > 0.7 sustained: critical. Investigate now, before the alarm.
• ratio = 1.0: memory alarm fires, publishers blocked cluster-wide.

Paging is the intermediate stage. On RabbitMQ 3.x, at vm_memory_high_watermark_paging_ratio (default 0.5, meaning 50% of the watermark), RabbitMQ starts moving queue contents from RAM to disk. With default settings that is a ratio of 0.5. The setting is deprecated and no longer used since RabbitMQ 4.0, so on 4.x there is no paging-ratio threshold to read. Paging does not block publishers, but it adds disk I/O, slows queue processes, and can trigger per-connection flow control. It is the earliest mechanical warning that the climb is real.

flowchart LR
  A["ratio < 0.5: normal"] --> B["ratio = 0.5 (3.x): paging to disk begins"]
  B --> C["ratio > 0.7: warning, investigate"]
  C --> D["ratio = 1.0: memory alarm"]
  D --> E["all publishers blocked cluster-wide"]

Two properties of this system surprise operators:

1. The limit does not cap memory. The watermark only triggers publisher blocking. The node can still grow past it and get OOM-killed if consumers do not drain queues.
2. There is no hysteresis. The alarm clears at the same threshold it triggered. If usage hovers at the watermark, the alarm flaps, so any alert on mem_alarm needs a sustained-duration condition.

Why memory climbs

The first two rows are not the same thing. Unacked messages are the more dangerous driver: they are pinned in memory because the broker must be able to redeliver them instantly if the consumer dies. In classic queues they cannot be paged to disk. A consumer holding 10,000 unacked messages is the most common precursor to memory alarms.

Quick checks

All read-only and safe to run during an incident.

# Current ratio per node - the gauge this article is about
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, mem_used, mem_limit, ratio: (.mem_used / .mem_limit)}'

# Is the alarm already firing anywhere?
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[].mem_alarm'

# Global queue totals: ready vs unacked
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.queue_totals'

# Per-queue depth and consumer presence
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers

# Heaviest queues by memory
curl -s -u guest:guest 'http://localhost:15672/api/queues?sort=memory&sort_reverse=true' | \
  jq '.[:10] | .[] | {name, memory, messages_ready, messages_unacknowledged}'

# Category-level breakdown from the CLI
rabbitmq-diagnostics memory_breakdown

# OS-side view: is RSS far above mem_used?
grep VmRSS /proc/$(pgrep -f beam.smp | head -1)/status

The guest:guest credentials only work over localhost by default; substitute your monitoring user on remote nodes.

How to diagnose it

1. Compute the ratio and the trend. One sample tells you where you are; the slope tells you how long you have. Estimate runway: runway_minutes = (mem_limit - mem_used) / memory_growth_rate_per_minute. If the ratio is above 0.7 and the runway is under an hour, treat it as an active incident even though the alarm has not fired.

2. Split the driver: ready vs unacked. Check queue_totals. If messages_unacknowledged is the growing component, consumers are receiving but not completing. If messages_ready is growing, consumers are absent, too few, or too slow. The fix paths are different.

3. Get the memory breakdown. Query the node with ?memory=true:

curl -s -u guest:guest 'http://localhost:15672/api/nodes/rabbit@myhost?memory=true' | \
  python3 -c "
import sys,json; d=json.load(sys.stdin)
print('used:', d['mem_used']//1024//1024, 'MB')
print('limit:', d['mem_limit']//1024//1024, 'MB')
print('alarm:', d['mem_alarm'])
for k,v in sorted(d.get('memory',{}).items(), key=lambda x: -x[1]):
    print(f'  {k}: {v//1024//1024} MB')
"

The breakdown shows binary, queue process, connection, and Mnesia contributions. Match the dominant category to a cause: large binary means message bodies in flight or retained references, large queue process memory means deep queues, large connection categories mean object proliferation.

4. Check binary heap retention explicitly. If queues have drained but mem_used stayed high, inspect the binary heap:

rabbitmqctl eval 'erlang:memory(binary).'

Erlang binaries are reference counted. A process that touched a message body but has not garbage collected since pins that binary in memory. rabbitmq-diagnostics force_gc triggers GC across all processes and reclaims this if retention is the issue. Warning: it forces GC on every process at once, so expect a pause on a busy node. If memory drops and stays down, the “leak” was lazy GC. If it does not drop, the memory is genuinely in use.

5. Verify the limit itself is sane. mem_limit is calculated at startup from detected system RAM. In containers, RabbitMQ may read the host’s RAM rather than the container limit, producing a watermark far above what the cgroup will allow. Compare mem_limit against the container’s memory limit. If they disagree, the container can be OOM-killed before the alarm ever fires. Use total_memory_available_override_value or set an absolute vm_memory_high_watermark.absolute to correct the detection, then restart so mem_limit is recomputed.

6. Check whether paging is already active. Look for messages_paged_out on queues and rising disk I/O on the data partition (3.x classic queues; 4.x does not emit the per-queue field). Paged-out messages mean the ratio crossed the paging threshold at some point, and throughput is already paying for it even though publishers are not blocked.

RSS vs mem_used: do not chase the wrong number

The OS often reports more memory for the beam.smp process than RabbitMQ reports as mem_used. This is normal. The Erlang VM allocators reserve memory from the OS and do not return it eagerly after queues drain, so RSS can sit well above mem_used indefinitely. That is not a leak.

The inverse case is the dangerous one: RSS climbing while mem_used is flat, or the host swapping while RabbitMQ thinks it is fine. That means allocator overhead itself has become the problem. Alert on mem_used / mem_limit for broker behavior, but watch RSS at the OS level for OOM risk, because the OOM killer acts on RSS, not on RabbitMQ’s self-reported figure.

Metrics and signals to monitor

Bringing usage down

Do not restart the broker as a first response: restart loses in-memory state, triggers queue repopulation from disk, and memory often spikes higher during warm-up than before the restart.

Unacked accumulation. Fix the consumers. Check consumer application health, downstream dependencies, and whether a few consumers are hoarding prefetch buffers while others idle. Closing zombie connections (client process dead, socket still open) requeues their unacked messages, which converts pinned memory into pageable memory. Tradeoff: requeued messages go back to the queue head and may be redelivered to the same broken code path, so fix the consumer logic first or lower prefetch.

Ready-message backlog. Add or restore consumers if processing capacity is the gap. If no consumer exists and the data is expendable, purging non-critical queues is legitimate emergency relief. Warning: purging is destructive and irreversible. Confirm the queue’s contents are safe to lose first.

Object proliferation. Identify the source of connection or channel growth (leaks, channel-per-message anti-patterns, reconnect storms) and fix the client. See the related guides on connection storms and channel churn for the diagnostic path.

Binary retention. Use force_gc as described above, then watch whether the binary category regrows without matching traffic. If it does, a process is genuinely holding references, usually a plugin or a long-lived consumer, and the breakdown points at which.

Watermark misconfiguration. If the alarm fires under normal peak load, raise vm_memory_high_watermark or set an absolute limit so the broker can use the RAM the machine actually has. If the node routinely runs far below the watermark while other resources starve, lowering it is rarely the right move; fix the driver instead. Leave headroom for the OS and for the allocators’ tendency to retain memory above mem_used.

Container misdetection. Make RabbitMQ’s view of total RAM match the container limit, then restart so mem_limit is recomputed. Until you do, the watermark protects against a ceiling that does not exist while the real ceiling (the cgroup limit, enforced by the OOM killer) goes unmonitored.

Prevention

• Alert at ratio > 0.7 sustained, not at the alarm. The alarm is the outage; the ratio is the warning. Add a low-severity alert on any consistent upward trend over hours, even below 0.7.
• Monitor messages_unacknowledged separately from messages_ready. Total queue depth hides the unacked component, which is the one that cannot be paged and most often precedes the alarm.
• Track messages_paged_out and data-partition I/O. Stable memory with growing queue depth and rising page-outs means performance is degrading while every alarm stays silent.
• Set prefetch deliberately. Worst-case unacked memory is prefetch x consumer count x average message size. Size prefetch so a full buffer across all consumers cannot, by itself, push the node past the paging threshold.
• Suppress ratio alerts during warm-up. After a restart, memory spikes while queues repopulate and consumers reconnect. Gate the alert on uptime (for example, ignore the first 5 to 30 minutes) to avoid paging someone for normal cold-start behavior.
• Verify memory detection in containers. Confirm mem_limit reflects the container’s limit, not the host’s RAM, on every node after provisioning changes.
• Correlate depth with age. Depth alone does not tell you urgency. A queue with 1,000 messages that are seconds old is healthy; 1,000 that are hours old is a stalled consumer. Pair depth with head_message_timestamp where publishers set it.

How Netdata helps

• Netdata charts mem_used against mem_limit per node, so the ratio trend and its slope are visible at a glance rather than computed by hand during an incident.
• Alerting on the ratio crossing 0.7, with a sustained-duration condition, fires before mem_alarm and gives you the runway window this article is about.
• Queue totals (ready and unacknowledged) on the same dashboard as memory let you see which component is driving the climb without switching tools.
• Per-queue metrics, including messages_paged_out, surface the paging stage where memory pressure starts costing disk I/O and throughput before any alarm fires.
• Correlating memory with object totals (connections, channels) and churn rates distinguishes backlog-driven growth from object proliferation, which have different fixes.
• OS-level RSS and disk I/O from the same host close the gap between RabbitMQ’s self-reported mem_used and what the kernel (and the OOM killer) actually sees.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic