RabbitMQ monitoring maturity model: from survival to expert

Most RabbitMQ monitoring setups fail in one of two ways. Either they watch almost nothing and discover outages from user reports, or they chart everything and page on noise. The difference is rarely tooling. It is knowing which signals actually predict the failure modes RabbitMQ is prone to: the memory wall, the disk halt, consumer black holes, split-brain, and silent message loss.

This guide organizes RabbitMQ signals into four maturity levels. Level 1 tells you the broker is on fire. Level 2 tells you where the fire is. Level 3 gives you leading indicators so you see the fire before it starts. Level 4 explains the internals well enough to debug the incidents that reach your third or fourth postmortem.

The model assumes RabbitMQ 3.13 or newer with quorum queues as the default queue type. Classic mirrored queues were removed in 4.0 and are not covered. Where behavior differs between 3.x and 4.x, it is called out inline. The default memory watermark is 0.4 on 3.x and 0.6 on 4.x, which is why the memory-ratio bands below read differently across series.

How to use this model

Do not treat the levels as a checklist to complete in one sprint. Treat them as a diagnostic: find the highest level where every signal is collected, alerted on with sane thresholds, and understood by the on-call team. A team with Level 3 dashboards but no runbook for messages_unacknowledged growth is operationally at Level 1.

Two principles apply across all levels:

• Ratio signals beat absolute values. Queue depth, connection count, and memory in bytes are workload-dependent. mem_used / mem_limit, fd_used / fd_total, and publish-versus-deliver imbalance travel across environments.
• RabbitMQ failure modes are cliff-edge, not gradual. The memory and disk alarms do not degrade throughput; they stop all publishers cluster-wide the instant a threshold is crossed. Your monitoring must catch the approach, because there is no graceful degradation to observe at the threshold itself.

flowchart TD
  L1["Level 1: Survival
node alive, alarms, total depth, connections"]
  L2["Level 2: Operational
per-queue depth, consumers, rates, ratios, partitions"]
  L3["Level 3: Mature
memory breakdown, consumer utilisation, head age, churn, quorum leaders"]
  L4["Level 4: Expert
Erlang GC, Mnesia, dist-port queue, binary heap, redelivery"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The minimum to know whether RabbitMQ is alive and not on fire. Every signal here is binary or nearly binary, and every one maps to a page-level incident.

# Level 1 spot check
rabbitmq-diagnostics -q ping
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, running, mem_alarm, disk_free_alarm}'
curl -s -u guest:guest http://localhost:15672/api/overview | jq '{queue_totals, connections: .object_totals.connections}'

Alerting notes that matter even at this level:

• Require sustained failure. A single failed probe can be a heavy GC pause. Page only after the condition survives more than two collection intervals, and suppress alerts during warm-up (uptime under 5 minutes) because memory spikes while queues repopulate after a restart.
• The alarms are cluster-wide. One node’s mem_alarm blocks publishers on every node. Alert text should say “cluster publishing halted,” not “node X memory high.”
• Do not use GET /api/health/checks/is-in-service as a liveness check. It intentionally returns 503 on nodes paused or drained for maintenance. It is a readiness and drain signal, not a dead-node signal.
• Check the default disk_free_limit. The default is 50MB, which is dangerously small on any modern server. Routine log rotation can trigger a cluster-wide halt. Set disk_free_limit.relative to 1.0-2.0 (1-2x RAM) or an absolute value of 2GB or more.

Level 2: operational

What a competent team running RabbitMQ in production monitors. This level moves from “is it up” to “is message flow healthy,” and it is where per-queue visibility starts.

Everything in Level 1, plus:

Three distinctions separate Level 2 teams from Level 1 teams:

1. Unacknowledged messages are tracked separately from ready messages. Unacked messages are pinned in memory and cannot be paged to disk. A consumer holding thousands of unacked messages with a zero ack rate is the single most common precursor to a memory alarm.
2. deliver_get, not deliver, is compared against publish. Pull-mode consumers using basic.get appear only in get and deliver_get. Comparing publish to deliver alone manufactures phantom imbalances.
3. Flow and blocked are not the same alert. state: flow is per-connection, credit-based backpressure: surgical, transient, and normal in bursts. Alert only when more than 10% of connections sit in flow for over 5 minutes. state: blocked or blocking means a resource alarm is active cluster-wide and is already covered by the Level 1 alarm signals.

Threshold guidance: alert on the memory ratio above 0.7 sustained, the FD and socket ratios above 0.8, and publish-versus-deliver_get imbalance exceeding 2x for 10 or more minutes relative to a rolling baseline.

Level 3: mature

Full operational coverage: internals, leading indicators, and the composite patterns that cause real incidents. This is the level where monitoring starts predicting instead of reacting.

Everything in Level 2, plus:

The patterns this level exists to catch:

• Consumer black hole: consumers connected, deliver rate near zero or positive, ack rate zero, ready depth rising. Connection health is not consumer health.
• Poison message loop: stable depth, consumers present, elevated redeliver rate, head message age climbing. Fix with a delivery limit (x-delivery-limit on quorum queues) so the message dead-letters instead of looping.
• Silent routing loss: publish rate healthy, deliver_get zero, depth flat. If publishers do not set mandatory=true, unroutable messages are discarded silently and return_unroutable never moves. The depth-and-deliver composite is the only broker-side signal.
• False partitions: GC pauses exceeding net_ticktime (default 60s) can produce transient partition detection. This is why partition alerts require a sustained condition, and why run queue and partitions belong on the same dashboard.

Head age caveats: it is computed from the publisher-set timestamp message property, so it is absent or zero when publishers do not set timestamps. The Prometheus per-object endpoint exposes rabbitmq_detailed_queue_head_message_timestamp; age itself is a computed value (now minus timestamp), not a direct metric. Treat age as workload-relative: a queue with 1,000 one-second-old messages is healthy, and 100 two-hour-old messages is an incident.

Level 4: expert

The deep, frequently missed signals that experienced operators add after their third or fourth major incident. Most require going beyond the Management API: Erlang eval, the Prometheus endpoint, or OS tooling.

Everything in Level 3, plus:

• Erlang process ratio (proc_used / proc_total). Each connection, channel, and queue is an Erlang process. Exhaustion crashes the node. Alert above 0.8.
• Erlang GC rate and bytes reclaimed (rabbitmqctl eval 'erlang:statistics(garbage_collection).'). Sustained GC pressure is a latency source and keeps reported memory elevated. Use rabbitmq-diagnostics force_gc diagnostically: if memory drops afterward, the growth was lazy GC, not a leak. Warning: force_gc triggers garbage collection across processes and can cause a noticeable CPU spike; do not run it on a production node during peak traffic.
• Binary heap memory (rabbitmqctl eval 'erlang:memory(binary).'). Message bodies live in the reference-counted binary heap, which can stay bloated after queues drain because the last referencing process has not GC’d yet. This explains “memory stable, queues empty” confusion. In Prometheus per-object metrics, related process-level values appear as erlang_vm_dist_proc_heap_size_words and erlang_vm_dist_proc_min_bin_vheap_size_words.
• Mnesia table sizes and transaction pressure (rabbitmqctl eval '[{T, mnesia:table_info(T, size)} || T <- mnesia:system_info(tables)].'). Mnesia holds cluster metadata, not messages. Past roughly 100,000 queues, declaration operations and node rejoin sync become slow. This eval applies to 3.x and to 4.0–4.2 clusters that still use Mnesia. Khepri, the Raft-based successor metadata store, became the default for new clusters in 4.2 and is the only metadata store since 4.3 (Mnesia was removed), so on current 4.x watch Raft metrics (rabbitmq_raft_*) instead.
• Distribution port send queue (ss -tnp on port 25672). All inter-node traffic (metadata replication, Raft, internal RPC) shares one TCP connection per node pair. A growing send queue is the earliest sign of cluster-link saturation, which otherwise surfaces as confusing metadata timeouts and false partitions. The Prometheus plugin exposes this as erlang_vm_dist_port_queue_size_bytes and erlang_vm_dist_node_queue_size_bytes.
• Per-channel and global redelivery counters (rabbitmq_global_messages_redelivered_total). Prefer the global counters over per-connection or per-channel aggregates; older dashboards that sum per-connection counters produce nonsensical rate spikes after connection churn.
• Raft health for quorum queues (rabbitmq_raft_* metrics such as rabbitmq_raft_commit_latency_seconds, rabbitmq_raft_commit_index, and rabbitmq_raft_wal_files). Raft metric names were renamed in RabbitMQ 4.2 (for example rabbitmq_raft_log_commit_index became rabbitmq_raft_commit_index), so verify names against your version before writing alerts.
• Publisher confirm latency. Not available broker-side; it must be measured from the publishing client. It is a 5-10 minute early warning before flow control or alarms bite.

Collection notes for this level:

• The Prometheus plugin on port 15692 is the right transport for most Level 3 and 4 metrics. Use the aggregated /metrics endpoint by default; /metrics/per-object is computationally expensive and should be reserved for the queues you actually need per-entity data on.
• Management API polling more frequently than every 5-10 seconds can itself load the broker. Keep per-object scrapes at 30-second intervals in production.
• The management plugin’s internal stats collection can be disabled (management_agent.disable_metrics_collector = true) without affecting the Prometheus plugin, which is useful on very large deployments.

How Netdata helps

The signals in this model are most useful when correlated, and correlation is where manual curl polling falls apart during an incident.

• Netdata’s RabbitMQ collector pulls the Management API signals behind Levels 1 through 3 (node alarms, memory and FD ratios, queue totals, message rates, churn, per-queue depth and consumers) at per-second resolution, so cliff-edge transitions are visible instead of averaged away.
• Queue depth, unacknowledged count, and ack rate on one dashboard makes the consumer black hole pattern (consumers present, ack rate zero, depth rising) readable at a glance.
• Memory ratio alongside paged_out counts separates genuine memory growth from paging activity, which is the leading indicator before mem_alarm fires.
• Churn counters next to connection and FD usage expose the “stable count, massive churn” leak that a point-in-time connection count hides.
• Uptime tracking lets you gate memory and rate alerts during post-restart warm-up instead of tuning them down globally.
• Anomaly detection on publish and deliver rates catches the publish-versus-deliver imbalance and silent-routing-loss composites without hand-tuned per-queue thresholds.

Related guides

• RabbitMQ monitoring checklist: the signals every production broker needs
• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ paging messages to disk: the paging ratio as an early memory warning
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ queue backlog growing: messages_ready climbing and how to drain it
• RabbitMQ unacknowledged messages growing: the pinned-memory leak behind most alarms