RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse

Your RabbitMQ dashboard shows connections that are not in running state, your alert fired, and you need to know: is the broker in trouble, or working as designed? The answer depends entirely on whether those connections show flow or blocked. The difference is not a matter of degree. They are two independent mechanisms with different scope, different causes, and different required responses.

Teams that alert on flow the same way they alert on blocked generate constant false positives: any sustained load produces flow connections, so the alert fires constantly and gets muted. Then, when a real resource alarm blocks every publisher in the cluster, the alert that should have paged someone is buried in a channel nobody reads.

What this means

RabbitMQ has two throttling systems that share nothing except that both show up in the connection state field.

Credit-based flow control is internal pipeline backpressure. RabbitMQ is a pipeline of Erlang processes: connection process, channel process, queue process, message store. Each stage grants credit to the stage upstream. When a downstream stage cannot keep up (a queue process is busy paging to disk, a persistence write is slow, a hot queue is saturated), credit runs out and the publishing connection is throttled. That connection shows state: flow. It is per-connection, surgical, and highly transient: the state can toggle many times per second. A connection in flow is publishing at a restricted rate, but it is publishing.

Resource alarms are emergency circuit breakers. When a node’s memory crosses vm_memory_high_watermark (default 0.4 of RAM in RabbitMQ 3.x, raised to 0.6 in 4.x) or free disk drops below disk_free_limit (default 50MB), the alarm fires and propagates cluster-wide. Every publishing connection on every node is frozen. Connections show state: blocked if they attempted to publish, or state: blocking if the alarm is active but they have not tried yet. Consumers are not affected: they keep draining queues, which is how the broker recovers.

flowchart TD
  P[Publisher connection] --> Q{Downstream keeping up?}
  Q -- "Yes" --> R[state: running]
  Q -- "No, one queue or store is slow" --> F[state: flow
per-connection, transient, normal]
  F --> F2[Publishing continues at reduced rate]
  N[Node memory or disk threshold breached] --> A[Resource alarm fires
propagates cluster-wide]
  A --> B[state: blocked / blocking
all publishers, all nodes]
  B --> B2[Publishing halted
consumers still drain]

The practical summary: flow means one publisher is outpacing one part of the write path, and the broker is coping. blocked means the broker has decided it is about to run out of a resource it cannot afford to lose, and it has stopped all ingestion everywhere until that changes.

How to tell them apart in 30 seconds

Run one command and look at the distribution:

# Count connections by state (-q suppresses the header line)
rabbitmqctl -q list_connections state | sort | uniq -c

Or via the management API:

# Same count via HTTP (guest/guest only works from localhost by default)
curl -s -u guest:guest http://localhost:15672/api/connections | jq 'group_by(.state) | map({state: .[0].state, count: length})'

Then interpret:

If you see blocked or blocking, confirm which alarm fired and on which node:

# Check both alarm flags across nodes
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_alarm, disk_free_alarm, mem_used, mem_limit, disk_free, disk_free_limit}'

The alarm is cluster-wide: the node reporting mem_alarm: true or disk_free_alarm: true may not be the node your publishers are connected to, but their connections are frozen anyway. Alert text that says “node X memory high” undersells it. The operational truth is “cluster publishing halted.”

Why flow control is not a problem to page on

Flow control exists to prevent exactly the resource alarms you should be paging on. It is the broker shedding load gracefully at the edge instead of dying at the center. Key properties:

• Per-connection. Only connections publishing to the constrained path are throttled. Other connections on the same node, publishing to healthy queues, are unaffected.
• Transient. The state toggles as credit is granted and spent, potentially many times per second. Any single sample under load will catch some connections mid-throttle. That is sampling noise, not an incident.
• Not necessarily resource-driven. Flow control fires whenever one part of the pipeline cannot keep up with another: a single hot queue process (each queue is one Erlang process, bounded by one core), slow persistence writes, or contention on the cluster link for queues owned by remote nodes.
• Invisible to many clients. AMQP client libraries often do not surface flow control to the application. The publisher just sees reduced throughput. That is why broker-side visibility matters.

The legitimate alert on flow is about persistence, not presence: alert when more than roughly 10% of connections are in flow state sustained for more than 5 minutes. That pattern indicates a systemic write-path bottleneck and often precedes a memory alarm by minutes. A handful of flow connections during a burst is INFO at most.

One collection caveat: there is no aggregate metric for per-state connection counts in the Management API or the Prometheus plugin. The count must be derived by iterating /api/connections, which is expensive on deployments with many connections. Sample at a sensible interval rather than polling aggressively.

Why resource alarms are always serious

A resource alarm is a cliff edge, not a slope. RabbitMQ does not gradually slow down as memory approaches the watermark: the moment the threshold is crossed on any node, all publishers on all nodes stop. There is no hysteresis either. The alarm clears at the same threshold it triggered at, so a node hovering at the watermark can flap, repeatedly blocking and unblocking the cluster. This is why alert conditions on mem_alarm and disk_free_alarm should require sustained state (for example, active for more than 60 seconds) plus evidence of traffic impact, and why flapping near the threshold deserves its own investigation into watermark sizing.

The two alarms behave identically at the connection layer but differ in cause and recovery:

• Memory alarm (mem_alarm): usually driven by queue growth, and most often by messages_unacknowledged rather than messages_ready. Unacked messages are pinned in RAM and cannot be paged out. Check queue totals, then the top queues by memory: GET /api/queues?sort=memory&sort_reverse=true. Do not restart the broker; that discards in-memory state and usually worsens things.
• Disk alarm (disk_free_alarm): check actual free space on the data partition with df -h. If space is plentiful, the cause is almost always disk_free_limit left at the default 50MB, which any log rotation can trip. If space is genuinely low, let consumers keep draining (they are not blocked) while you free space.

The early warning for the memory alarm is paging: at vm_memory_high_watermark_paging_ratio (default 0.5 of the watermark) queues start moving messages to disk, and messages_paged_out and disk I/O rise before the alarm fires. If you only watch the alarm boolean, you get zero warning. If you watch memory ratio and paging, you get minutes.

Signals to watch in production

Note the diagnostic shape of a real alarm: publish rate collapses while connection count stays stable, because clients are connected but frozen. That combination, plus blocked connection states, is unambiguous.

Fixing your alerting

The concrete changes, in order of impact:

• Split the alert rules. One rule pages on mem_alarm or disk_free_alarm (sustained, with traffic impact). A separate, lower-severity rule tickets on sustained widespread flow. Never one rule on “connection not running.”
• Rewrite alert text. Replace “node X memory high” with “cluster publishing halted: memory alarm on node X.” On-call should know the blast radius from the notification alone.
• Add leading indicators. Alert at ticket level on mem_used / mem_limit > 0.7 and on disk_free < max(3 * disk_free_limit, 1GB) so you are woken by the approach, not the cliff.
• Gate on uptime. Suppress alarm-based pages while node uptime is under about 10 minutes; memory spikes during queue repopulation after a restart are normal warm-up behavior.
• Expect flapping. With no hysteresis, an alarm hovering at the threshold will fire and clear repeatedly. Require sustained duration, and treat flapping itself as a signal that the watermark or the workload needs attention.

How Netdata helps

• Netdata collects RabbitMQ node and alarm state, so mem_alarm and disk_free_alarm transitions are visible on a timeline alongside the metrics that caused them, instead of being isolated boolean events.
• Memory ratio, queue totals (messages_ready, messages_unacknowledged), and publish rates are collected together, which makes the classic cascade readable in one view: unacked growth, memory ratio climbing, paging starting, then the alarm.
• Because publish rate and connection count are charted together, the fingerprint of a resource alarm (publish rate collapsing while connections stay stable) is easy to confirm visually before you start running rabbitmqctl.
• Sustained flow pressure shows up indirectly as degraded publish throughput and rising paging, which helps separate chronic write-path bottlenecks from transient burst behavior.

Related guides

• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly
• RabbitMQ paging messages to disk: the paging ratio as an early memory warning
• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ monitoring maturity model: from survival to expert