RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic

Your RabbitMQ cluster is acting haunted. Quorum queues lag behind their leaders. Metadata operations time out intermittently. Nodes occasionally report network partitions even though latency between nodes is sub-millisecond and nothing is actually down. You investigate each symptom separately and find nothing.

The likely common cause is Erlang distribution port saturation. In a RabbitMQ cluster, all inter-node traffic flows over a single TCP connection per node pair, by default on port 25672. Mnesia metadata replication, quorum queue Raft consensus traffic, management stats aggregation, and internal RPC all share that one connection. There is no separate channel, no QoS, no prioritization. When the volume of inter-node traffic exceeds what that one connection can carry, everything it carries degrades at once.

This bottleneck is invisible to most monitoring setups. RabbitMQ does not expose distribution link queue depth as a metric. The symptoms appear in completely different subsystems: Raft commit lag looks like a disk problem, Mnesia timeouts look like a metadata problem, false partition detection looks like a network problem. This guide ties them together and shows you how to confirm or rule out the distribution link in a few minutes.

What this means

The Erlang distribution protocol is how BEAM nodes talk to each other, and RabbitMQ clustering is built on it. For every pair of nodes in the cluster, there is exactly one TCP connection carrying all node-to-node communication:

• Mnesia replication: cluster metadata (users, vhosts, queue, exchange, and binding definitions) is synchronously replicated across nodes.
• Quorum queue Raft traffic: every publish, ack, and metadata update on a quorum queue requires consensus messages between the leader and its followers. This is usually the largest and most latency-sensitive traffic class.
• Stats aggregation: management statistics are collected and aggregated across the cluster over the same link.
• Internal RPC: queue operations routed to remote owners, cluster management commands, and coordination messages.

A TCP connection is a flow-controlled pipe. When senders produce data faster than the receiver drains it, the kernel send queue on the sending side grows, and everything behind the head of that queue waits. A Raft heartbeat and a bulk stats payload are peers in the same FIFO. If the link is saturated, the heartbeat waits behind the stats payload.

The downstream consequences are the confusing part:

• Raft replication falls behind: followers cannot acknowledge entries fast enough, so the leader accumulates uncommitted entries. Writes that require consensus stall or time out.
• Mnesia transactions time out: metadata operations slow down or fail intermittently.
• False partition detection: Erlang nodes exchange ticks over the distribution connection. With the default net_ticktime of 60 seconds and the default tick intensity of 4, a tick is sent at least every 15 seconds and a node is considered down only after roughly 60 seconds of silence (Erlang detects it between 45 and 75 seconds); sustained queueing delay can push ticks past their deadline the same way a dead network does. The cluster can declare a partition that does not exist.

flowchart LR
  subgraph A["Node A"]
    MN1[Mnesia replication]
    RAFT1[Quorum Raft traffic]
    STATS1[Stats aggregation]
    RPC1[Internal RPC]
  end
  LINK["Single TCP connection per node pair - port 25672"]
  subgraph B["Node B"]
    MN2[Mnesia]
    RAFT2[Raft followers]
    STATS2[Stats]
    RPC2[RPC handlers]
  end
  MN1 --> LINK
  RAFT1 --> LINK
  STATS1 --> LINK
  RPC1 --> LINK
  LINK --> MN2
  LINK --> RAFT2
  LINK --> STATS2
  LINK --> RPC2
  LINK -. "saturation" .-> SYM["Send queue grows: Raft lag, Mnesia timeouts, delayed ticks, false partition detection"]

Common causes

Quick checks

All read-only. Run on the node you suspect is the bottleneck (usually the one originating the most traffic).

# 1. Watch the TCP send queue depth on the distribution port.
# Look at the Send-Q column for established connections to peer nodes on 25672.
# A persistently non-zero Send-Q means the link is backed up.
ss -tnp | grep 25672

# 2. Check cluster link traffic volumes as seen by RabbitMQ.
# recv_bytes/send_bytes are cumulative counters; sample twice and diff for a rate.
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, cluster_links}'

# 3. Confirm whether a partition has been (falsely) detected.
rabbitmq-diagnostics cluster_status

# 4. Check the Erlang run queue. Scheduler saturation delays tick processing
# and distribution handling, which produces the same false-partition symptom.
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, run_queue}'

# 5. Check quorum queue leader distribution.
rabbitmqctl list_queues name type leader | grep quorum | awk '{print $3}' | sort | uniq -c

# 6. Check node uptimes. A recently restarted node that is still
# synchronizing quorum followers will generate a large inter-node burst.
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, uptime_seconds: (.uptime / 1000)}'

# 7. Search the broker log for distribution buffer and Mnesia warnings.
grep -i "distribution\|mnesia\|buffer" /var/log/rabbitmq/rabbit@$(hostname).log | tail -50

How to diagnose it

1. Establish the send-queue baseline. Run ss -tnp | grep 25672 several times over a few minutes during the symptom window. The Send-Q column for an established distribution connection should be at or near zero on a healthy link. A Send-Q that is persistently non-zero, or that grows over successive samples, is direct evidence of saturation: the local kernel has data buffered that the connection cannot drain.

2. Correlate with cluster link rates. Sample cluster_links from the Management API twice, 60 seconds apart, and compute per-second rates for send_bytes and recv_bytes per peer. Compare against what your inter-node network can actually carry. Asymmetry (one direction much higher than the other) points to unbalanced queue leadership or one-directional replication catching up.

3. Rule out the mimics. False partition detection also comes from GC pauses and scheduler saturation, not just link saturation. Check the Erlang run queue: a sustained run queue above the number of CPU cores means every operation, including tick processing, is delayed. If the run queue is high and the send queue is empty, the problem is CPU, not the link. If the send queue is deep and the run queue is fine, the problem is the link. See RabbitMQ flow control vs resource alarms for the related confusion between throttling mechanisms.

4. Identify the dominant traffic class. If saturation is confirmed, work out what the link is carrying. High quorum queue publish rates concentrated on remote leaders means Raft/message traffic. A node that recently restarted (low uptime) plus a send spike means follower resync. High queue churn means Mnesia replication of metadata. The fix differs for each.

5. Check the timeline against partition events. If cluster_status reported partitions, note when they were detected and whether they self-healed. Transient partitions that clear on their own, coinciding with traffic peaks and elevated send-queue depth, are the signature of tick delay from a saturated link rather than a real network failure.

Metrics and signals to monitor

Fixes

Reduce traffic crossing the link

The most effective fix is to send less data over the distribution connection.

• Rebalance quorum queue leaders. Leaders do all publish and dispatch work; followers only replicate. If leaders are concentrated on one or two nodes, their distribution links carry disproportionate load. Redistribute leadership so each node holds roughly 1/N of leaders. Leader transfer causes brief per-queue unavailability, so do this off-peak.
• Co-locate publishers with queue leaders. A client publishing through node A to a queue led by node B pushes every message across the distribution link. Routing clients to the node that leads their hot queues converts inter-node traffic into local traffic. This is a load-balancer and connection-string concern, not a broker setting.
• Smooth out resynchronization. Rolling restarts force followers to catch up over the link. Stagger restarts, and verify a node has fully resynced before restarting the next. rabbitmq-diagnostics check_if_node_is_quorum_critical tells you whether stopping a node would cost any queue its quorum.

Increase what the link can absorb

• Distribution buffer size. RabbitMQ applies temporary throttling when the inter-node communication buffer is full. The buffer is controlled by RABBITMQ_DISTRIBUTION_BUFFER_SIZE and defaults to 128 MB (128000 kB); values below 64 MB are not recommended. Raising it absorbs bursts but does not increase sustained throughput, so treat it as headroom for spikes, not a cure for chronic overload.
• Network capacity between nodes. If the send queue is deep while CPU and run queue are healthy, the link itself may be the constraint. Inter-node traffic should live on a low-latency network with genuine headroom; running cluster nodes across congested or rate-limited paths invites this failure mode.

Tune detection sensitivity, carefully

• net_ticktime controls how quickly missed ticks become a partition. Raising it makes detection more tolerant of transient link congestion, but it also slows detection of real failures, and all nodes must agree on the value. Do not use ticktime tuning to mask a genuinely saturated link; fix the traffic first.
• Do not confuse this with resource alarms. If publishers are blocked cluster-wide, that is a memory or disk alarm, a different mechanism with different fixes. See RabbitMQ connection blocked / blocking.

Prevention

• Monitor the send queue, not just the byte counters. Byte rates tell you volume; the kernel send queue tells you backpressure. Alert on sustained non-zero Send-Q on the distribution connections. This is the earliest reliable warning and is only visible via OS tooling such as ss, not the Management API.
• Baseline cluster link rates per peer. Know what normal send/recv rates look like at peak. Deviation from the peak baseline combined with rising Send-Q is the alertable condition.
• Keep leadership balanced as a standing practice. Check leader distribution after every restart, upgrade, or node replacement, not just during incidents.
• Design client placement deliberately. Document which node leads which hot queues and point connection strings or load-balancer pools accordingly.
• Gate partition alerts on corroboration. Require a sustained non-empty partitions array plus a corroborating signal (cluster link traffic to the peer at zero, or quorum queues in minority) before paging. This filters transient false detections caused by tick delay without masking real splits.
• Plan capacity for the link, not just the broker. When sizing a cluster, include peak quorum replication bandwidth in the inter-node network budget. The distribution connection is a single flow-controlled pipe with no percentage-based headroom metric; it either keeps up or the send queue grows.

How Netdata helps

• Cluster link traffic per peer: Netdata collects RabbitMQ node metrics including cluster link traffic from the Management API and charts them as rates, making traffic asymmetry and resync bursts visible without manual sampling.
• Correlation across the symptom set: viewing cluster link rates alongside network partition status, Erlang run queue, and quorum queue depth on one dashboard is how you connect “Mnesia timeouts plus quorum lag plus a transient partition” to a single saturated link instead of chasing three phantom incidents.
• Run queue vs. link discrimination: per-second run queue history next to cluster link throughput lets you quickly separate scheduler saturation (false ticks from CPU) from genuine link saturation (deep send queues with healthy schedulers).
• Restart context: node uptime charts line up resynchronization traffic spikes with the restart that caused them, so a post-restart burst reads as expected recovery rather than a new fault.
• Flow-state visibility: connection state counts show when remote-owner backpressure is propagating to publishers, the client-facing symptom of a congested distribution path.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ head message age: the queue latency that depth alone cannot show
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up