RabbitMQ network partition detected: split-brain in cluster_status

You ran rabbitmqctl cluster_status (or your monitoring scraped /api/nodes) and saw the “Network Partitions” section listing node names, or a non-empty partitions array on one or more nodes. The cluster has split: some nodes can no longer see each other over the Erlang distribution link, and each side now has its own view of the world.

This is a paging condition. What happens next depends on cluster_partition_handling: the two sides may be accepting writes independently and diverging (the ignore default), the minority side may have frozen and stopped serving clients (pause_minority), or nodes may be about to restart themselves (autoheal). Each outcome has a different blast radius, and the wrong response makes it worse.

The most important rule before anything else: do not intervene while the network is still flapping. Confirm the partition is real and sustained, identify which side has the majority, and let the network stabilize first.

What this means

RabbitMQ nodes in a cluster communicate over the Erlang distribution protocol, by default a single TCP connection per node pair on port 25672. It carries all inter-node traffic: Mnesia metadata replication, quorum queue Raft traffic, and internal RPC. Nodes also exchange heartbeat ticks on this link, governed by net_ticktime (default 60 seconds). If a node stops hearing from a peer for longer than the tick window, it declares the peer down and records a partition.

Each node keeps its own list of detected partitions, so one node’s view is not proof of a network failure. A long GC pause or a saturated Erlang run queue on a single node can delay tick processing enough to trigger a false partition that self-heals within an interval or two.

What the partition means for your data depends on queue type:

• Classic queues: mastered on one node. Clients connected to the other side of the partition lose access to them. With ignore handling, both sides keep accepting writes to their local state and the metadata diverges. On rejoin, one side’s state is discarded, which can silently lose messages.
• Quorum queues: Raft consensus means only the majority side can commit writes. The minority side shows the queue in minority state: previously committed messages may still be delivered, but new writes are refused. This is the safer failure mode and the reason quorum queues are the recommended type.

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. Confirm the partition and see this node's view
rabbitmqctl cluster_status
# Look for the "Network Partitions" section listing peer node names

# 2. Check the partitions array on every node via the API
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, running, partitions}'

# 3. Check cluster link traffic between peers (cumulative bytes)
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, cluster_links}'

# 4. Check Erlang run queue (CPU saturation can cause false partitions)
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, run_queue}'

# 5. Find quorum queues stuck in minority
curl -s -u guest:guest http://localhost:15672/api/queues | jq '.[] | select(.state != "running" and .state != "idle") | {name, vhost, state, type}'

# 6. Verify OS-level connectivity to a partitioned peer (run from one side)
ping -c 3 <peer-hostname>
# and probe the Erlang distribution port
timeout 3 bash -c 'cat < /dev/null > /dev/tcp/<peer-hostname>/25672' && echo "port open" || echo "port unreachable"

# 7. Check which partition handling strategy is configured
grep -i cluster_partition_handling /etc/rabbitmq/rabbitmq.conf

Note on check 2: a node can report running: true while partitioned. “Running” only means the local Erlang application is alive; it says nothing about peer visibility. Always read partitions alongside running.

How to diagnose it

flowchart TD
    A[partitions array non-empty] --> B{Sustained over more than 2 intervals?}
    B -->|No, clears on its own| C[Transient false positive: check run_queue, GC, VM migration]
    B -->|Yes| D{OS-level connectivity between peers?}
    D -->|Broken| E[Real partition: fix network first, do not intervene on nodes]
    D -->|Intact| F[Check distribution port 25672 and firewall rules]
    E --> G[Identify majority side]
    F --> G
    G --> H{Partition handling mode?}
    H -->|ignore| I[Both sides accept writes: divergence risk, stop minority publishers]
    H -->|pause_minority| J[Minority side paused: availability loss, data safe]
    H -->|autoheal| K[Losing side will restart: brief unavailability, non-replicated data lost]

1. Confirm it is sustained. Scrape /api/nodes on at least two different nodes a couple of minutes apart. If the partitions array clears within one or two collection intervals, treat it as a transient detection event and pivot to step 5. GC pauses exceeding net_ticktime are the classic cause.

2. Verify the network independently. Do not trust RabbitMQ’s view alone. From one side of the partition, ping and TCP-probe port 25672 on the partitioned peer. If the OS cannot reach the peer either, you have a real network partition. If the OS can reach it but Erlang cannot, suspect a firewall rule change targeting the distribution port specifically.

3. Corroborate with cluster link traffic. cluster_links in /api/nodes gives cumulative recv_bytes and send_bytes per peer. Traffic dropping to zero between previously communicating peers, combined with the partitions array, confirms isolation. Caveat: an idle cluster can legitimately show near-zero inter-node traffic, so never alert on zero traffic alone.

4. Identify the majority side. Count running nodes on each side. In a 3-node cluster split 2-and-1, the pair has quorum. Quorum queues on the single node go minority and stop accepting writes. Check which queues are affected with check 5 above.

5. If it was transient, find the cause. Look at run_queue and host-level CPU and steal time around the detection window. Sustained run queue above the core count delays heartbeat processing. VM live migration and host freezes produce the same signature: a silent gap, then a false partition, then recovery.

6. Determine the configured handling strategy before predicting what happens next (check 7). The strategy decides whether you are dealing with divergence, pauses, or restarts.

Metrics and signals to monitor

Fixes

There is no safe in-place repair for an active partition. The fixes are about what you do before, during, and after.

During the partition: hold

Do not restart nodes, do not force cluster rejoins, and do not delete and re-add cluster members while connectivity is still unstable. Record the partition start time for post-incident analysis. If your handling mode is ignore and the partition is sustained, the priority is limiting divergence: stop or fence publishers on the minority side so only one side keeps accepting writes. Under pause_minority the minority has already frozen itself, which is the data-safe outcome at the cost of availability.

After the network stabilizes: let the strategy work, then verify

With autoheal, nodes on the losing side restart automatically. Expect brief unavailability and the loss of any non-replicated messages on restarted nodes. Autoheal restarts every node except the designated winner, which in some topologies includes nodes that were healthy throughout. With ignore, the partition state can persist after the network recovers and Mnesia divergence may need manual resolution; the safe pattern is restarting the minority-side nodes so they rejoin and resync from the majority. With pause_minority, paused nodes resume once they can see a majority again.

After healing, verify: partitions is empty on all nodes, no quorum queues remain in minority, and uptime resets match the nodes you expected to restart.

If it was a false positive: fix the root cause

Raise capacity or reduce load if run queue saturation delayed the ticks. On virtualized hosts, address live-migration freezes and CPU steal. net_ticktime can be tuned, but raising it also delays detection of real partitions, so treat it as a last resort rather than a default fix.

Prevention

• Use quorum queues for anything that matters. Raft majority semantics mean the minority side cannot accept writes, which removes the divergence problem that classic queues have under ignore.
• Choose the partition handling strategy deliberately and test it. pause_minority trades availability for safety; autoheal trades non-replicated data for automation; ignore (the default) does nothing and leaves split-brain resolution to you at 3 a.m. Most teams set it once and never rehearse what it actually does. Rehearse it.
• Size CPU headroom so schedulers never saturate. Sustained run queue above core count is the leading cause of false partitions. Keep average utilization with headroom under peak load.
• Protect the distribution path. Keep port 25672 reachable between nodes, exempt it from aggressive NAT or firewall idle timeouts, and give inter-node traffic enough bandwidth that Mnesia and Raft traffic cannot congest the link.
• Alert correctly. Page only on a partitions array that is non-empty for more than 2 consecutive intervals, corroborated by cluster link traffic or quorum queues in minority. A single-interval blip is almost always a GC pause, not a network event.

How Netdata helps

• Partition detection with noise control: Netdata tracks the per-node partitions array from the management API, so a sustained non-empty state surfaces as an alert rather than a single-sample false positive from a GC pause.
• Corroboration on one screen: cluster link traffic, node running state, and quorum queue states sit next to the partition signal, so you can confirm real isolation without SSHing to three nodes.
• False-positive triage: Erlang run queue and host CPU per node let you see immediately whether the “partitioned” peer was actually scheduler-saturated or frozen.
• Blast radius: per-queue state and message rates show which queues went minority and whether publishers were still writing on the minority side during the event.
• Timeline reconstruction: per-second metrics around the incident give you the exact partition start and heal times for the post-incident review.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ head message age: the queue latency that depth alone cannot show
• How RabbitMQ actually works in production: a mental model for operators