RabbitMQ node is quorum critical: checking before a rolling restart

You are about to restart a RabbitMQ node: a rolling upgrade, an OS patch, an instance resize. Before you stop it, one question decides whether this is routine maintenance or a queue outage: if this node goes down right now, does any quorum queue or stream lose its online majority?

RabbitMQ ships a purpose-built check for exactly this. rabbitmq-diagnostics check_if_node_is_quorum_critical returns unhealthy when stopping the target node would drop a quorum queue below the number of online members it needs to accept writes. The same logic is exposed over HTTP at GET /api/health/checks/node-is-quorum-critical, which makes it usable from automation, load balancer health gates, and CI-driven upgrade pipelines.

This guide covers what the check evaluates, how to run it from the CLI and the API, how to sequence a rolling restart around it, and the edge cases where it can mislead you.

What the quorum-critical check does

Quorum queues replicate each message across a group of members using Raft consensus. A queue with 3 members needs 2 online members (a majority) to accept writes and elect a leader. A queue with 5 members needs 3. When a queue falls below majority, it shows state: minority in the management API: it may still serve already-committed messages to consumers, but it cannot accept new publishes.

The check inverts that logic from the operator’s perspective. Instead of asking “is any queue currently in minority?”, it asks “if I stop this node, would any queue end up in minority?” For every quorum queue and stream with a member on the target node, it looks at the currently online members and computes whether removing this node leaves fewer than a majority online.

The critical detail: the check cares about online members, not configured members. With 3 members configured and 1 already down, only 2 are online. Taking a second node down leaves 1 of 3 online, below majority. The queue goes unavailable. The check on the second node returns unhealthy precisely because of this.

flowchart TD
    A[Node restart requested] --> B{Run quorum-critical check}
    B -->|healthy, exit 0| C[Drain and stop node]
    C --> D[Restart node]
    D --> E{All members back online?}
    E -->|yes| F[Proceed to next node]
    E -->|no| G[Do not continue. Investigate the lagging member]
    B -->|unhealthy, exit 69| H[Do not restart. A queue would lose majority]
    H --> I[Find which queue members are offline]
    I --> J[Restore the offline member first]
    J --> B

Why this is the gate that matters during rolling restarts

Rolling restarts fail in a predictable way with quorum queues:

1. Node 1 goes down for patching. Quorum queues with a member on node 1 now run at 2 of 3 online. Still accepting writes. Leader elections happen for queues led by node 1. All normal.
2. The operator, or the automation, does not wait for node 1 to fully return and rejoin its queue groups. Node 2 goes down.
3. Every queue with members on both node 1 and node 2 is now at 1 of 3 online. Minority. Publishing to those queues stops. Consumers keep draining committed messages, but producers block or fail.

The trap is that a quorum queue with one member down looks like it is working fine, because commits still succeed. Fault tolerance is gone, and one more failure makes the queue unavailable. The check surfaces that hidden fragility before you act on it.

Two behaviors during restarts are worth expecting rather than fighting:

• Leader elections are normal. Rolling restarts cause Raft leader elections as members leave and return. Do not alarm on leadership changes during the maintenance window.
• Followers need resync time. After a node restarts, its quorum queue followers resynchronize with the leader. Depending on queue depth this can take minutes to hours. The node being “up” does not mean its queue members are caught up. Wait for full membership before moving to the next node.

How to run the check

CLI

# Quorum-critical gate before restarting this node
rabbitmq-diagnostics check_if_node_is_quorum_critical

The command is also available as rabbitmq-queues check_if_node_is_quorum_critical.

Exit behavior:

• Exit 0: stopping this node leaves every quorum queue and stream with an online majority. Safe to proceed.
• Exit 69: stopping this node would leave at least one quorum queue or stream without an online majority. Do not proceed.
• Other non-zero codes: the check itself failed (for example, it could not reach the node). Treat this as “unknown”, not as “safe”.

The exit code distinction matters for automation. The Kubernetes Cluster Operator’s preStop hook loops on this command and blocks pod termination only on exit code 69; other failures are not treated as quorum-critical. Mirror that behavior in your own scripts: gate on 69 specifically, and fail closed (do not restart) on anything you cannot classify.

HTTP API

# Quorum-critical gate over the management API
curl -s -o /dev/null -w "%{http_code}\n" \
  -u "$USER:$PASS" \
  http://localhost:15672/api/health/checks/node-is-quorum-critical

A 200 response means the node is not quorum critical. A failure status means stopping it would break a majority somewhere. This endpoint is the right choice for load balancer health gates and external orchestration that should not shell into the node.

There is also a management CLI equivalent, rabbitmqadmin health_check node_is_quorum_critical.

Scope note

The check covers both quorum queues and streams in current releases. Classic mirrored queues are a separate concern (check_if_node_is_mirror_sync_critical), and mirroring was removed entirely in RabbitMQ 4.0, so on current versions the quorum check is the only gate of this kind you need.

Interpreting the result

When the check says unhealthy, find out why before doing anything else:

# Which quorum queues exist, where are their members, and who is online
rabbitmqctl list_queues name type state members online

Look for queues where the online member count is lower than the members count. That gap is your exposure. Common reasons a member is offline:

• Another node is down, intentionally or not. Check cluster status and node uptime across the cluster.
• A node recently restarted and its followers are still resyncing. Wait for resync to complete.
• A queue was declared with members on nodes that no longer exist or were renamed.

The fix is almost always “restore the missing member first, then re-run the check”. Do not work around the gate by stopping the node anyway: the check is telling you a specific queue will stop accepting writes the moment you do.

Sequencing a rolling restart safely

For a 3-node cluster, the safe sequence is:

1. Baseline. Confirm all nodes are running, no partitions are detected, and every quorum queue shows its full member count online.
2. Gate node 1. Run the quorum-critical check on node 1. Only proceed on a healthy result.
3. Drain and stop node 1. Put it into maintenance mode if your workflow uses it, stop the node, do the work, start it again.
4. Wait for full rejoin. Do not move on when the node process is up. Wait until its quorum queue members are online and caught up, leader elections have settled, and resync has finished. rabbitmq-upgrade await_online_quorum_plus_one exists for exactly this wait in automated flows: it blocks until there are enough nodes online to maintain quorum plus one.
5. Re-verify the cluster. Confirm list_queues name type state members online shows full membership everywhere and no queue is in minority.
6. Gate node 2. Run the check on node 2. Repeat.

The discipline is the same in Kubernetes, where the Cluster Operator already implements most of it: the preStop hook blocks pod termination while the node is quorum critical, and the operator surfaces a quorumStatus field on the RabbitmqCluster resource so you can see the state externally. If you run RabbitMQ outside the operator, you own this sequencing yourself, and the check is the primitive to build it on.

One cadence note: environments with very high queue churn (tens of queue declarations and deletions per second) are harder to restart cleanly, because queue groups are forming and dissolving while members are offline. If your workload churns queues aggressively, schedule restarts in the quietest window you have and re-run the check immediately before each stop, not at the start of the whole procedure.

Known false positives and blind spots

The check is the right tool, but it is not perfect. Know its failure modes before you build a hard gate on it:

• False positive after recent replica changes (fixed in 3.13.4). The check could report a node as quorum critical when some quorum queue replicas had been very recently added or very recently restarted. If you are on an older 3.13 release and the check disagrees with what list_queues name type state members online shows, upgrading past 3.13.4 resolves it.
• False positive during maintenance mode (fixed in 3.8.10). A node drained via maintenance mode could leave a stale process entry that made the check report quorum critical incorrectly. Only relevant on very old versions.
• False negative with a dead follower. In a 3-node cluster with a 3-member queue where one follower process has crashed (shown as “noproc”), the check historically did not flag the leader as quorum critical, even though stopping the leader would leave no quorum. A fix was merged (PR #12727) and has shipped in stable releases: RabbitMQ 4.0.4 and 4.1.0. Until you can verify the fix is in your running version, do not rely on the check alone: verify member online status directly with list_queues name type state members online before each stop.
• The check is point-in-time. A healthy result at 09:00 says nothing about 09:05 if another node fails in between. Re-run the check immediately before each stop, and never run two stops concurrently “because both checked out earlier”.

Signals to watch during the restart

How Netdata helps

• Netdata collects per-node availability, uptime, and partition status, so you can see exactly when each node left and returned during the rolling restart, and catch a node that came back but partitioned.
• Per-queue state and message metrics let you spot a queue stuck in minority or not recovering after its member node rejoined, which is the failure mode the quorum-critical check is designed to prevent.
• Publish, deliver, and ack rate charts give you the “did traffic actually resume” confirmation after each node returns, before you gate the next one.
• Correlating uptime resets with queue state changes on one timeline makes it obvious whether a minority event lines up with your maintenance window or with an unplanned failure that should stop the rollout.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted