RabbitMQ vhost stopped or partially available: queues unreachable on some nodes

You are looking at GET /api/vhosts and one vhost does not report running on every node. Its cluster_state map shows stopped on one or more nodes, or nodedown, while the rest of the cluster looks fine. Applications that connect to the affected node and target that vhost fail to open connections, or their channels error out immediately.

This is a per-node condition, which is what makes it confusing. A vhost is not a single process with one state; it is a supervision tree that runs independently on each cluster node. When it shows stopped on one node, that node’s message store, vhost supervisor, and queue processes for that vhost are not running. Clients pinned to that node by a load balancer or connection string see failures, while clients on other nodes keep working.

Treat this as a ticket, not a page: partial vhost availability is expected transiently during rolling restarts and deliberate maintenance. It stops being routine when the state persists after the maintenance window, when no restart was scheduled, or when client impact reports line up with the stopped node.

What this means

In the Management API, each vhost object carries a cluster_state field: a map of node name to state string. The three values you will see are running, stopped, and nodedown. Older versions (before 3.7) exposed a single state field for the whole vhost, which hid exactly this failure mode; the per-node cluster_state map replaced it in RabbitMQ 3.7 because a vhost can be healthy on some nodes and dead on others.

When a vhost is stopped on a node, everything scoped to that vhost on that node is unavailable: its message store, its exchanges, its bindings, and its queue processes. Clients that connect to that node and request the vhost get connection or channel failures, typically surfacing as an error that the vhost’s supervisor is not running. Classic queues whose process lived on the stopped node show as down; quorum queue members on the stopped node fall behind, and the queue can enter minority if enough members are affected.

The nodedown value is different in kind: it means the node itself is unreachable or down (the value is nodedown when the per-node query cannot reach the node), so the vhost state there cannot be assessed. Treat nodedown as a node-level incident with a vhost symptom attached, and treat stopped as a vhost-level incident on a live node.

flowchart TD
  A[cluster_state shows stopped or nodedown on a node] --> B{Is the node itself up?}
  B -->|No| C[nodedown: node-level incident]
  B -->|Yes| D[stopped: vhost supervisor down on live node]
  C --> E[Check node availability and partitions]
  D --> F[Clients on this node fail on this vhost]
  D --> G[Queues in this vhost down or behind on this node]
  D --> H[rabbitmqctl restart_vhost]
  C --> I[Recover node, then verify vhost returns to running]

Common causes

Quick checks

All of these are read-only.

# Show per-node state for every vhost
rabbitmqctl list_vhosts name cluster_state

# Same data via the Management API
curl -s -u guest:guest http://localhost:15672/api/vhosts | jq '.[] | {name, cluster_state}'

# Is the affected node actually running?
rabbitmq-diagnostics -q ping
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, running, partitions, uptime}'

# Which queues in the affected vhost are impacted?
curl -s -u guest:guest http://localhost:15672/api/queues | jq '.[] | select(.vhost=="<vhost>") | {name, node, state, type, messages_ready, messages_unacknowledged}'

# Check for stuck processes on the affected node
rabbitmq-diagnostics maybe_stuck

# Resource alarms and limits on the affected node
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_alarm, disk_free_alarm, fd_used, fd_total, proc_used, proc_total}'

Notes on interpreting the output:

• If cluster_state shows nodedown for a node, confirm node health first. Do not try to fix the vhost before you know whether the node is alive.
• If queue state shows down for classic queues in that vhost, that is consistent with their host node or vhost supervisor being unavailable. If quorum queues show minority, count how many members are affected before touching anything.
• maybe_stuck matters when a later restart_vhost appears to hang: vhost supervisor processes have been observed stuck in shutdown with large mailboxes, which shows up here.

How to diagnose it

1. Map the blast radius. Run rabbitmqctl list_vhosts name cluster_state and record exactly which nodes report stopped or nodedown for the affected vhost. A single-node stopped is a vhost problem. An all-node stopped is a bigger one. Any nodedown entries redirect you to node-level diagnosis.
2. Rule out maintenance. Check node uptime in /api/nodes. A node with uptime in the low hundreds of seconds was just restarted, and partial vhost availability during a rolling restart is expected. Give it time and re-check before acting.
3. Check the node, not just the vhost. For nodedown entries, verify running and partitions in /api/nodes, and rabbitmq-diagnostics -q ping from the node itself if reachable. A node that reports itself healthy while peers see it down is a partition, which is a different incident.
4. Inventory the queues in the vhost. List queues filtered by vhost and check state, type, and message counts. You want to know: are classic queues down, are quorum queues in minority, and how much unconsumed data is sitting behind the failure. This decides urgency.
5. Look for a proximate cause on the node. Check resource alarms and ratios: mem_alarm, disk_free_alarm, fd_used/fd_total, proc_used/proc_total. A vhost supervisor that died while the node was under memory or FD pressure will likely die again after a restart unless the pressure is relieved. Review the RabbitMQ log on the affected node for supervisor crash entries around the time the state flipped.
6. Check client impact. Connections to healthy nodes targeting this vhost still work. If clients report failures, confirm which node their connection string or load balancer backend lands them on. The failure pattern should match the stopped nodes exactly; if it does not, something else is also wrong.

Metrics and signals to monitor

Fixes

Expected transient state during rolling restarts

If a rolling restart or upgrade is in flight, the correct action is usually no action. Verify the state clears as each node rejoins, and confirm queue states return to running afterwards. If a node has been back up for several minutes and its vhost is still stopped, move to the restart procedure below.

Node is down or partitioned

Fix the node first. Recover the host, resolve the partition, and confirm the node rejoins the cluster cleanly. The vhost should come up as part of normal node startup. Verify with list_vhosts name cluster_state afterwards rather than assuming. If quorum queues in the vhost went to minority while the node was out, confirm they regain their members before closing the incident.

Vhost stopped on a healthy node

RabbitMQ provides a targeted recovery command:

# Restart a failed vhost's data stores and queues (defaults to vhost "/")
rabbitmqctl restart_vhost --vhost <vhost>

This restarts the vhost’s supervision tree without touching other vhosts or requiring a node restart. The management UI exposes the same mechanism as a per-vhost restart action. Two cautions:

• restart_vhost has been observed hanging when the vhost supervisor is deeply stuck, with no log output. If it hangs, run rabbitmq-diagnostics maybe_stuck on the node to look for the stuck supervisor process before retrying or escalating.
• If the restart succeeds but the vhost stops again, treat the restart as a diagnostic, not a fix. Go back to step 5 of the diagnosis: something on that node (resource pressure, corrupted store, plugin fault) is killing it.

Last resort: node application restart

A sequential rabbitmqctl stop_app / start_app on the affected node reliably clears stuck vhosts, but it disrupts every vhost on that node, not just the broken one. Do it only when restart_vhost fails or hangs, and treat it like any disruptive action: off-peak if possible, quorum criticality checked first (rabbitmq-diagnostics check_if_node_is_quorum_critical), and clients prepared for reconnection. Do not jump straight to a full service restart; it loses in-memory state and risks making things worse.

Vhost stuck at creation time

In large clusters, vhost creation has been observed to time out and leave the vhost stopped on some nodes. Whether rabbitmqctl add_vhost waits for all nodes before returning varies by version, so do not rely on the exit code alone: after provisioning a vhost through automation, verify cluster_state on every node and run restart_vhost --vhost <vhost> on any node that did not come up.

Prevention

• Alert on the state, with maintenance awareness. Page on nothing here by default, but ticket when any node’s vhost state is not running for longer than a rolling-restart window. Suppress alerts for nodes with very low uptime to avoid firing during planned restarts.
• Watch the correlated signals, not just the vhost. Node availability, partition status, and per-queue state in each vhost will usually show the problem earlier and with more context than the vhost state alone.
• Keep nodes away from resource cliffs. Memory alarms, disk alarms, FD exhaustion, and Erlang process exhaustion all kill things you did not intend to kill. Alerting on the leading ratios (memory, FD, process count above 0.7 to 0.8 sustained) reduces the chance a vhost supervisor dies as collateral damage.
• Verify after automation. Any provisioning flow that creates vhosts should assert running on all nodes afterwards and call restart_vhost or fail loudly if not.
• Plan rolling restarts deliberately. Restart one node at a time, check check_if_node_is_quorum_critical before stopping a node, and verify vhost and queue states are green before moving to the next node.

How Netdata helps

• Per-vhost state visibility: Netdata’s RabbitMQ collector surfaces vhost-level signals from the Management API, so a vhost that stops on one node shows up as a state change rather than as a vague client-side error report.
• Correlation with node health: vhost state charts sit next to node availability, uptime, and partition signals, which is exactly the join you need to separate a vhost crash from a node-down event.
• Queue state in context: per-queue state, ready, and unacked counts per vhost let you quantify impact (which queues are down, how much data is waiting) without hand-writing API queries mid-incident.
• Resource alarm proximity: memory ratio, FD ratio, and process count trends on the affected node explain why the vhost died, not just that it died.
• Restart transients made legible: uptime charts make it obvious when a partial vhost state lines up with a rolling restart, so you do not chase an expected transient.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ head message age: the queue latency that depth alone cannot show