vCenter HA (VCHA) replication broken: protection that is not protecting

VCHA makes vCenter look protected. The VAMI dashboard shows a green cluster, the active node serves traffic, and the passive node exists as a standby. When PostgreSQL streaming replication between the active and passive node stops, the passive node holds a stale database. A failover loses every transaction written since replication broke.

VCHA’s health surface is shallow. The VAMI summary can report healthy while pg_stat_replication shows NOT_REPLICATING. The passive VM is powered on. The witness is unreachable, so automated failover cannot reach quorum. None of this surfaces until an operator triggers failover and discovers the passive is hours behind, or until WAL accumulation on the active fills /storage/db and vpxd stops.

What this means

VCHA is an active/passive/witness topology for the vCenter Server Appliance. The active node runs all vCenter services and owns the management IP. The passive node is a hot standby that receives database changes through PostgreSQL streaming replication over a dedicated network. The witness is the third vote in a quorum that decides which node can become active during failover.

Healthy state:

• pg_stat_replication on the active node shows state = streaming for the passive node’s IP
• Replay lag is near zero (low single-digit MB at worst)
• All three nodes are reachable across the VCHA network
• Automated failover can reach quorum (2 of 3 nodes agree)

When any of these drift, VCHA stops protecting without making it obvious. The passive is powered on, the active is serving traffic, but a failover either does not happen (no quorum because the witness is down) or happens onto a stale database (replication broken).

Three operational constraints make silent VCHA breakage worse:

• Failover is not instant. It restarts all vCenter services on the passive node. Expect 5-10+ minutes of management plane downtime, not seconds. ESXi-level HA continues independently of vCenter.
• The management IP moves during failover. Client DNS caching extends apparent downtime beyond the failover window. Tools with long-lived connections to vCenter must reconnect.
• Large file-backed artifacts such as local Content Library payloads or Update Manager patch repositories can create a failover gap if they are not shared or covered by the build-specific VCHA file-replication set. Verify their location and coverage for the deployed vCenter version.

flowchart TD
    ACT[Active VCSA
all services run here] -->|streaming replication| PAS[Passive VCSA
hot standby]
    ACT -.->|heartbeat| WIT[Witness
quorum vote]
    PAS -.->|heartbeat| WIT
    ACT -->|writes WAL| PG[(vPostgres
pg_wal)]
    PAS -->|replays WAL| PASDB[(vPostgres
database)]
    FAIL[Automated failover] -->|needs 2-of-3 quorum| WIT
    FAIL -->|restarts all services
5-10+ min| PAS
    IP[Management IP] -.->|moves during failover| DNSC[Client DNS cache
extends apparent downtime]
    WIT -.->|unreachable| NOQ[no quorum
no automated failover]
    ACT -.->|replication broken| STALE[passive DB is stale
failover loses data]

Common causes

Quick checks

Run these on the active VCSA node. They are read-only and safe.

# Check VCHA cluster state and health. The shell command has varied by vCenter build; prefer the VCHA page in the vSphere Client
# or the vCenter Automation API operation com.vmware.vcenter.vcha.cluster.get.
vcha cluster get 2>/dev/null || /opt/vmware/share/vcha/vcha-util status 2>/dev/null || echo "use VAMI, vSphere Client VCHA page, or vCenter API"

# Check PostgreSQL replication state and lag (the ground truth)
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "
SELECT client_addr, state, sync_state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes,
       pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS sent_lag_bytes
FROM pg_stat_replication;"

# Check disk pressure on /storage/db (WAL accumulation indicator)
df -h /storage/db
du -sh /storage/db/vpostgres/pg_wal/

# Check VCHA network interface (dedicated NIC; name varies by Photon version)
ip addr show | grep -A2 "vcha\|eth1"

# Check VCHA logs for failover, heartbeat, or replication events
grep -i "failover\|replication\|heartbeat\|split.brain" /var/log/vmware/vcha/vcha.log | tail -50

# Check time synchronization
timedatectl status

When the psql -U postgres command prompts for a password, use the password in /etc/vmware-vpx/embedded_db.cfg on that node; local authentication can differ between VCSA builds.

How to diagnose it

1. Verify replication state. On the active node, run the pg_stat_replication query above. If the result is empty or shows state other than streaming, replication is broken. This is the single most important check. A VAMI health summary that says “healthy” does not override a NOT_REPLICATING state here.

2. Check the VCHA network. VCHA uses a dedicated NIC separate from the management network. Verify the interface is up, has the correct IP, and can reach both the passive and witness nodes. Packet loss, MTU mismatch, or bandwidth saturation on this network degrades replication silently before it breaks.

3. Check replay lag in bytes. If replay_lag_bytes is growing, the passive cannot keep up with WAL replay. Sustained lag above 10MB warrants investigation. If it grows unboundedly, the active accumulates WAL that the passive has not consumed, and /storage/db on the active fills.

4. Verify witness reachability. The witness is required for automated failover quorum. If the witness is unreachable, automated failover cannot happen regardless of replication health. Check the witness VM power state and network connectivity from both active and passive.

5. Check /storage/db on the active node. Replication lag manifests as WAL accumulation. If /storage/db is filling and replication lag is high, the replication break is the root cause. Do not manually delete WAL files. Doing so breaks PostgreSQL crash recovery.

6. Check time synchronization. Photon OS on the VCSA syncs time from the ESXi host. If NTP is misconfigured on the hosts, VCHA nodes drift and can trigger repeated failover every 5-10 minutes.

7. Check root password expiry on passive and witness. In vSphere 8.0 versions before Update 3e, root password expiry on a passive or witness node caused file replication to fail, health to degrade, and vpxd to fail (PR 3456483). Verify the patch level and password expiry.

8. Check for snapshots on the VCSA VM. Snapshots on a VCHA-enabled vCenter are not supported. They can trigger failover under high load and break VCHA operability on failback.

Metrics and signals to monitor

Fixes

Replication broken from network partition

If pg_stat_replication shows NOT_REPLICATING and the VCHA network is the cause, fix the network first. Check the dedicated NIC, verify IPs, test connectivity between all three nodes. Do not attempt a manual failover while replication is broken unless the active node is permanently lost. If you must fail over onto a stale passive, treat it as a data-loss event and reconcile from file-based VAMI backups.

WAL accumulation filling /storage/db

If /storage/db is filling due to WAL from replication lag, the fix depends on the root cause. If replication is broken, fixing replication allows the passive to catch up and WAL to be consumed. If the passive cannot keep up due to a resource constraint, investigate the passive node’s resources. Do not manually delete WAL files, and do not turn off WAL archiving as an ad hoc VCHA workaround without Broadcom support guidance; PostgreSQL retains archived WAL when archiving is misconfigured.

Passive node resource mismatch (vSphere 8.0)

If VCHA setup stalls at “PostgreSQL replication is not in progress,” the passive node’s resources may not match the active. In vSphere 8.0, the max_connections setting in postgresql.conf is auto-configured based on total memory. If the passive has different memory or vCPU than the active, the active cannot open connections to the passive database and replication never starts. Do not change virtual hardware on VCHA nodes after cloning. They must be exact mirrors.

Witness node unreachable

If the witness is down, automated failover is disabled. The cluster still has a replication pair, but without the witness there is no quorum vote for automated takeover. Fix the witness: verify the VM is powered on, check the VCHA network, and re-establish connectivity. If the witness is permanently lost, VCHA must be destroyed and reconfigured.

Time drift causing repeated failover

If VCHA is failing over every 5-10 minutes, check NTP on the ESXi hosts hosting the VCHA nodes. Photon OS syncs time from the host, and drift between hosts causes the nodes to disagree on timing, triggering failover. Configure NTP correctly on all ESXi hosts before re-enabling automated failover.

Root password expired on passive or witness

If running a version before 8.0 Update 3e, root password expiry on passive or witness breaks file replication and can cause vpxd to fail. Update to 8.0u3e or later (PR 3456483) and reset the root password on all nodes.

Snapshots taken on the VCSA VM

Snapshots on a VCHA-enabled vCenter can cause failover under high load and leave VCHA inoperable after failback. Remove snapshots from the VCSA VM and reconfigure backup tooling to use file-based VAMI backups instead of VM-level snapshots.

Prevention

• Monitor pg_stat_replication directly, not just VAMI health. VAMI can report healthy while replication is broken. The PostgreSQL view is the ground truth.
• Alert on replay lag bytes, not just replication state. A state of streaming with growing lag is still a problem. Sustained lag above 10MB means the passive is falling behind and WAL is accumulating on the active.
• Monitor /storage/db on the active node with VCHA context. WAL accumulation from replication lag is a VCHA-specific failure mode. A filling /storage/db on a VCHA active node should trigger a replication check immediately.
• Monitor witness reachability independently. A healthy replication pair with a dead witness means no automated failover. This is a protection gap, not an active outage, but it should be a ticket-level alert.
• Never snapshot a VCHA-enabled vCenter. Snapshots can cause failover under load and break VCHA on failback. Use file-based VAMI backups, which are the only VMware-supported backup method for VCSA.
• Verify VCHA is supported in your topology. VCHA is not supported for vCenter Server instances managed by VMware Cloud Foundation (VCF). Network requirements include latency under 10ms and at least 1 Gbps between nodes, and multi-homed NICs are not supported.
• Patch VCHA nodes together. Version drift between active, passive, and witness is not supported and causes replication failures.

How Netdata helps

• PostgreSQL replication metrics surface pg_stat_replication state and WAL lag per-second, so a flip from streaming to NOT_REPLICATING is visible immediately rather than at the next manual check.
• Per-partition disk utilization on /storage/db catches WAL accumulation before it fills the active node, with rate-of-change alerts that distinguish steady-state growth from replication-lag-driven spikes.
• Network interface metrics on the dedicated VCHA NIC show packet drops, errors, and bandwidth saturation that silently degrade replication before it breaks entirely.
• NTP offset and time sync metrics detect drift between VCHA nodes and their ESXi hosts, the precursor to repeated failover loops.
• Correlating VCSA service health, disk pressure, and replication state in one timeline shortens diagnosis when VAMI reports healthy but the cluster is not actually protecting.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong