vCenter '503 Service Unavailable': the vSphere Client will not load

A 503 from the vSphere Client means the reverse HTTP proxy (rhttpproxy) accepted the TLS connection but could not reach the backend it routes to. The proxy itself is healthy. One of its dependents, typically vpxd, vmware-vapi-endpoint, vmware-stsd (STS), or the HTML5 client backend (vsphere-ui), is stopped, still starting, or crash-looping. The error string often reads “Initialization of one of the components failed.”

Running VMs are unaffected. The hypervisor plane keeps scheduling and serving I/O. What you lose is the management plane: DRS stops rebalancing, HA cannot be reconfigured, vMotion orchestration is gone, and provisioning is blocked. The urgency is operational visibility and control, not workload survival.

What this means

rhttpproxy is VMware’s HTTP reverse proxy on port 443; SSL connections to individual vCenter services go through it. When the proxy cannot connect to the backend for a path, or the backend returns an error the proxy interprets as unavailable, the client sees 503. The failure is almost always one of:

• a backend service is STOPPED (vmon reports it down, or it crashed and vmon gave up restarting it)
• a backend is still starting (vpxd rebuilds its inventory cache from vPostgres, which can take minutes)
• a backend is crash-looping (vmon has a max restart count, after which it stops trying)
• the backend is up but overloaded (rare; produces slow 503s and timeouts rather than instant ones)
• on affected vCenter 8.0 U3 builds, Envoy-sidecar overload returns 503 when heap pressure crosses its threshold

The proxy routing is the key mental model. A 503 on /ui points at vsphere-ui; a 503 on /sdk points at vpxd; a 503 on /websso points at vmware-stsd. The path in the URL is the first signal about which service to investigate.

flowchart TD
  Client[Client browser or API] -->|HTTPS :443| Proxy[rhttpproxy]
  Proxy -->|/ui path| UI[vsphere-ui]
  Proxy -->|/sdk path| VPXD[vpxd]
  Proxy -->|/websso path| STS[vmware-stsd]
  VPXD --> DB[(vPostgres on /storage/db)]
  VPXD -.->|down or starting| X1[503 on /sdk]
  UI -.->|down or starting| X2[503 on /ui]
  STS -.->|expired signing cert| X3[503 on /websso]
  DB -.->|disk full| X4[vpxd cannot start]

Common causes

Quick checks

Run these from the VCSA shell over SSH. All are read-only.

# List every vmon-managed service and its state
/usr/lib/vmware-vmon/vmon-cli --list

# Status for the four services most likely behind a 503
/usr/lib/vmware-vmon/vmon-cli --status vpxd
/usr/lib/vmware-vmon/vmon-cli --status vsphere-ui
/usr/lib/vmware-vmon/vmon-cli --status vmware-vapi-endpoint
/usr/lib/vmware-vmon/vmon-cli --status vmware-stsd

# Legacy wrapper (verify against vmon-cli for accuracy)
service-control --status --all

# Disk on every partition, not just root
df -h

# Inode exhaustion can fill a partition before bytes do
df -i

# Machine SSL certificate dates
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT | openssl x509 -noout -dates

# Externally served certificate (what the browser sees)
echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -dates

# VCSA uptime - services take 5-15 minutes to settle after boot
uptime

If a core service is STOPPED, do not restart it yet. Read its log first. Restarting before you understand why it crashed often produces another crash and burns the time you need to diagnose.

How to diagnose it

Work top-down: identify the path that 503s, identify the backend, identify why the backend is down.

1. Confirm the path. Open the vSphere Client URL and note whether the 503 hits /ui, /sdk, or another path. Also try the VAMI on port 5480. If VAMI works but /ui does not, vpxd or vsphere-ui is the suspect, not the whole appliance.

2. List service states. Run vmon-cli --list. Any core service (vpxd, vpostgres, stsd, rhttpproxy, vsphere-ui, vapi-endpoint) in STOPPED or FAILED state is your target. Run it again 60 seconds later. State oscillation is a crash-loop.

3. Check disk before anything else. Run df -h. The VCSA has dedicated partitions; root can be healthy while /storage/db, /storage/log, or /storage/seat is full. At the documented 95% database-storage threshold, vCenter 6.x, 7.x, and 8.x prevent vmware-vpxd from running to protect the database from corruption. /storage/archive is an exception and may be full by design on 6.7 and later. A full /storage/log can stop services that are unable to write their logs.

4. Read the right log. For vpxd, start with /var/log/vmware/vpxd/vpxd-alert.log, then move to vpxd.log. For STS, look in /var/log/vmware/sso/. For the proxy itself, /var/log/vmware/rhttpproxy/rhttpproxy.log. Grep for “error”, “certificate”, “No space left”, and “out of memory”.

5. Check certificate dates. Use the VECS commands from the quick checks. Then check the STS signing certificate specifically, because it has a different lifecycle from the machine SSL cert and is not visible in a browser. On vCenter 7.0 through 9.0, download Broadcom’s vCert package and use View Certificate Info → STS signing certificates. checksts.py is deprecated. For vCenter 6.x, use Broadcom’s version-specific recovery procedure.

6. Check vpxd startup progress. If vpxd recently restarted, planned or after a crash, it rebuilds its inventory cache from vPostgres. In large environments this can take 10-15 minutes. The SDK endpoint returns errors during this window. Look for cache rebuild entries in vpxd.log.

7. Check Envoy-sidecar on vCenter 8.0 U3. On 8.0 U3g and earlier U3 builds, Envoy’s default max_heap_size_bytes is 1 GiB; at 98% consumption it returns overload responses. The defect is fixed in 8.0 U3h.

Metrics and signals to monitor

Fixes

Group the fix by the cause you identified. Do not restart everything as a first move. On vCenter 7.x before 7.0 U3c, do not repeatedly run service-control --restart <service>: each invocation can leak two vMonTelemetry.db file descriptors in vmon until its descriptor limit is exhausted. Broadcom documents the defect as fixed in 7.0 U3c.

Disk full on a /storage partition

First identify which partition is full. Then find the largest consumer with du -sh /storage/<partition>/* (be aware of I/O impact on /storage/db).

For /storage/log, truncate the offending file rather than deleting it. Services hold open file descriptors, so deleting the file does not reclaim space until the service restarts.

Warning: this destroys log evidence. Capture relevant log lines first, or copy the file aside before truncating if you need it for support.

# Truncate, do not delete, a runaway log file
> /storage/log/vmware/vpxd/vpxd.log

Then address the underlying cause, which is usually a log bomb from another failing service. For /storage/db filling, check VCHA replication lag if applicable, and check SEAT table retention. For /storage/core filling, archive or remove old core dumps after capturing them for support.

Once space is reclaimed, vmon should restart the affected service. If it does not, start it explicitly with service-control --start <service> or vmon-cli --restart <service>.

Expired certificate

The fix depends on which certificate expired.

• Machine SSL cert expired. Use /usr/lib/vmware-vmca/bin/certificate-manager to renew. This requires service restarts and a maintenance window.
• STS signing cert expired. Follow the VMware KB procedure for STS certificate renewal. This is a documented but delicate procedure. Do not improvise.
• VMCA root cert expired. Renewing the root cascades to all dependent certificates. Plan for 60-90 days of lead time; do not attempt as a 3 a.m. fix.

After renewal, restart the services that cached the old cert. A “certificate renewed but still failing” state is common and means a service needs an explicit restart.

Service crash-loop

Read the service log to find why it is crashing. Common patterns:

• vpxd cannot reach vPostgres (check vmware-vpostgres status and /storage/db)
• vpxd OOM (check VCSA memory, swap usage, and whether the inventory has outgrown the deployment size)
• STS Java heap exhaustion while processing a large smart-card CRL. On vCenter 6.7 U3 and later, Broadcom documents cloudvm-ram-size -l, then cloudvm-ram-size -C 1024 vmware-stsd, followed by an STS restart. Follow the KB safeguards because increasing one service’s allocation can reduce allocations to others.
• A service hit vmon’s max restart count and stays STOPPED. Start it explicitly after addressing the root cause.

vpxd still rebuilding

Wait. Watch vpxd.log for cache rebuild completion. If it takes longer than 15 minutes in a small environment, something else is wrong; check disk I/O latency on the VCSA datastore and vPostgres health.

Envoy-sidecar overload (8.0U3)

Patch to 8.0 U3h or later. If patching is not immediately possible, follow Broadcom KB 384498 to back up /etc/vmware-envoy-sidecar/config.yaml, increase max_heap_size_bytes, and restart envoy-sidecar. Review SDK client session counts and throttle noisy integrations.

Prevention

• Monitor every /storage partition, not just root. Alert at 85%, page at 95% or any partition at 100%.
• Track all certificate expiry dates, including STS signing. Page at expiry, ticket at 30 days for STS, plan at 60-90 days for VMCA root.
• Monitor vmon restart counts. A service that vmon restarts more than once in 15 minutes is a crash-loop, even if external probes miss it.
• Run an authenticated SDK probe, not just a TCP check. An unauthenticated /sdk request returning the WSDL only proves rhttpproxy and vpxd are listening; it does not prove auth works.
• Keep the VCSA VM off overcommitted hosts. CPU ready, memory balloon, and datastore latency at the hypervisor layer are invisible from inside the guest.
• Patch to current. Broadcom documents fixes for the vmon file-descriptor leak and the Envoy overload in specific patch levels.

How Netdata helps

Netdata’s value for a vCenter 503 is signal correlation: per-second metrics let you see the cascade instead of guessing which service failed first.

• Correlate disk utilization on each /storage partition with vpxd service state. This distinguishes the documented database-storage protection threshold from failures caused by another full partition.
• Layer VCSA guest metrics under ESXi host metrics. A vpxd crash that coincides with a spike in host CPU ready or memory balloon on the VCSA VM points at hypervisor contention, not a vCenter bug.
• Track certificate expiry as a first-class signal. Page-level alerts at 30 days for STS and 14 days for machine SSL give time to plan renewal instead of reacting at expiry.
• Watch vmon restart counts as an anomaly. A service moving from zero restarts to several in an hour is the earliest crash-loop signal, before the service stays down.
• Pair SDK probe latency with vpxd CPU and task queue depth. When SDK latency rises alongside vpxd CPU saturation, you are looking at overload, not a crash.
• Alert on log generation rate per service. A log bomb that precedes a 503 often shows up as a 10-100x spike in log volume minutes before the service fails.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong
• vSphere HA ‘Insufficient resources to satisfy configured failover level’: admission control
• vSphere ESXi hardware health: ECC errors, fan failure, and thermal throttling