vCenter service will not start: vmon dependency order and the max-restart wall

When vCenter services refuse to start, the failure is rarely the service you see stuck STOPPED. vmon (the VMware service lifecycle manager) starts each child service in dependency order and supervises it with a per-service restart policy. If a low-level dependency like vPostgres or STS is down or slow, every service layered on top of it fails its health check, retries within vmon’s bounded budget, and then silently stops trying.

The operator-facing symptom is a vCenter that looks half-alive: service-control --status --all shows a cascade of STOPPED services, the vSphere Client times out or shows a white screen, and vmware-vpxd is the one everyone focuses on even though it is almost never the root cause. The real failure is one layer down, in vPostgres, STS, or vmon itself.

A freshly restarted VCSA takes 5 to 15 minutes to bring all services up in order. Transient STOPPED states during that window are expected.

What this means

vmon is the systemd unit vmware-vmon.service that spawns and supervises roughly 30 interdependent vCenter services. It reads per-service JSON configuration that declares startup order, health checks, timeouts, and recovery actions. Two properties drive most of the behavior you will debug.

Dependency order. Services have a declared dependency chain. The canonical chain for the core management daemon is:

vmware-vpostgres -> lookupsvc -> vpxd-svcs -> vmware-vpxd

with rhttpproxy also required for external access. If vPostgres is not accepting connections, every service above it in the chain will fail its health check. vmon will not skip ahead. It waits, retries, and eventually marks each dependent as failed in turn.

The max-restart wall. Each service has a recovery action profile that tells vmon what to do on crash and on health-check failure. A common default profile performs two RESTART_SERVICE actions and then switches to NO_ACTION; verify the JSON for the service and build. When the budget is exhausted, vmon leaves the service STOPPED. This is silent: there is no event, no alarm, and no further automatic recovery until an operator intervenes or the VCSA is rebooted.

flowchart TD
    A["vPostgres STOPPED"] --> B["vpxd-svcs cannot reach DB"]
    B --> C["vpxd health check fails"]
    C --> D["vmon retries per service policy"]
    D --> E{"vpxd healthy?"}
    E -- no --> F["retry budget exhausted"]
    F --> G["vmon: NO_ACTION"]
    G --> H["vpxd STOPPED silently"]
    H --> I["SDK returns errors to clients"]

Two things make this pattern confusing in practice:

• The service reported as failed is usually not the broken one. vpxd STOPPED almost always means vPostgres or STS is the real problem. Chasing vpxd first wastes the time you need to spend on the layer below.
• Some STOPPED services are intentional. Auto Deploy (vmware-autodeploy) is correctly STOPPED if you do not use it. A VCHA passive node runs a deliberately reduced service set. Knowing your expected service profile prevents you from chasing services that are stopped by design.

Common causes

Quick checks

Run these read-only before changing anything. They give you the dependency-layer view.

# Overall service state from vmon
/usr/lib/vmware-vmon/vmon-cli --list

# Status of the four core services
/usr/lib/vmware-vmon/vmon-cli --status vpxd
/usr/lib/vmware-vmon/vmon-cli --status vmware-vpostgres
/usr/lib/vmware-vmon/vmon-cli --status sts
/usr/lib/vmware-vmon/vmon-cli --status rhttpproxy

# Legacy wrapper; cross-check, may lag vmon state in newer versions
service-control --status --all

# Is the supervisor itself healthy?
systemctl status vmware-vmon.service
systemctl list-unit-files | grep vmware-vmon.service

# Disk space on the partitions that kill services when full
df -h
df -i

# Startup profile; ALL on standalone, reduced set on VCHA passive
cat /storage/vmware-vmon/defaultStartProfile

# Stray backup JSON files that crash vmon at parse time
ls -la /etc/vmware/vmware-vmon/svcCfgfiles/

Add a functional probe so you know what consumers actually see:

# Unauthenticated /sdk probe; proves rhttpproxy and vpxd are listening.
# Connection refused = rhttpproxy down; 503 = rhttpproxy up but vpxd not responding.
curl -sk -o /dev/null -w "%{http_code}\n" https://localhost/sdk

# Authenticated VAMI health endpoint (requires valid root credentials)
curl -sk -u 'root:<password>' https://localhost:5480/rest/appliance/health/system

How to diagnose it

1. Confirm you are outside the boot window. If the VCSA came up less than 15 minutes ago, wait. Services start in dependency order and the full set takes 5 to 15 minutes. Paging on transient STOPPED states during boot creates noise and masks real failures.

2. Find the lowest-layer STOPPED service. Start with vmon-cli --list and walk the dependency chain from the bottom. If vmware-vpostgres is STOPPED or unhealthy, that is your root cause and vpxd cannot start until it is fixed. If vPostgres is STARTED but vpxd is STOPPED, the failure is at the vpxd or STS layer.

3. Check the supervisor before the supervised. If no services are starting at all, vmon itself is the problem. Run systemctl status vmware-vmon.service and look for result: start-limit (systemd gave up restarting vmon) or a masked unit. A masked vmon unit is a known state after image-based backup restores.

4. Read the vmon log for the failed service. The per-service recovery action and the restart attempts are logged. Repeated RESTART_SERVICE entries followed by NO_ACTION means vmon has hit the max-restart wall and will not try again without intervention. Check /var/log/vmware/vmon/vmon.log and, on builds that emit it, vmon-syslog.log; message format varies by release.

5. Check disk space per partition. Use df -h, not the VAMI UI, which rounds aggressively. The relevant partitions are /storage/log, /storage/db, /storage/core, and /storage/seat. A single partition at 100% with the others healthy is the normal failure shape.

6. Check certificates if STS is involved. STS STOPPED with no disk pressure and no vPostgres issue is almost always a certificate problem. Check the STS certificate with Broadcom KB 318968 and inspect the STS_INTERNAL_SSL_CERT store with vecs-cli.

7. Check the startup profile if only some services start. Run cat /storage/vmware-vmon/defaultStartProfile. On a standalone vCenter it must contain ALL. If it contains HACore left over from a destroyed VCHA configuration, vmon starts only the minimal HA set and leaves the rest STOPPED.

8. Check for duplicate config files. Run ls /etc/vmware/vmware-vmon/svcCfgfiles/. Any file that is not a legitimate service config (.orig, -modified, .bak, editor swap files) can crash vmon at startup with a PANIC and a ServiceManager.cpp verify message.

Metrics and signals to monitor

Fixes

Recovery order matters. Fix the lowest-layer failure first, then let vmon bring the dependent services up in order. Restarting vpxd while vPostgres is still down just burns another restart attempt.

vPostgres down or unhealthy

Start here when vpxd will not start. Confirm vPostgres is the failure with vmon-cli --status vmware-vpostgres.

• If /storage/db is full, free space before restarting. Truncate oversized logs with > /path/to/logfile (do not delete the file), check WAL accumulation, and review stats and event retention. Do not run VACUUM FULL under pressure without free space equivalent to the table size.
• If vPostgres is STARTED but slow, check /storage/db I/O latency from the ESXi host perspective. A vPostgres health check that times out looks identical to vPostgres being down from vmon’s point of view.
• On vCenter 8.0 U2, a corrupted postgresql.conf (file near zero bytes) is a known failure mode. The health check fails with Failed to read health xml file: /dev/shm/vmware-postgres-health-status.xml. Broadcom KB 424252 documents replacing the corrupted postgresql.conf from a healthy same-version vCenter, or using the matching PostgreSQL sample file, then restoring permissions and restarting services.

Once vPostgres is healthy, restart the dependent services in order, or reboot the VCSA if the cascade is wide.

STS down (certificate or otherwise)

STS STOPPED with vPostgres healthy is almost always certificate-related.

• Run checksts.py to confirm which certificate is expired or corrupt.
• For an expired STS signing certificate, use the current Broadcom certificate procedure — KB 318968 for STS checks and KB 385107 vCert for scripted replacement. This is a documented but delicate procedure. Do not improvise.
• After renewal, services that cache the old certificate may need explicit restarts.

vmon hit the max-restart wall

When vmon has given up, the service stays STOPPED until you clear the state. Fix the underlying cause first, then restart the service or vmon:

# Start the specific service through vmon after fixing the root cause
service-control --start vmware-vpxd

# Or restart vmon itself to reset retry budgets across all services.
# Disruptive: briefly interrupts all managed services.
systemctl restart vmware-vmon.service

There is no --restart flag on service-control; you stop then start. Avoid repeated service-control --start calls on vCenter versions before 7.0 U3c because of a known file descriptor leak in vmon that eventually prevents new services from starting.

vmon itself masked or start-limited

# Check whether systemd masked the unit
systemctl list-unit-files | grep vmware-vmon.service

# Unmask if masked
systemctl unmask vmware-vmon.service
systemctl start vmware-vmon.service

# If systemd hit its own start-limit (distinct from vmon's per-service limit)
systemctl reset-failed vmware-vmon.service
systemctl start vmware-vmon.service

Wrong startup profile (VCHA orphan)

# Overwrites the startup profile. Standalone vCenter must start ALL services.
# Do NOT run this on a VCHA passive node; the reduced profile there is intentional.
echo -n ALL > /storage/vmware-vmon/defaultStartProfile
systemctl restart vmware-vmon.service

Duplicate JSON config files

# Inventory the config directory
ls -la /etc/vmware/vmware-vmon/svcCfgfiles/

# Move suspicious files out; do not delete until you are sure
mkdir -p /tmp/vmon-cfg-backup
mv /etc/vmware/vmware-vmon/svcCfgfiles/*.orig /tmp/vmon-cfg-backup/ 2>/dev/null
mv /etc/vmware/vmware-vmon/svcCfgfiles/*-modified.json /tmp/vmon-cfg-backup/ 2>/dev/null
systemctl restart vmware-vmon.service

StartTimeout too low

For services that consistently fail their health check during boot on slow I/O (notably statsmonitor), increase StartTimeout in the per-service JSON. The vpxd JSON documents a 300-second default in Broadcom KB 321336; other services vary, so read the specific JSON rather than assuming a common default. Editing these files is a VMware-support-guided change. Keep a backup and restart vmon afterward.

Prevention

• Monitor the four core services by name, not just the overall health roll-up. vpxd, vPostgres, STS, and rhttpproxy each warrant individual alerting. A green overall roll-up can hide a single failed core service during the boot window.
• Track vmon restart counts per service. A service that vmon has stopped retrying is invisible to anything that only checks whether the process is alive. Alert when the restart count stops climbing but the service remains STOPPED.
• Alert on per-partition disk usage, not root. VCSA splits data across /storage/log, /storage/db, /storage/core, /storage/seat, and others. Root can be healthy while one storage partition is at 100%.
• Track STS signing certificate expiry separately from machine SSL. The STS cert is invisible in a browser and is the one that takes down all authentication when it expires. Alert at 30 days minimum; 60 days is safer given the renewal procedure.
• Suppress service-state alerts during the boot window. A VCSA that came up in the last 15 minutes is expected to have transient STOPPED services. Page only on core services outside that window.
• Document the expected service profile for each VCSA role. Standalone, VCHA active, and VCHA passive have intentionally different service sets. Alerting on Auto Deploy STOPPED on a vCenter that does not use Auto Deploy is noise.
• Verify vmon state after any image-based restore. A masked vmon unit after restore is a known failure mode. Run systemctl list-unit-files | grep vmware-vmon before declaring a restore complete.

How Netdata helps

• Correlate vmon per-service state with /storage/* partition utilization in one view, so a vPostgres failure caused by a full /storage/db is obvious in seconds rather than inferred from separate dashboards.
• Per-second metrics on VCSA CPU, memory, and disk let you see the boot window as it happens and distinguish a normal startup ramp from a service stuck in a restart loop.
• Anomaly detection on vpxd log error rate and vmon restart count surfaces the max-restart wall before an operator notices the service has gone silent.
• Certificate expiry tracking, including the STS signing certificate, gives weeks of lead time on the most common root cause of STS-down cascades.
• NTP offset monitoring catches the clock skew that turns into SAML token validation failures, which otherwise look like STS problems.
• Synthetic SDK probes with latency tracking give ground truth on whether vCenter is actually serving consumers, separate from whether individual services report STARTED.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong
• vSphere HA ‘Insufficient resources to satisfy configured failover level’: admission control