vCenter vpxd crash loop: the core service that keeps restarting

vpxd is the C++ core of vCenter Server. It holds the entire managed inventory in memory, dispatches every management task to ESXi hosts via hostd, runs DRS, executes statistics rollups, and serves every SDK client (vSphere Client, PowerCLI, Veeam, NSX Manager, Aria Operations, custom automation). When vpxd dies, vCenter is functionally down: no provisioning, no vMotion orchestration, no DRS, no HA reconfiguration. VMs already running on hosts keep running, and FDM still restarts them after a host failure, because HA does not depend on vpxd.

A crash loop is the worst-case vpxd failure. vpxd hits an unrecoverable condition (most often heap exhaustion on a large inventory), aborts, and vmon (vmware-vmon) restarts it. Each restart rebuilds the inventory cache from vPostgres: minutes on a small environment, tens of minutes on a large one. During that window every SDK client sees timeouts or stale data, hosts flap to “Not Responding” while vpxd re-establishes connections, DRS skips cycles, and HA admission control cannot be reconfigured.

The silent escalation is what catches teams. vmon has a per-service MAX RESTART count. After N failures it stops restarting vpxd and leaves it STOPPED, with no auto-recovery. The threshold is not documented publicly; Broadcom KB 401096 shows a “Crash count 9” entry before vmon gives up. Without monitoring vmon-cli --status vpxd directly, the final transition looks like a hard outage with no obvious trigger.

What this means

A crash loop has three properties that separate it from a single restart.

Each restart cycle has a long blind window. While the inventory cache rebuilds, the SDK endpoint may answer unauthenticated probes but return errors or partial data to authenticated clients. Hosts mass-flap to “Not Responding” and back. Do not page on these signals alone; correlate with vmon crash-count entries.

Core dumps accumulate. Each vpxd crash writes a core file to /storage/core (sometimes /var/core). A tight loop can fill this partition in minutes, which then causes secondary failures: no useful cores captured for support, and other services unable to write.

vmon gives up silently. The service transitions to STOPPED with no further recovery attempts. The SDK endpoint stays down.

The classic OOM signature in vpxd.log is Panic: Memory exceeds hard limit with the memory checker logging Current value X exceeds hard limit Y. Shutting down process. vmon’s per-restart entry is Service exited unexpectedly. Crash count N. Taking configured recovery action. (Broadcom KB 380921). Watch the crash count.

flowchart TD
  A[vpxd STARTED] --> B{Hits unrecoverable
condition?}
  B -- No --> A
  B -- Yes --> C[vpxd crashes
heap, DB, cert, segfault]
  C --> D[vmon logs:
Crash count N]
  D --> E[vmon restarts vpxd]
  E --> F[Inventory cache rebuild
from vPostgres
minutes to tens of minutes]
  F --> G[SDK errors, hosts flap,
DRS skips cycles]
  G --> H{Crash count over
MAX RESTART?}
  H -- No --> B
  H -- Yes --> I[vmon gives up
vpxd STOPPED
no auto-recovery]
  I --> J[/storage/core fills
with core dumps]

Common causes

Quick checks

Safe read-only commands to run from the VCSA shell over SSH:

# Current vmon-managed service state
/usr/lib/vmware-vmon/vmon-cli --status vpxd

# Every service and its current state
/usr/lib/vmware-vmon/vmon-cli --list

# Legacy wrapper; may differ from vmon state on newer builds
service-control --status vmware-vpxd

# OOM panic and vmon crash-count entries
grep -iE "Panic|Memory exceeds|exited unexpectedly|Crash count" \
  /var/log/vmware/vpxd/vpxd.log /var/log/vmware/vpxd/vpxd-alert.log

# vmon's view of the crash count
grep -i "vpxd" /var/log/vmware/vmon/vmon.log | tail -50

# All storage partitions, not just root
df -h
df -i

# vpxd's current memory ceiling
cloudvm-ram-size -l

# SDK endpoint reachability (does not prove SOAP works, only that the listener answers)
time curl -sk -o /dev/null -w "%{http_code}\n" https://localhost/sdk

How to diagnose it

Work in this order. The first three steps route most incidents.

1. Confirm it is actually a crash loop, not a single restart. Pull the last 100 vmon log lines and count “Crash count” entries. A climbing count is a loop. A count reset to 0 with vpxd now STARTED means you had a single restart and the more interesting question is what triggered it.

2. Identify the crash signature in vpxd.log before doing anything else. The signature dictates the rest of diagnosis. The four common ones are:

   • Panic: Memory exceeds hard limit - heap exhaustion
   • Scheme error: '/' failed due to divide-by-zero - 8.0 U1 DRS/vGPU bug
   • Segfault with no preceding panic - dangling session pointer, OOM kill, or heap corruption
   • SystemError: Too many outstanding operations - LRO queue exhaustion (7.0.x)

3. If the signature is memory exhaustion, identify who is holding the inventory. Run the ContainerView profiler analysis to find the leaking session:

   cd /var/log/vmware/vpxd
   zcat vpxd-profiler*.gz | grep SessionStats | grep Container \
     | cut -d '/' -f5-10 | sort | uniq -c | sort -nr | head
   

   Then check whether the suspect session is destroying its views:

   zcat $(ls -1 vpxd*.gz | grep -v profiler) | grep <session_Id> \
     | egrep "ContainerView|View.destroy" \
     | awk -F " -- " '{print $4}' | sort | uniq -c
   

4. If the signature is not memory, check certificate validity for every cert class, not just the machine SSL:

   /usr/lib/vmware-vmafd/bin/vecs-cli store list
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A2 "Not After"
   python /usr/lib/vmware-vmca/bin/checksts.py
   

   The STS signing certificate is the one that takes vCenter down when it expires, and it is invisible from a browser.

5. Check vCenter version against the known crash-loop bugs listed in Fixes. If the crash signature matches a known bug and you are below the fixed build, the fix is to upgrade.

6. Check all /storage partitions, not just root. A crash loop with /storage/core at 100% means you have lost your ability to capture cores for support. /storage/log at 100% amplifies the original failure because services cannot write.

7. Only after exhausting the above, consider a database-level problem. Connect to vPostgres and look at connection count vs max_connections, dead tuple ratio on major tables, and WAL directory size.

Metrics and signals to monitor

Fixes

Grouped by cause. Try the cause-matching fix first; the version-specific bugs only respond to upgrade.

Heap exhaustion from inventory size or third-party load

First, raise the heap. The supported path is cloudvm-ram-size. A reasonable starting point is to double the current limit of the failing service:

# Show current per-service limits
cloudvm-ram-size -l

# Set a new limit for vpxd (size in MB)
cloudvm-ram-size -C <sizeInMB> vmware-vpxd

# Restart vpxd to apply
vmon-cli -r vpxd

Two caveats operators get wrong:

• Document this change. vCenter updates revert cloudvm-ram-size to default. Re-apply the override after every patch or upgrade, or vpxd will crash-loop again at the next inventory spike (Broadcom KB 320871 calls this out explicitly).
• Raising the heap without removing the underlying leak only delays the next crash. Pair it with the third-party-client investigation below.

ContainerView leak from a backup, monitoring, or orchestration client

This is the most common cause of vpxd OOM in the field. The pattern is vim.view.ViewManager.createContainerView calls without corresponding vim.view.View.destroy calls. Leaked views accumulate in vpxd’s inventory cache until it hits the heap.

The fix is on the client side. Identify the leaking session with the profiler command in diagnosis step 3, map it to a client (PowerCLI session, Veeam proxy, Aria Operations adapter, NSX-vSphere Adapter), and either patch the client, throttle its polling, or remove it. The NSX-vSphere Adapter in Aria Operations is a known offender with no fix because the adapter is end of life; the only resolution is to stop using it (Broadcom KB 379179).

Version-specific bugs

Match the crash signature to the known bugs and upgrade past the fixed build.

• 8.0 U1 divide-by-zero in DRS attempting vMotion on a vGPU VM with no vMotion bandwidth. Crash string: Scheme error: '/' failed due to divide-by-zero. Fixed in 8.0 U1b build 21860503. (KB 318166)

• 8.0 U3x dangling session pointer segfault when an account logs in with an already-authenticated session. Fixed in 8.0 U3e build 24674346. Workaround if you cannot upgrade immediately: add <authorize><sessionCanOutliveToken>true</sessionCanOutliveToken></authorize> to /etc/vmware-vpx/vpxd.cfg and restart vpxd. (KB 380921)

• 8.0 startup thread exhaustion on environments with more than 500 hosts using VM encryption or vTPM. Encryption health checks exhaust the internal thread pool on boot and vmon times out vpxd. Workaround: set all hosts to disconnected in VCDB, start vpxd, then bulk-reconnect hosts. No permanent fix as of the KB update in April 2026. (KB 434860)

  WARNING: destructive. Never run this against a production VCDB without a snapshot and an open Broadcom support case.

  UPDATE vpx_host SET enabled = 0;
  

• 8.0 U2/U3 vCLS version mismatch after snapshot revert from U3 to U2. vCLS VMs end up with a higher product version in the DB, triggering NoCompatibleHost faults. Workaround: power off and unregister affected vCLS VMs on the hosts. (KB 378716)

• 7.0.x LRO queue exhaustion when the LRO job queue is full and vScheduleCheckVsanConfigLro fires. Fixed in 7.0 U3i build 20845200. (KB 318208)

• 8.0.3 SMS thread pool starvation from high SPS activity. Default SMS core pool size of 20 is reached and starvation causes vpxd OOM. (KB 416955)

Database saturation

If vpxd cannot get connections or queries time out, address the database. Common actions: reduce statistics level to 1 or 2, reduce event and task retention, run VACUUM on bloated tables only with VMware support guidance, and verify /storage/db has at least 30% free. Full procedure is outside the scope of this article.

Disk full on /storage/core or /storage/log

Truncate, do not delete, the offending log file so the file descriptor stays valid. Then address the underlying log generator. Delete captured core files after you have collected them for support. Re-running df -h every 60 seconds during a crash loop is the only reliable way to catch this before it cascades.

Duplicate VM row in VCDB after failed storage vMotion

Documented in KB 401096. The duplicate, inaccessible VM row causes vpxd to crash on every start. This requires Broadcom support to confirm and remove the duplicate row from the database. Do not edit VCDB directly without support.

Prevention

• vmon crash count per core service: a climbing count is a precursor to vmon giving up silently.
• Certificate expiry tracking for STS, machine SSL, solution user, and VMCA root independently: STS expiry is the most devastating and easiest to miss because it is invisible from the browser.
• SDK session count by client IP: a new integration that leaks ContainerViews is the most common crash-loop source.
• At least 40% free on /storage/log and 30% free on /storage/db: both partitions can turn a single vpxd restart into a cascade.
• VCSA on a non-overcommitted host with a memory reservation: hypervisor-level CPU ready and balloon must be monitored, not just guest CPU and memory.
• Stay patched: the 8.0 U1, 8.0 U3x, and 7.0 U3i bugs above are all fixed in current builds.
• Document every cloudvm-ram-size override in the runbook: re-apply it after every upgrade.
• Close CVE-2024-37079: a heap overflow in the DCE/RPC protocol on TCP ports 2012, 2014, and 2020, actively exploited and a remote-code-execution vector that itself induces crashes. Patched builds close the hole.

How Netdata helps

• Per-second tracking of the vpxd PID and process state catches restarts that synthetic probes miss. A five-minute SDK probe can completely miss a tight crash loop.
• VCSA per-partition disk utilization on /storage/log, /storage/db, and /storage/core surfaces the secondary failures that turn a single vpxd crash into an outage.
• vpxd.log error-pattern tracking, including “Panic”, “Memory exceeds”, “Crash count”, and certificate warnings, lets you see the signature before the crash propagates.
• Hypervisor-level metrics on the VCSA VM itself (CPU ready, balloon, swap, datastore latency) distinguish “vpxd overloaded” from “VCSA starved by its host.”
• SDK response time and SDK session count by client let you identify the misbehaving integration before the next crash.
• ML anomaly detection on vpxd RSS, vpxd CPU, and SDK latency surfaces the gradual drift that precedes the cliff.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong
• vSphere HA ‘Insufficient resources to satisfy configured failover level’: admission control