vCenter slow and unresponsive: vpxd overload and the task queue backlog

vCenter is sluggish. The vSphere Client hangs on login, PowerCLI calls time out, and tasks that normally finish in seconds sit in Running for minutes. ESXi hosts start flipping to Not Responding even though the hosts themselves are healthy and VMs keep running. This is the vpxd overload cascade, and it is one of the most commonly misdiagnosed vCenter incidents.

The reflex is to restart vpxd. That is usually wrong. Restarting clears the queue for 5 to 15 minutes (the time it takes vpxd to rebuild its inventory cache from PostgreSQL), but whatever saturated the task pool will refill it the moment vpxd is back. You end up in a slower, noisier version of the same incident, with less forensic evidence left in the logs.

The right move is to identify the source of concurrent work, throttle or pause it, and let vpxd drain the backlog.

What this means

vpxd is a large multithreaded C++ daemon that maintains an in-memory cache of the entire inventory and dispatches every management operation as a task. It runs an internal Long-Running Operation (LRO) job queue with a finite worker pool. When too much concurrent work arrives at once, the pool saturates: new tasks queue, the SDK becomes slow, and host heartbeat processing falls behind. After the heartbeat timeout (default 120 seconds, configurable via config.vpxd.heartbeat.notRespondingTimeout), vCenter marks hosts Not Responding. The hosts are fine; vpxd just cannot service the heartbeat channel.

Two overload outcomes have very different signatures:

• Thread pool exhaustion manifests as tasks stuck in Queued. vpxd is not picking up new work. The offending load is usually a flood of API calls from a single client.
• Slow execution manifests as tasks stuck in Running for far longer than baseline. vpxd is executing, but slowly. The cause is usually downstream: database latency, storage latency on the VCSA VM, or stats rollup contention.

In the terminal stage, vpxd exhausts its hard memory limit and panics with Memory exceeds hard limit. Panic, then restarts. On a full LRO queue you may also see vmodl.fault.SystemError with reason Too many outstanding operations returned to SDK callers (confirmed in Broadcom KB 318208).

flowchart TD
    A["Concurrent work spike
backup, scripts, DRS storm"] --> B["vpxd task pool saturates"]
    B --> C["Tasks queue"]
    C --> D["SDK response time climbs"]
    D --> E["Clients retry, adding load"]
    E --> B
    B --> F["Host heartbeat processing delayed"]
    F --> G["Hosts appear Not Responding"]
    B --> H{"Memory pressure?"}
    H -- yes --> I["Memory exceeds hard limit
vpxd panics, vmon restarts"]
    H -- no --> J["Stuck degraded until source throttled"]

Common causes

Quick checks

Start with read-only checks. Do not restart anything yet.

# Confirm vpxd is actually running (not crash-looping)
/usr/lib/vmware-vmon/vmon-cli --status vpxd

# Overall partition health - the most common adjacent killer
df -h /storage/log /storage/db /storage/seat

# Check the authenticated SDK endpoint, not just the WSDL
time curl -sk -X POST https://<vcsa>/sdk \
  -H "Content-Type: text/xml" \
  -d '<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
      <RetrieveServiceContent xmlns="urn:vim25">
        <_this type="ServiceInstance">ServiceInstance</_this>
      </RetrieveServiceContent>
    </soap:Body>
  </soap:Envelope>'

# Count tasks in each state - distinguishes Queued vs Running overload
Get-Task | Group-Object State

# See what is actually running right now
Get-Task | Where-Object {$_.State -eq "Running"} | Select Name, StartTime, EntityName

# Identify which hosts vCenter currently considers unreachable
Get-VMHost | Where-Object {$_.ConnectionState -ne "Connected"} | Select Name, ConnectionState

# Database latency indicator from vpxd profiler entries
grep "\[VpxProfiler\]" /var/log/vmware/vpxd/vpxd.log | grep "DoHostSyncTime" | tail -20

# ContainerView create vs destroy balance - imbalance indicates a leak
# NOTE: field position is version-dependent; adjust the awk column as needed
grep "createContainerView" /var/log/vmware/vpxd/vpxd.log | grep "BEGIN" | wc -l
grep "vim.view.View.destroy" /var/log/vmware/vpxd/vpxd.log | grep "BEGIN" | wc -l

How to diagnose it

1. Confirm overload, not a sibling failure. Certificate expiry, disk full on /storage/db, and vPostgres being down all produce similar symptoms. Run the quick checks above. If vpxd is up, partitions are not at 100%, and the SDK probe returns slowly rather than refusing connection, you are in overload territory.

2. Distinguish Queued from Running. This single distinction tells you where to look next.

   • Mostly Queued with low vpxd CPU: thread pool exhausted by a flood of API calls. Look for a single offending SDK client.
   • Mostly Running with high vpxd CPU or high DB latency: vpxd is genuinely compute- or database-bound. Look at stats rollup, DRS, or storage.

3. Trace the offending session. vpxd-profiler.log records per-session metrics including ClientIP and Username. Find a session ID that appears in the slow task entries, then trace it.

   find /var/log/vmware/vpxd/ -iname "vpxd-profiler*" -type f \
     -exec grep -H "<session-ID>" {} \; | grep "ClientIP" | head -n 5
   

   Correlate ClientIP with your backup server, monitoring tool, or automation host.

4. Check the ContainerView balance. If createContainerView counts dwarf destroy counts, a third-party monitoring or backup integration is leaking views. This is a common cause of vpxd memory exhaustion crashes.

5. Check the database side. Look for [VpxProfiler] entries with DoHostSyncTime values well above your environment’s baseline. Confirm from the ESXi host running the VCSA VM: DAVG/cmd in esxtop above 25 ms sustained means the underlying datastore is slow.

6. Check the stats rollup cadence. If the 5-minute rollup job is taking longer than 5 minutes, it overlaps the next interval and competes with live operations.

   # Read-only query. Do not modify VCDB tables directly.
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c \
     "SELECT sample_time FROM vc.vpxd_hist_stat1 ORDER BY sample_time DESC LIMIT 1;"
   

Metrics and signals to monitor

Fixes

Throttle the source first

Before changing anything on vCenter, find the client identified in step 3 of diagnosis and throttle it.

• Backup solution: pause the job. Most enterprise backup products have a per-vCenter concurrency setting. Veeam, Commvault, and similar tools can saturate vpxd with concurrent snapshot operations across hundreds of VMs. Reduce the parallelism or shift the window off the stats rollup cadence.
• PowerCLI or custom automation: limit concurrent runspace count, add retries with backoff instead of tight loops, and ensure every createContainerView is paired with a destroy. A script that opens SDK sessions in a loop without closing them will exhaust vpxd within minutes in a large inventory.
• DRS storm: temporarily switch the cluster to manual. This stops new migrations immediately while you investigate. See the DRS thrashing guide.
• Stats rollup overlap: reduce the statistics level to 1 or 2. Level 3 and 4 generate more database write load and are a common slow-burn cause of vpxd pressure. Lower the level, let the backlog drain, and review whether the higher granularity was needed.

Memory pressure and the ContainerView leak

If vpxd.log shows Memory exceeds hard limit. Panic or the create/destroy check shows a wide imbalance, a third-party integration is leaking ContainerViews. Updating or patching the offending client is the durable fix. In the short term, throttling that client’s session count stabilizes vpxd without a restart.

Database fragmentation

When DoHostSyncTime is consistently high and vpxd_hist_stat* tables have a high dead tuple ratio, database fragmentation is the bottleneck. The remediation is a VACUUM (FULL, ANALYZE) on the affected statistics table. This locks the table and requires equivalent free space, so schedule it in a maintenance window. Confirm the table is the problem first by checking pg_stat_user_tables.n_dead_tup.

Under-provisioned VCSA

If inventory counts are approaching the deployment size maximums and overload symptoms appear under modest concurrent load, the appliance is undersized. Vertical scaling from Small to Large or X-Large resolves the chronic pressure. This is not a quick fix; plan it as a maintenance operation.

When to actually restart vpxd

Restart only when:

• vpxd is in a crash loop and vmon has given up restarting it.
• The queue is so deep that throttling the source will not drain it before the next business window.
• You have already captured the diagnostic evidence (profiler logs, task states, ContainerView counts).

A restart drops all in-flight tasks and requires 5 to 15 minutes to rebuild the inventory cache from PostgreSQL. In large environments, all hosts will briefly appear Not Responding during reconnection. Expect a follow-on DRS burst as catch-up migrations fire after vpxd is healthy again.

# Destructive: drops all in-flight tasks and disconnects all hosts temporarily
service vpxd restart

Prevention

• Right-size the VCSA for inventory and concurrency, not just host count. A heavily tagged environment with many distributed portgroups hits limits sooner than raw VM count suggests.
• Rate-limit SDK sessions per client. Know which integrations hold persistent sessions and audit them quarterly. Backup, monitoring, and orchestration tools are the usual offenders.
• Keep statistics at level 1 or 2 unless you have an active reason to raise it. Document the reason and the expected end date if you do.
• Monitor snapshot age daily. Backup jobs that fail to clean up snapshots create both datastore pressure and vpxd task load.
• Track vpxd-profiler.log session patterns proactively, not just during incidents. A baseline makes the offending client obvious during the next overload.
• Apply vCenter patches that resolve overload bugs. Several specific vpxd crash modes have been fixed in recent releases; check the release notes for your target version against current Broadcom KBs.

How Netdata helps

• Per-second vpxd CPU and memory metrics surface the saturation climb before the queue is fully exhausted, giving you a window to throttle the source rather than recover from a panic.
• Task queue depth and SDK response time correlated with vpxd CPU distinguish thread pool exhaustion from slow execution. Two lines on one chart replace several minutes of log grepping.
• Host connection state changes correlated with vpxd load make it obvious when Not Responding is a vCenter-side problem rather than a host failure.
• VCSA per-partition disk usage catches the adjacent failure modes (/storage/log log bombs, /storage/db database pressure) that often co-occur with overload or trigger it.
• Anomaly detection on vpxd error rate and SDK latency flags the slow-burn version of this incident, where task duration creeps upward over weeks as inventory outgrows the deployment size.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong