vCenter CPU and memory pressure: vpxd heap, swap, and the undersized appliance

vCenter slows down before it fails. The vSphere Client takes 30 seconds to load a VM summary, API calls time out from backup and monitoring tools, hosts flap to “not responding” even though they are healthy, and DRS recommendations stop appearing. Inside the appliance, top shows vpxd consuming most of the memory and a small but non-zero amount of swap. From the hypervisor, the VCSA VM either looks busy or, worse, looks idle while sitting at 50% CPU ready time.

The root cause is almost always the same: the vCenter Server Appliance (VCSA) is undersized for the inventory it manages, and vpxd has run out of room. vpxd is a large C++ process that caches the entire managed inventory in memory. Its resident set size (RSS) scales with managed object count: hosts, VMs, portgroups, datastores, tags, alarms, permissions, snapshots, and every other object vCenter tracks. A VCSA correctly sized at deployment silently becomes undersized as the environment grows.

The dangerous failure mode is the swap death spiral. Unlike a generic Linux box where a small amount of swap is benign, any sustained swap on the VCSA is bad. The management services are not built to tolerate swap latency, and once swapping starts, vpxd slows down, clients retry, retries allocate more memory, and the spiral deepens until vpxd hits its hard memory limit and vmon restarts it. The restart then has to rebuild the inventory cache from PostgreSQL, generating even more memory pressure during the rebuild window.

What this means

There are two layers to examine independently, and the worst incidents happen when only one is monitored.

Inside the guest (Photon OS), the signals that matter are MemAvailable, swap usage, and per-process RSS for vpxd. MemFree is misleading because Linux uses most “free” memory for page cache; MemAvailable accounts for that and is the value to alert on. Any sustained swap activity is a paging emergency, not a tuning opportunity.

From the hypervisor, the VCSA is just a VM. It can be CPU-starved (high ready time), memory-contended (ballooning, host swap), or stuck on slow storage, and none of that is visible from inside the guest. The most common operator mistake is monitoring the VCSA only from inside the guest: a VCSA reporting 0% CPU usage can simultaneously be at 50% CPU ready time at the ESXi scheduler, which means it is starved. Never place the VCSA on an overcommitted host, and always give it a memory reservation so the host cannot reclaim its memory through ballooning or swap.

One nuance worth stating explicitly: vpxd memory climbs for 2 to 15 minutes after every restart while it rebuilds the inventory cache from PostgreSQL. Alert on the steady-state plateau, not the post-restart ramp.

flowchart TD
  A[Inventory grows or SDK leak] --> B[vpxd RSS climbs]
  B --> C[MemAvailable drops]
  C --> D[OS begins swapping]
  D --> E[vpxd task latency rises]
  E --> F[Clients retry, amplify load]
  F --> G[Swap thrash deepens]
  G --> H[vpxd hits hard limit or OOM]
  H --> I[vmon restart]
  I --> J[Cache rebuild, more pressure]
  J --> D

Common causes

Quick checks

These are safe, read-only checks. SSH to the VCSA as root.

# Overall memory and swap (use MemAvailable, not MemFree)
free -h
grep -E "MemTotal|MemAvailable|MemFree|SwapTotal|SwapFree" /proc/meminfo

# Swap-in/swap-out rate. Any sustained non-zero si/so is a paging emergency.
vmstat 1 5

# vpxd resident set, virtual size, and swap contribution
cat /proc/$(pidof vpxd)/status | grep -E "VmRSS|VmSize|VmSwap"

# Top memory consumers across all services
ps aux --sort=-%mem | head -20

# Per-service memory allocation as configured by cloudvm
cloudvm-ram-size -l

# vmon-managed service health
/usr/lib/vmware-vmon/vmon-cli --list

# Disk pressure on the partitions that kill vpxd when full
df -h /storage/db /storage/log

# Recent vpxd memory pressure signals in the log
grep -iE "out of memory|oom|cannot allocate|hard limit" /var/log/vmware/vpxd/vpxd.log | tail -50

From a host running the VCSA VM, the ESXi shell gives you the other half of the picture:

# CPU ready time, co-stop, memory balloon and host swap for the VCSA world
esxtop
# press 'c' for CPU, then 'm' for memory; sort by the VCSA world

How to diagnose it

1. Confirm the layer. Check free -h and SwapFree inside the VCSA first. If MemAvailable is healthy and SwapFree is unchanging, the issue is not guest-side memory; pivot to the hypervisor view in step 4.
2. Triage vpxd specifically. Compare VmRSS for vpxd against the per-service allocation from cloudvm-ram-size -l. If RSS is approaching the configured ceiling, vpxd is the dominant consumer; if RSS is modest but MemAvailable is still low, another service (vPostgres, a Java heap, the wcp service) is the cause.
3. Look for swap activity, not just swap usage. SwapFree can be constant while swap-in rate is non-zero (check vmstat 1). Any sustained swap-in is the death spiral starting.
4. Check the hypervisor side. On the host running the VCSA, esxtop should show near-zero %RDY and %CSTP for the VCSA world, zero MCTLSZ (balloon), and zero SWCUR (host swap). Anything else means the VCSA is being starved or reclaimed by the host.
5. Confirm the VCSA has a memory reservation. Without one, host memory pressure will balloon or swap the VCSA before any guest-side alarm fires.
6. Pull the managed object count and compare against the deployment size maximums. Inventory growth is the most common silent cause.
7. Identify the noisy SDK clients. Long-lived or repeatedly-created ContainerView objects from third-party tools are a known vpxd memory leak source. Broadcom ships a ViewLeakTracer.sh script for this.
8. Match the build number against known leak fixes. If the wcp service is the consumer and you are below 8.0 U3g (8.0 P06), the leak is documented and fixed in that release.

Metrics and signals to monitor

Fixes

Each subsection targets one cause. None of these should be your first move during an active incident; during the incident, stop the bleeding (pause backup jobs, throttle SDK clients, set DRS to manual if vpxd is overloaded) and then plan the structural fix.

Upsize the VCSA

If managed object count has crossed the deployment size boundary, the only durable fix is to redeploy or resize the VCSA to the next tier. vCenter 8.0 raised the memory baselines from 7.0 specifically because the older sizes were insufficient: Tiny requires 14 GB (was 12 GB), Small 21 GB (was 19 GB), Medium 30 GB (was 28 GB), Large 39 GB (was 37 GB). Tiny is explicitly intended for lab or test, not production. Do not manually reduce memory back to 7.0 levels; the services are sized against the new baselines and will swap.

Move the VCSA off the overcommitted host and reserve its memory

If the hypervisor view shows ready time, ballooning, or host swap on the VCSA, the fix is placement, not tuning. vMotion the VCSA to a host with headroom, give it a memory reservation equal to its full configured size, and keep DRS from moving it back onto a contended host. A VCSA at 0% guest CPU and 50% ready time is starved, and no amount of in-guest tuning will fix that.

Increase the vpxd heap with cloudvm-ram-size

For a known, bounded vpxd consumer that needs more headroom, you can raise the per-service allocation:

# Inspect current per-service allocation
cloudvm-ram-size -l

# Increase vpxd allocation (replace <MB> with the target size; plan a maintenance window)
cloudvm-ram-size -C <MB> vpxd

Two caveats. First, this treats a symptom, not the inventory growth underneath it; you will be doing this again unless the underlying growth is addressed. Second, vCenter updates can revert cloudvm-ram-size changes to defaults. Document the change so you can reapply it after patching.

Apply the version-specific fix

If the consumer is the wcp service and you are below 8.0 U3g (8.0 P06), update. Earlier releases require a wcp service restart as a workaround. Similarly, the vmware-pod service on 7.x and 8.x has a documented path where dying memory cgroups, especially with CPU Hot Add enabled, drive percpu memory usage; the workaround involves stopping or disabling vmware-pod, dropping caches, adding cgroup.memory=nokmem to the kernel command line, and disabling CPU Hot Add. All of these steps are disruptive and require a maintenance window.

Throttle SDK clients and clean up container views

If the heap pressure tracks a specific integration (backup, monitoring, custom automation), throttle it. Long-lived ContainerView objects that are not explicitly destroyed accumulate in vpxd memory; the offending client IP and username can be identified with the ViewLeakTracer.sh script. The durable fix is in the client code, not the VCSA.

Reduce stats collection and inventory sprawl

Statistics level 3 or 4 multiplies database and vpxd memory load. Drop to level 1 or 2 unless you have an active reason to keep it higher. Orphaned VMs and stale objects inflate the managed object count and therefore vpxd RSS; periodic inventory hygiene is preventative maintenance, not optional cleanup.

Prevention

• Memory reservation on the VCSA VM. Prevents the host from ballooning or swapping the management plane during unrelated host pressure.
• Dedicated, non-overcommitted host (or DRS rule). Keeps the VCSA off hosts that may transiently oversubscribe CPU or memory.
• Alert on MemAvailable, not MemFree. MemFree will read near zero on a healthy VCSA because of page cache; MemAvailable is the real headroom number.
• Alert on any sustained swap, not on a percentage. The VCSA services do not tolerate swap latency; treat any non-zero swap-in rate as an incident.
• Track managed object count against deployment size limits. The VCSA does not warn you when it crosses the line; you have to monitor it yourself.
• Alert on the vpxd steady-state plateau, not the post-restart ramp. Inventory cache rebuild takes minutes and will look like a leak if you are not aware of it.
• Monitor the VCSA from the hypervisor alongside the guest. CPU ready, balloon, and host swap on the VCSA world are invisible from inside Photon OS.

How Netdata helps

• Per-second MemAvailable and swap-in rate on the VCSA catch the death spiral in its first minute, before vpxd has time to slow down enough for clients to start retrying and amplifying the load.
• Per-process RSS for vpxd correlated against managed object count lets you distinguish inventory-driven growth from a leak, which determines whether the fix is upsizing or patching.
• VCSA VM-level metrics from the hypervisor (CPU ready, co-stop, balloon, host swap) sit next to the guest-side metrics in the same dashboard, so a VCSA that is starved at the scheduler is obvious without pivoting to esxtop.
• ML anomaly detection on vpxd RSS flags the slow trend that a static threshold misses, which is exactly the pattern of an environment quietly outgrowing its deployment size.
• Correlated views across the VCSA, its database disk, and the underlying hosts distinguish vpxd-driven memory pressure from disk-fill or host-side contention as the root cause.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong