vCenter appliance undersized: inventory outgrowing the deployment size

A VCSA deployed as “Small” three years ago is not the same as a fresh “Small” today if your inventory has grown. The VCSA’s resource footprint is not fixed by CPU and RAM alone. It is dominated by the size of the in-memory inventory cache that vpxd maintains, and that cache grows with managed-object count: VMs, hosts, datastores, dvSwitch portgroups, tags, alarms, permissions, and snapshots.

The symptom is rarely abrupt. Approaching the deployment-size maximum degrades gracefully first. The vSphere Client becomes sluggish, DRS cycles stretch past the five-minute interval, the SDK responds in seconds rather than sub-second, and tasks queue. Then the appliance cliff-edges: vpxd exceeds its heap, vmon restarts it, and the inventory cache rebuild from PostgreSQL takes minutes to tens of minutes in large environments. During that window the management plane is blind.

This article covers how to identify when inventory has outgrown the deployment size, how to count the objects that actually matter, and what to do about it before the cliff.

What this means

vpxd, the vCenter Server daemon, is a large multithreaded C++ process that maintains an in-memory cache of the entire managed inventory. Every VM, host, datastore, network, portgroup, cluster, resource pool, permission, alarm, and tag occupies memory in that cache. The cache is rebuilt from PostgreSQL on every vpxd start, which is why vCenter restarts are slow in large environments.

VMware publishes per-size maximums for each VCSA deployment size. For example, Small targets up to roughly 100 hosts or 1,000 VMs, and Large targets up to roughly 2,000 hosts or 35,000 VMs. These numbers are useful guidance, but they understate the real constraint. The effective limit is total managed objects, not just VMs.

A heavily customized environment with many tags, custom alarms, granular permissions, and thousands of dvSwitch portgroups hits the limit sooner than a simple environment with the same VM count. Snapshots also add managed objects: a backup policy that leaves snapshot chains in place inflates the effective inventory count beyond what the VM count alone suggests.

The progression looks like this:

flowchart TD
  A[Inventory below size cap] --> B[Graceful degradation]
  B --> C[Slower UI, longer DRS cycles]
  C --> D[Task queue builds]
  D --> E[SDK response in seconds]
  E --> F[vpxd memory headroom narrows]
  F --> G[vpxd exceeds heap]
  G --> H[vmon restarts vpxd]
  H --> I[Cache rebuild from PostgreSQL]
  I --> J[Management plane blind 5-15 min]
  J --> G

Common causes

Quick checks

All commands below are read-only. SSH to the VCSA as root or as a user with shell access.

# Filesystem health across all /storage partitions
df -h
df -i

# Managed object count by type from the inventory database
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT type_id, count(*) FROM vc.vpx_entity GROUP BY type_id ORDER BY count(*) DESC;"

# Total managed object count
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT count(*) FROM vc.vpx_entity;"

# vpxd RSS - the dominant memory consumer in the VCSA
cat /proc/$(pidof vpxd)/status | grep -E "VmRSS|VmSize|VmSwap"

# Top memory consumers to confirm vpxd dominates
ps aux --sort=-%mem | head -20

# Memory and swap pressure
free -h
cat /proc/meminfo | grep -E "MemTotal|MemAvailable|SwapTotal|SwapFree"

# SDK endpoint response time - ground truth for vCenter health
time curl -sk -o /dev/null -w "%{http_code} %{time_total}s\n" https://localhost/sdk

# Statistics level currently configured
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "
SELECT * FROM vc.vpx_parameter WHERE name LIKE '%stats%';"

For a snapshot count from a machine with VMware.PowerCLI:

# Get-VM | Get-Snapshot | Measure-Object
# Get-VM | Get-Snapshot | Where-Object { $_.Created -lt (Get-Date).AddDays(-3) }

How to diagnose it

1. Compare managed object count against the deployment-size maximum. A Small deployment with 1,200 VMs is past its guideline. A Small deployment with 950 VMs but 4,000 portgroups, 800 tags, and 2,000 active alarms is also past its effective limit.

2. Inspect the breakdown by type_id. The type_id mapping differs by version, but you can identify clusters of objects (networks, portgroups, alarms) that are disproportionate to the VM and host count. These are the objects that make a customized environment hit limits earlier than the raw VM count suggests.

3. Look at vpxd RSS against a baseline. Rough guideline: 2 to 4 GB for environments under 100 VMs, 8 to 16 GB for environments at 1,000 or more VMs. Numbers vary with customization and snapshot load. If vpxd RSS is climbing and the appliance is at 80 percent or more of physical memory, you are approaching the OOM cliff.

4. Check for swap activity on the VCSA. Any swap usage on the appliance is abnormal. Even a small constant level means memory-tight pages were pushed out, and those pages may be hot.

5. Check vmon restart counts for vpxd. Repeated silent restarts indicate the appliance is bouncing on the heap limit. vmon has a maximum restart count per service; once exceeded, it stops trying and leaves the service STOPPED. If vpxd has restarted more than once in 15 minutes, the sizing problem is no longer theoretical.

6. Check task execution duration. In an undersized appliance, VM power-on, vMotion, and snapshot operations queue and run slowly. Power-on over 30 seconds, vMotion over 5 minutes for a typical small VM, and snapshot creation over 2 minutes all indicate vpxd is bottlenecked.

7. Check SDK response time. A healthy VCSA responds to /sdk in under one second. Response time correlates strongly with vpxd load. Multiple seconds of response time without topology change is a strong sign of vpxd pressure.

8. Check statistics rollup lag. If the 5-minute rollup is taking 4 or more minutes, or the latest sample_time for any rollup level is more than two intervals behind current time, rollup is overlapping with management operations and compounding the load.

9. Confirm the VCSA VM itself is not contended at the hypervisor layer. A VCSA reporting low internal CPU but high CPU ready time at the ESXi host is being starved, not undersized. This is a different problem but presents with the same symptoms. VMware documentation recommends the VCSA VM be placed on a host with sufficient CPU and memory headroom to avoid contention.

Metrics and signals to monitor

Fixes

Resize the VCSA to a larger deployment size

This is the only permanent fix when inventory has genuinely outgrown the configured size. Resizing changes CPU, memory, disk allocation, and the vPostgres configuration. Treat resizing as a one-way operation: it cannot be reversed down once applied. Schedule a maintenance window, take a file-based VAMI backup first, and plan for the full vCenter service restart that follows.

Recent vSphere versions have introduced simplified deployment-size change procedures. Earlier versions require a restore-based resize or a Reduced Downtime Upgrade path. Read the version-specific Broadcom documentation before starting: the steps are not interchangeable between major versions.

After resize, vpxd must rebuild its inventory cache from PostgreSQL on first boot. In large environments this takes 5 to 15 minutes. Do not alert on missing inventory or SDK errors during the cache rebuild window.

Reduce inventory complexity without resizing

If you cannot resize immediately, reduce the effective managed-object count:

• Remove orphaned VMs and stale resource pools. These consume vpxd memory even when unused.
• Consolidate tags. Tag counts scale with the objects they apply to, so a tag attached to thousands of objects adds thousands of managed-object edges.
• Prune custom alarms. Alarms that have fired and cleared on hundreds of objects generate event entries and metadata.
• Clean up unused dvSwitch portgroups. Each portgroup is a managed object.
• Consolidate active snapshots. Every snapshot in the chain adds objects. Anything older than 72 hours in production should be reviewed.

This is triage, not a fix. It buys time before the resize.

Lower statistics level and retention

Statistics level 3 or 4 collects dramatically more data than level 1 or 2. If a consultant or operator set level 3 or 4 for troubleshooting and forgot to lower it, the database grows, the rollup jobs lag, and vpxd is starved for I/O and CPU during rollup windows.

Lower the statistics level to 1 or 2. Reduce retention periods for events and tasks. The purge job runs as an internal vpxd task; if vpxd is overloaded, purge falls behind and the database continues to grow. After lowering the level, monitor the vpxd_hist_stat* table sizes to confirm the purge is reclaiming rows.

Identify misbehaving SDK clients

Some integrations are particularly hostile to undersized VCSAs. Backup tools, monitoring platforms, and orchestration products maintain persistent SDK sessions and create inventory views (ContainerViews) that vpxd must hold in memory. A single misbehaving integration that creates views without destroying them leaks vpxd memory over time and can push it past its heap on a deployment that is otherwise within sizing limits.

Look at active SDK session counts by source IP. A single client opening hundreds of concurrent sessions is the most common cause of “vCenter is slow” that is not actually a sizing problem. Throttle or disconnect the offending client, then revisit whether the appliance still needs resizing.

Raise vpxd memory limit temporarily

The vpxd memory limit can be raised manually as a stopgap by adjusting the memChecker settings in vpxd.cfg (see /etc/vmware-vpx/vpxd.cfg). This is not a permanent fix: it masks the underlying sizing problem and the appliance will hit the new limit if inventory keeps growing. Use it only to bridge the gap between now and a scheduled resize.

Prevention

• Track managed object count monthly against the deployment-size maximum. The count is a single SQL query against vpx_entity. Compare it to the maximum for the configured size, not for the size you wish you had.
• Calculate runway in objects, not VMs. Divide remaining headroom by growth rate per month to get months until the limit. Plan resize at 80 percent of maximum, not at 100 percent.
• Track vpxd RSS as a function of managed object count. If RSS grows without inventory growth, suspect a leak or a misbehaving SDK client rather than a sizing problem.
• Track /storage/db growth rate. Alert on more than 1 GB per day in a stable environment with no new hosts or VMs.
• Track statistics level as a configuration signal. Any change to level 3 or 4 should trigger a capacity review of the database.
• Track active SDK session counts by client IP. A new integration that opens many sessions should trigger a review before it becomes the dominant load.
• Include the VCSA in capacity planning the same way you include ESXi hosts. It is infrastructure. Treat it as such.

How Netdata helps

• Per-second metrics on vpxd RSS, VCSA CPU, swap activity, and per-partition disk utilization surface the slow degradation that operators otherwise adapt to. Trending the vpxd RSS curve over weeks is the single most useful leading indicator for an undersized VCSA.
• Managed object count and per-type breakdown, collected periodically, let you compare inventory size against the deployment-size maximum and project runway in months.
• Task queue depth and average task duration expose user-visible impact before operators complain about a slow vCenter.
• vmon restart counts for vpxd catch silent crash loops that brief external probes miss.
• Statistics rollup lag and the vpxd_hist_stat* table sizes correlate vCenter slowness with database pressure rather than with hypervisor contention.
• Correlating SDK response time with vpxd CPU, vpxd RSS, and per-client SDK session counts distinguishes “vCenter is undersized” from “one integration is leaking sessions”.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong