How vSphere actually works in production: a mental model for operators

vSphere is a layered virtualization stack with three interdependent planes, and most production incidents cross plane boundaries. A guest that “feels slow” may be starved at the hypervisor scheduler, throttled by a forgotten CPU limit, fighting for IOPS at the storage layer, or sitting behind a vCenter whose database has bloated to the point that DRS stopped rebalancing the cluster.

The runbooks treat each failure mode individually. This article is the connective tissue: how to read a symptom in one plane and trace it to its origin in another.

The three planes are: the ESXi hypervisor plane (the VMkernel that owns the hardware), the VM plane (the sandbox and the VMware Tools channel that lets the hypervisor see into the guest), and the vCenter management plane (the orchestration layer that turns individual hosts into a cluster).

Why plane separation matters

The hypervisor plane is the only one with direct access to physical resources. CPU cycles, RAM, storage I/O, and network bandwidth are all arbitrated here by the VMkernel. The VM plane runs entirely inside the sandbox the hypervisor carves out and can only observe what the hypervisor chooses to expose. The management plane sits one layer further out and has no direct data-path role. When vCenter is down, running VMs keep running, DRS stops calculating, and HA still restarts VMs because the HA agents live on the ESXi hosts.

This separation is the source of both vSphere’s resilience and its debugging difficulty. Every metric you collect comes from a specific plane, and each plane has blind spots the others fill. Guest CPU utilization is reported by the guest and cannot see hypervisor descheduling. CPU ready time is reported by the hypervisor and cannot see guest-internal lock contention. vCenter task duration is reported by vpxd and tells you nothing about whether the host underneath is healthy.

How it works

flowchart TD
  subgraph Mgmt["vCenter management plane"]
    vpxd[vpxd inventory cache]
    vpg[vPostgres: inventory, stats, events, tasks]
    orch[DRS every 5 min, HA FDM agents, vMotion]
  end
  subgraph Hyp["ESXi hypervisor plane (VMkernel)"]
    sched[CPU scheduler: shares, limit, reservation]
    mem[Memory: TPS, balloon, compress, swap]
    stor[Storage I/O: pvscsi, VMkernel SCSI, VAAI]
    net[Network I/O: vmxnet3, vSwitch, NIOC]
    numa[NUMA scheduler: home node locality]
  end
  subgraph VM["VM plane (sandbox)"]
    tools[VMware Tools / open-vm-tools]
    balloon[vmmemctl balloon driver]
    hb[guest heartbeat]
  end
  vpxd -->|SOAP over HTTPS via hostd| Hyp
  orch -->|placement and migration decisions| Hyp
  VM -->|inflate, heartbeat, stats| Hyp
  Hyp -->|counters and connection state| vpxd

The ESXi hypervisor plane

The VMkernel is a purpose-built microkernel that owns the hardware. Five subsystems matter to operators.

CPU scheduler. Maps virtual CPUs (vCPUs) to physical CPUs (pCPUs). Each VM is a schedulable “world” using proportional-share allocation with shares, limits, and reservations. For multi-vCPU VMs the scheduler must co-schedule, finding N free pCPUs simultaneously, which makes large VMs harder to place. The world states that matter in esxtop are RUN (executing), READY (runnable but waiting for a pCPU), COSTOP (co-scheduling wait), and WAIT (idle or blocked on I/O). READY and COSTOP are the two invisible-from-the-guest taxes on every overcommitted host.

Memory management. ESXi overcommits memory through a four-tier reclamation hierarchy invoked in order of increasing desperation. Transparent Page Sharing (TPS) deduplicates identical pages and is largely disabled by default since vSphere 6.0 for security. Ballooning inflates the vmmemctl driver inside the guest, forcing the guest to page internally. Compression compresses 4KB pages in host memory before swapping to disk. Host-level swapping writes VM memory pages to .vswp files on the datastore. The tiers have radically different performance implications, from invisible to catastrophic, and the cascade can complete in minutes during a workload spike.

Storage I/O path. VM disk I/O traverses the guest OS, virtual SCSI adapter (pvscsi or LSI), VMkernel SCSI stack, storage driver, and physical storage. The VMkernel maintains I/O queues at each layer with configurable depth. VAAI (vStorage APIs for Array Integration) offloads operations like clone, zero, and hardware locking to the array. When VAAI is broken or unsupported, those operations fall back to the host and consume CPU and IOPS.

Network I/O path. VM network I/O flows through the guest OS, virtual NIC (vmxnet3), port group on a vSwitch or dvSwitch, and physical uplink (vmnic). The VMkernel handles its own traffic (vMotion, NFS, iSCSI, management, vSAN) on separate VMkernel adapters. Network I/O Control (NIOC) and traffic shaping throttle bandwidth per traffic class. Without NIOC, a single VM can starve others on a shared uplink.

NUMA. Modern multi-socket servers are non-uniform memory access architectures. Each CPU socket owns a region of local memory, and accessing remote memory costs roughly 1.5-2x latency. ESXi’s NUMA scheduler tries to keep each VM within a single NUMA node. VMs that span nodes (“wide VMs”) experience silent performance degradation, typically 10-30% memory throughput loss, that is invisible in any single counter except locality percentage.

The VM plane

Each VM runs in a sandbox with emulated hardware. VMware Tools (or open-vm-tools) inside the guest provides the balloon driver, the guest heartbeat, time synchronization, quiesced snapshots, and metrics reporting. Without VMware Tools, the hypervisor is partially blind to guest health: ballooning cannot run, so the host skips directly to compression and swapping, and HA VM Monitoring has no signal to act on.

A “Tools OK” status in the vSphere Client does not guarantee the balloon driver (vmmemctl) is actually running. Outdated VMware Tools can report healthy while the balloon driver is non-functional, which silently removes the gentlest tier of memory reclamation and forces the host into compression and swap earlier than the counters suggest.

The vCenter management plane

vCenter Server (typically deployed as VCSA, the vCenter Server Appliance running Photon OS) is a set of interdependent Java and C++ services orchestrated by vmware-vmon, sitting atop an embedded PostgreSQL instance (vPostgres), fronted by the rhttpproxy reverse proxy, and secured by the STS (Security Token Service) for token-based authentication. The exact number of services varies by vCenter version and enabled features; use service-control --status or vmon-cli -l to list them on a given appliance.

The core daemon is vpxd, a large multithreaded C++ process that maintains an in-memory cache of the entire managed inventory (hosts, VMs, networks, datastores, clusters, resource pools, permissions, alarms) and dispatches every management operation as a SOAP-over-HTTPS task routed to hostd on each ESXi host. vPostgres stores the persistent state: inventory, statistics, events, alarms, and tasks. If vpxd restarts, it must rebuild the inventory cache from the database, which takes minutes to tens of minutes in large environments.

Three orchestration features sit on top of vpxd and matter for incident response.

DRS (Distributed Resource Scheduler). Evaluates VM placement every 5 minutes by default, comparing imbalance improvement against vMotion cost. It considers CPU and memory, not storage. After a vCenter outage or a host reconnect, DRS evaluates all placements at once, which can produce a migration storm for 1-2 hours.

HA (High Availability). Monitors host liveness using network heartbeats and datastore heartbeats, with master-slave election among the ESXi hosts. The HA agents (FDM) live on the hosts, so HA continues to restart VMs even during a vCenter outage. The datastore heartbeat is what distinguishes “isolated” (network down, host still running, split-brain risk) from “dead” (both heartbeats lost, safe to restart elsewhere).

vMotion. Live-migrates VM memory and device state between hosts. Memory is pre-copied iteratively and the final switchover involves a brief stun, typically under a second but potentially seconds for memory-dirty workloads. A stuck vMotion can leave a VM in a degraded state between two hosts, and aggressive DRS levels can trigger migration storms.

Where it shows up in production

Symptoms surface in one plane and the cause lives in another.

• Guest slowness with low guest CPU. Originates at the hypervisor scheduler. CPU ready time is the universal tax, but a forgotten CPU limit produces a different counter (cpu.maxlimited.summation) that ready time alone does not capture.
• Memory pressure cascade. Ballooning (moderate) precedes compression (noticeable) precedes swapping (catastrophic). The guest’s own internal paging caused by ballooning is invisible to ESXi.
• Storage latency cliff. Queue saturation or array-side slowdown pushes latency from under 5ms to 50-500ms. This affects every VM on a datastore simultaneously, unlike CPU or memory contention which is per-host.
• NUMA penalty. Large VMs spanning NUMA nodes silently lose memory throughput. Often discovered only after a hot-add of CPU or memory breaks alignment.
• Snapshot accumulation. Delta VMDKs grow until the datastore fills. The guest is completely unaware. Consolidation then stuns the VM, sometimes for hours.
• vCenter database bloat. Statistics, events, and tasks accumulate in vPostgres. vpxd slows, DRS skips cycles, and eventually vCenter becomes unresponsive while running VMs are unaffected.
• HA isolation or split-brain. A host loses the management network but VMs keep running. The isolation response policy determines whether VMs are shut down, restarted elsewhere, or left running and at risk of duplicates.
• Certificate expiration. STS or machine SSL certificates expire on fixed dates and cause cascading TLS failures. The STS signing certificate is the most devastating and is not visible in a browser.

Tradeoffs and common misuses

• Monitoring guest CPU instead of hypervisor CPU ready. A guest reporting 30% CPU with 15% ready time is running at roughly 85% of requested speed and the OS does not know it.
• Treating host memory as a percentage. A host at 85% consumed memory might be fine (no balloon) or in crisis (actively swapping). The percentage alone tells you nothing. Monitor reclamation indicators independently.
• Assuming vCenter being up means the cluster is healthy. vpxd can be running while DRS has stopped calculating because vPostgres is too slow to complete a rollup window.
• Sizing VMs for “more power.” A VM with more vCPUs than its workload uses is harder to co-schedule, accumulates ready time and co-stop, and runs slower. Right-sizing beats adding vCPUs.
• Forgetting that VCSA is a VM. If vCenter sits on an overcommitted host, hypervisor contention inside vCenter looks like vCenter slowness. A VCSA VM with low internal CPU but high ready time is severely degraded, and guest-only monitoring will miss it entirely.
• Ignoring the STS signing certificate. It has a different lifecycle from the machine SSL certificate, it is not visible in a browser, and it is the single most common cause of total management plane outages.

Signals to watch in production

How Netdata helps

The value is correlation across planes. Deploy Netdata agents inside performance-critical guests and on the VCSA VM to collect per-second data that makes inter-plane failures visible without stitching together multiple 5-minute dashboards.

• Per-second collection on CPU, memory, and disk I/O captures the reclamation cascade (balloon, compression, swap) and storage latency spikes that rollups average away.
• Anomaly detection flags gradual degradation (rising memory pressure, creeping latency) before it becomes an incident.
• Process and filesystem monitoring on VCSA tracks vpxd and vPostgres resource usage and catches /storage/db or /storage/seat filling before they take down vpxd.
• Layered dashboards let you pivot from a guest symptom to the underlying host and datastore signals in one place.

Related guides

• vSphere monitoring checklist: the signals every host, VM, and vCenter needs
• vSphere monitoring maturity model: from survival to expert
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere vCPU oversizing: why adding vCPUs made the VM slower
• vSphere NUMA locality low: wide VMs paying the remote-memory tax
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• vSphere memory compression: the reclamation tier between balloon and swap
• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere memory reclamation cascade: balloon to compress to swap in minutes