vSphere monitoring setups cluster into four maturity levels. Where you sit determines which incidents you catch early, which ones you discover after users complain, and which ones you never see coming until the management plane is already down. Use this as an assessment tool: read the signal lists for your current level and the one above, identify the gaps, and prioritize closing them.
Levels are cumulative. A team at Operational monitors everything in Survival plus operational signals. The progression tracks which failure modes you can detect before they become outages, not how sophisticated your tooling is.
flowchart TD
L1["Level 1: Survival
Is it alive?"]
L2["Level 2: Operational
Is it degraded?"]
L3["Level 3: Mature
Why is it degraded?"]
L4["Level 4: Expert
What will break next?"]
L1 --> L2 --> L3 --> L4Level 1: survival
Survival monitoring answers one question: is vSphere running? You know when a host is disconnected, when a VM is powered off, when a datastore is full, or when vCenter is unreachable. You do not know why performance is degrading, and you cannot detect resource contention until it becomes an outage.
What you can detect:
- Host and VM liveness. ESXi host connection state (connected, disconnected, notResponding) and VM power state (poweredOn, poweredOff).
- Datastore free space. Percentage and absolute bytes remaining.
- vCenter reachability. Service status (vpxd up or down) and SDK/API availability via HTTPS probe to
/sdk. - Certificate expiry dates. Machine SSL and STS signing certificate validity periods. You can catch the most common preventable vCenter outage before it happens.
- Aggregate host utilization. CPU and memory consumed at the host level. You know when a host is loaded but not whether individual VMs are being starved.
What you cannot detect:
- CPU ready time or co-stop. A VM can be starving for pCPU time while aggregate host CPU looks moderate. The guest OS has no idea it is being descheduled.
- Memory reclamation cascade. Ballooning, compression, and host-level swap are invisible if you only watch aggregate memory percentages. A host at 85% memory might be fine or it might be actively swapping.
- Storage latency breakdown. You know the datastore is not full, but you do not know whether device latency (DAVG) is spiking or kernel latency (KAVG) is queuing.
- Snapshot accumulation. The datastore fills before you know a two-week-old snapshot exists.
The survival trap is reactivity. Every alert is either a page-worthy outage or noise. The first sign of a memory pressure cascade is the page that fires when swap-in has already started.
Level 2: operational
Operational monitoring adds the signals needed to diagnose the most common production incidents: CPU contention, memory pressure, storage latency, snapshot growth, and cluster health problems.
What you add:
| Signal | What it catches |
|---|---|
| CPU ready time per VM (cpu.ready.summation) | Hypervisor-level CPU starvation invisible to the guest OS |
| Memory balloon and swap (per host and per VM) | The memory reclamation cascade before it becomes catastrophic |
| Datastore latency: GAVG, DAVG, KAVG | Where in the I/O path latency occurs (array, VMkernel, or both) |
| Snapshot age and count per VM | Forgotten snapshots before they fill the datastore |
| HA cluster health and host status | HA protection gaps, isolation events, admission control violations |
| DRS balance and migration count | Cluster imbalance, DRS thrashing, constraint conflicts |
| Per-partition disk usage on VCSA | Specific partition filling (/storage/db, /storage/log) while root looks fine |
| NTP synchronization offset | Time skew before it breaks SSO authentication and certificate validation |
The operational advantage is diagnosis. You can tell whether an “everything is slow” incident is storage latency (DAVG high, KAVG normal means the array is the problem) or VMkernel queuing (KAVG high, DAVG normal means the problem is in the ESXi I/O stack). You can detect ballooning before it escalates to compression and swap.
What you still miss at operational:
- Co-stop. Multi-vCPU VMs suffer co-scheduling delays that ready time alone does not reveal. A 16-vCPU database with high ready might actually be a co-stop problem.
- NUMA locality. A large VM spanning NUMA nodes silently loses memory throughput. The guest OS cannot see this.
- Queue depth saturation. High latency with no queue context tells you there is a problem but not whether the device queue is full.
- vPostgres internals. Database growth and bloat are visible at the filesystem level, but connection exhaustion, dead tuple accumulation, and WAL growth are not.
- CPU limits (max-limited). A VM with a forgotten CPU limit shows low ready time but is being throttled. If you only check ready, you miss this entirely.
Level 3: mature
Mature monitoring covers the signals that a professional SRE team needs for full incident diagnosis and proactive capacity planning.
What you add:
| Signal | What it catches |
|---|---|
| CPU co-stop (cpu.costop.summation) | Multi-vCPU VM scheduling overhead, usually from oversizing |
| CPU max-limited (cpu.maxlimited.summation) | Forgotten CPU limits silently throttling production VMs |
| NUMA locality percentage (numa.local vs numa.remote) | Memory accesses crossing the interconnect, adding latency per access |
| Memory compression rate (mem.compressionRate.average) | The danger zone between ballooning and host-level swapping |
| Outstanding I/Os and queue depth (QUED, ACTV) | Queue saturation as the direct cause of elevated KAVG |
| Per-virtual-disk latency (read and write separately) | Per-vdisk problems vs datastore-wide problems |
| vPostgres database size and table-level growth | Database bloat from stats retention or failed purge jobs |
| vpxd task queue depth and response time | vCenter overload before it becomes unresponsive |
| VMkernel log error rate | Earliest signal of hardware or driver failure |
| Capacity trending and runway estimation | When each resource will hit its cliff, expressed in days |
The mature advantage is precision. You detect the vCPU oversizing spiral: a 16-vCPU database VM has high co-stop because the scheduler cannot find 16 simultaneously-free pCPUs, and you know the fix is to reduce vCPU count, not add more. You catch NUMA locality violations before someone files a ticket about “the database being slow since the migration.” You monitor queue depth alongside latency, so you know whether high GAVG is a device problem (low queue, slow array) or a saturation problem (full queue, too much demand).
What you still miss at mature:
- SCSI sense codes. When storage fails, the specific sense key and ASC/ASCQ tell you exactly what happened. But these live in
vmkernel.logand most teams never parse them systematically. - STS internals. You monitor SSO failure rate, but you do not see the Java heap pressure or garbage collection pauses that cause intermittent token validation failures.
- vmon restart counts. A service that crashes and restarts within seconds may be invisible to external probes, but the restart count reveals a crash loop.
Level 4: expert
Expert monitoring adds the deep signals that operators typically instrument after a major incident taught them what they were missing. These are the signals that turn multi-hour investigations into minutes.
What you add:
| Signal | What it catches |
|---|---|
| Per-VM SCSI sense code tracking | Specific storage failure modes via sense key, ASC/ASCQ, and command opcode |
| VMkernel world CPU utilization | Helper and system worlds (vMotion helper, NFS client, iSCSI initiator) consuming pCPU |
| STS Java heap utilization and GC frequency | Intermittent SSO failures caused by GC pauses, not certificate expiry |
| vmon restart counts per service | Silent crash loops invisible to external availability probes |
| VCHA replication state and lag | Whether failover would actually work and how much data would be lost |
| vmdir replication health | SSO topology divergence between vCenter nodes in Enhanced Linked Mode |
| vPostgres checkpoint frequency and duration | I/O storms from checkpoints competing with query I/O |
| vPostgres autovacuum effectiveness per table | Tables where vacuum is falling behind, causing bloat and slow queries |
| Storage path state change rate | Path flapping, indicating unstable fabric or misconfigured multipathing |
The expert advantage is speed. You read SCSI sense codes in vmkernel.log and can distinguish medium errors from hardware errors from unit attention conditions based on the sense key, without waiting for a storage team to interpret array logs. You catch STS heap pressure before it causes the intermittent authentication failures that are notoriously hard to diagnose. You know whether vmon has restarted vpxd three times in the last hour, even if each restart was brief enough that the SDK probe never failed.
The cost of staying at each level
| Level | What kills you | Typical detection time |
|---|---|---|
| Survival | Certificate expiry cascade, snapshot growth, memory death spiral | Hours to days after user impact |
| Operational | NUMA penalties, co-stop from oversizing, queue saturation, vPostgres bloat | Minutes to hours after impact |
| Mature | SCSI sense code misinterpretation, STS heap pressure, silent crash loops | Root cause takes hours despite detection |
| Expert | Proactively caught before user impact | Minutes |
The gap between Survival and Operational is the largest single improvement a team can make. Moving from Operational to Mature catches the “mystery slow VM” problems that ready time and aggregate latency cannot explain. Moving from Mature to Expert collapses investigation time for storage failures, SSO degradation, and silent service crashes.
How Netdata helps
Netdata collects vSphere performance counters at per-second resolution and applies anomaly detection across the signals at each maturity level:
- CPU contention correlation. CPU ready, co-stop, and max-limited collected per VM. Anomaly detection flags the VM where ready time spikes while host CPU is moderate.
- Memory cascade detection. Balloon, compression, and swap rates tracked together. The ML layer detects the cascade pattern (balloon rising, then compression, then swap) before swap-in begins.
- Storage latency breakdown. DAVG, KAVG, and GAVG collected per device and per VM; confirm whether QUED and ACTV queue counters are exposed natively before treating queue context as collected.
- vCenter health. Per-partition disk usage on VCSA and certificate expiry tracked alongside host and VM performance metrics; vpxd queue depth and vPostgres database size are custom checks unless verified as native collector metrics.
- Composite pattern detection. The memory reclamation cascade, storage latency cliff, and CPU oversizing spiral are multi-signal patterns. Per-second resolution makes these patterns visible in the anomaly timeline without manual dashboard correlation.
Related guides
- vSphere monitoring checklist: the signals every host, VM, and vCenter needs
- How vSphere actually works in production: a mental model for operators
- vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
- vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
- vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
- vSphere vCPU oversizing: why adding vCPUs made the VM slower
- vSphere NUMA locality low: wide VMs paying the remote-memory tax
- vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
- vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
- vSphere memory compression: the reclamation tier between balloon and swap
- vSphere active vs consumed vs granted memory: why the percentage lies
- vSphere memory reclamation cascade: balloon to compress to swap in minutes