vSphere datastore IOPS and throughput: spotting storage saturation before latency bites

vSphere exposes IOPS and throughput counters at the datastore and virtual-disk level, but most teams only look at storage metrics after VMs are already slow. By the time DAVG or GAVG spike, the device queue is saturated and every VM on the datastore is paying for it. IOPS and throughput are leading indicators: they tell you how hard you are pushing the backend before the backend pushes back.

These counters do not have universal thresholds. A datastore backed by a single SAS disk saturates around 150-200 random IOPS. A flash LUN can handle 100,000 or more. What matters is not the absolute IOPS value but where you sit relative to the backend’s ceiling, and what latency does as you approach it.

What IOPS and throughput actually measure

vSphere collects storage rate counters at two levels. Datastore-level counters aggregate across every VM and every VMDK on the datastore. VirtualDisk-level counters show per-virtual-disk rates, which is what you need to identify a noisy neighbor.

In esxtop, the same information appears as READS/s, WRITES/s, and CMDS/s per device or per VM world. Press u for the disk device view or v for the per-VM disk view.

Three measurement nuances that trip people up:

ESXi IOPS can differ from in-guest IOPS. The VMkernel SCSI stack can coalesce small guest I/Os before they reach the device. A guest issuing many 4KB reads may show fewer, larger I/Os at the esxtop level. Do not expect the numbers to match what iostat reports inside the VM.

CMDS/s is not the same as IOPS. CMDS/s counts all SCSI commands, including metadata operations like SCSI reservations, locks, and unit attention queries. During metadata-heavy operations such as VM cloning or snapshot consolidation, CMDS/s can spike well above actual data IOPS. For capacity planning, use READS/s and WRITES/s, not CMDS/s.

IOPS and throughput together reveal I/O size. Divide throughput by IOPS and you get the average I/O size. This matters because a backend’s IOPS ceiling depends on I/O size. The same disk that handles 200 random 4KB IOPS may hit its sequential throughput ceiling at a lower effective IOPS with large I/Os. Knowing whether your workload is small-random or large-sequential changes which limit you hit first.

The saturation pattern: gradual, then sudden

Storage saturation follows a non-linear curve. Latency increases slowly as load approaches the device limit, then spikes exponentially once queues saturate. This is Little’s Law in action: as the arrival rate of I/Os approaches the service rate, time spent waiting in queue grows disproportionately.

There is no comfortable zone where you can ride at 95% of capacity. The relationship between IOPS and latency stays roughly flat until you hit the knee of the curve, then falls off a cliff. The queue fills, every additional I/O waits behind the ones already queued, and latency goes from a few milliseconds to tens or hundreds of milliseconds in seconds.

Keep sustained IOPS below roughly 70% of the backend’s rated maximum. Below that line, you have headroom for bursts and stay on the flat part of the latency curve. Above it, small increases in load cause large increases in latency. This is not a hard limit, but it is the point where you lose the safety margin that absorbs transient spikes.

Queue depth is the other half of the picture. When the device queue is full, additional I/Os back up in the VMkernel queue. In esxtop, ACTV shows the number of I/Os actively in flight at the device, and QUED shows the number waiting in the VMkernel because the device queue is full. QUED should be zero in steady state. Any sustained non-zero QUED means the device cannot keep up, and KAVG (kernel latency) rises as a direct result.

Reading the signal combinations

IOPS and throughput on their own tell you how much load exists. Latency tells you how the backend is handling it. The diagnostic power comes from reading them together. Four combinations cover the cases you will see in production.

flowchart TD
    A["IOPS trend over 5+ minutes"] --> B{"Latency trend"}
    B -->|"Flat or low"| C["Healthy load increase
backend has headroom"]
    B -->|"Rising with IOPS"| D["Saturation forming"]
    B -->|"Rising while IOPS drops"| E["Path failure or array stall"]
    B -->|"Dropping with IOPS"| F["App stopped I/O
or VM powered off"]
    D --> G{"QUED in esxtop"}
    G -->|"Zero"| H["Early saturation
approaching the knee"]
    G -->|"Greater than zero"| I["Active saturation
commands backing up"]

IOPS rising, latency flat or low. The backend has headroom. You are pushing more work through, and the device is absorbing it without queueing. This is normal growth. Track it for capacity planning, but there is no incident here.

IOPS rising, latency rising. The saturation signature. The backend is approaching its limit and the latency curve is bending upward. If QUED is still zero, you are in the early stage and can act before it bites. If QUED is greater than zero, the device queue is full and commands are backing up in the VMkernel. This is your window to intervene before the storage latency cliff hits every VM on the datastore simultaneously.

IOPS dropping, latency rising. Counterintuitive and dangerous. Fewer I/Os but higher latency usually means a storage path has failed and the surviving path is now handling all the traffic. The surviving path saturates under the redirected load. Alternatively, the array itself has stalled due to controller failover, a rebuild, or congestion. Check path state and array health immediately. This pattern can precede an All Paths Down event.

IOPS dropping, latency dropping. The application stopped generating I/O, or the VM was powered off. Benign unless unexpected. If a critical database suddenly stops doing I/O during business hours, that is a real problem, but it is not a storage saturation problem.

Backend-dependent thresholds

There is no universal IOPS alert threshold because the ceiling depends entirely on what sits behind the datastore. The same 5,000 IOPS that is trivial for an all-flash array is a crisis for a datastore backed by a few spinning disks.

The real threshold is whatever your specific array, LUN, or disk group can sustain while keeping latency below the level your applications tolerate. For latency-sensitive workloads such as databases, the target is GAVG under 10ms at peak. For all-flash arrays, DAVG should be under 1ms in steady state, and anything above 5ms indicates a problem. For HDD-based arrays, a 5-15ms DAVG baseline is typical, and above 20ms indicates degradation.

Baseline your datastores during normal operation. Learn what IOPS and latency look like at your typical peak. If a datastore normally runs at 8,000 IOPS with 1ms latency and you see 12,000 IOPS with 8ms latency, something changed even if neither number looks alarming in isolation.

Checking queue depth in esxtop

When you suspect saturation, queue depth is the confirming signal. Latency alone tells you something is slow. Queue depth tells you whether the slowness is because the device is genuinely at its limit or whether the array is slow for another reason.

# Disk device view: check QUED and ACTV per device
esxtop
# Press 'u' for disk device view
# ACTV = I/Os the HBA has sent to the device (active, in-flight)
# QUED = I/Os waiting in VMkernel before the HBA can accept them
# Press 'f' to add or remove visible columns

ACTV approaching the device queue depth limit means the device is at maximum concurrent I/O capacity. QUED greater than zero sustained for more than 30 seconds means the VMkernel is actively holding I/Os it cannot dispatch. Either condition, combined with rising latency, confirms saturation rather than a transient burst.

If you see high QUED with normal DAVG, the configured device queue depth may be too low for the workload, or VMFS metadata locking is causing contention during heavy snapshot or clone operations. If QUED is high and DAVG is also high, the array itself is the bottleneck.

Why IOPS drops matter as much as spikes

Teams naturally alert on high IOPS. A sudden IOPS drop can be equally significant and is frequently missed because it looks like relief.

A drop in datastore IOPS combined with rising latency on the surviving traffic is a classic path-failure signature. One storage path goes dead. Multipathing redirects I/O to the remaining path. The remaining path, now carrying double its normal load, saturates. The result is lower aggregate IOPS (because the surviving path is the bottleneck) but higher latency on every I/O that does get through. If you only alert on high IOPS, you miss this entirely.

A sudden IOPS drop with no latency change usually means the workload source stopped: a backup job finished, a batch process completed, or a VM was powered off. The diagnostic question is whether the drop was expected.

The pattern to watch for is a drop that does not match any known operational change. If IOPS fall by 50% on a production datastore and nobody powered anything off or finished a job, investigate. It may be a path failure, an application hang, or a VM that stopped processing its queue.

Signals to watch in production

How Netdata helps

Netdata collects per-second storage metrics from ESXi hosts and vCenter. This matters for saturation detection because the transition from fine to saturated can happen in seconds, not minutes. Five-minute rolled-up averages hide the spike that reveals the knee of the latency curve.

• Correlating IOPS, throughput, and latency on the same timeline lets you see whether latency rises with IOPS (saturation) or independently (array stall or path failure). The shape of the correlation is the diagnostic signal.
• Per-VM visibility helps identify which VM is driving the load on a saturated datastore, not just that the datastore is busy.
• Queue depth signals alongside latency confirm whether high latency is queue saturation or array-side slowness, which determines whether the fix is load distribution or array investigation.
• ML anomaly detection flags the combination of rising IOPS with rising latency as anomalous even when neither metric individually crosses a static threshold. This is exactly the early saturation pattern that static alerts miss.
• Tracking IOPS trends over days and weeks against the backend’s rated capacity gives you the runway to plan capacity before you hit the 70% line.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap
• vSphere memory reclamation cascade: balloon to compress to swap in minutes
• vSphere monitoring checklist: the signals every host, VM, and vCenter needs