vSphere physical uplink saturation: one flow uses one NIC, and storage shares the wire

A vSphere host with two 10GbE uplinks in a NIC team looks like 20Gbps. For aggregate multi-flow traffic across many VMs, it is. But a single TCP flow between one VM and one remote host uses exactly one of those uplinks. The team does not split that flow across both links. That is why a VM doing a bulk transfer can saturate one uplink while the other sits idle.

The problem compounds when storage traffic shares the wire. NFS datastores, iSCSI, vMotion, management, and VM network I/O all compete for bandwidth on the same uplinks. At moderate utilization this is invisible. Above 70% on a link carrying storage I/O, congestion begins producing application-visible latency. Above 90% on a storage-bearing uplink, page on it.

What it is and why it matters

The default vSphere load balancing policy is “Route based on originating virtual port ID.” When a VM powers on, the vSwitch assigns its virtual port to one uplink in the team. All traffic from that VM’s vNIC flows through that single uplink until the VM powers off or vMotions to another host. The assignment is deterministic per port, not per packet.

A NIC team with two 10GbE uplinks therefore provides 20Gbps of aggregate throughput for many flows, but only 10Gbps for any single flow. A VM backing up 500GB to a network target will use one uplink at line rate while the second uplink carries nothing for that flow.

The operational trap is monitoring the team as an aggregate. Average utilization across both uplinks, and a VM saturating one link at 10Gbps while the other is idle shows as 50% utilization. That looks healthy. The VM is actually at its bandwidth ceiling, and any storage traffic sharing that uplink is already contending.

How it works

vSphere offers several load balancing algorithms on vSwitches and Distributed Switches. The choice determines how flows are distributed across uplinks. None of them aggregate bandwidth for a single flow without additional physical switch configuration.

Route based on originating virtual port ID (default, vSwitch and vDS)

Each virtual port on the switch is assigned to an uplink using a hash of the port ID. The assignment happens at port creation and stays fixed for the life of that port. A single VM with one vNIC is pinned to one uplink regardless of how much bandwidth it needs.

Route based on source MAC hash

Similar to port-based, but hashes on source MAC. Same limitation: one flow, one uplink.

Route based on IP hash

Hashes on source and destination IP to distribute flows. This can put different flows from the same VM on different uplinks. However, it requires EtherChannel or LACP on the physical switch, does not support standby uplinks, and still does not split a single TCP connection across multiple links. A single flow between one source-destination IP pair still uses one uplink.

Route based on physical NIC load (Load-Based Teaming, LBT, vDS only)

LBT is the only algorithm that actively monitors uplink utilization. It checks whether any uplink exceeds 75% utilization approximately every 30 seconds. If so, it moves the busiest VM port to a less-loaded uplink. This helps distribute aggregate load over time, but it does not split a single flow. A VM transferring at line rate will use one uplink until LBT’s next check cycle, and even then LBT moves the whole port, not individual packets within the flow.

flowchart TD
    VMA["VM A: bulk transfer"] -->|"pinned to one uplink"| NIC0["vmnic0 - 10GbE"]
    VMB["VM B"] -->|"pinned to one uplink"| NIC1["vmnic1 - 10GbE"]
    NFS["NFS / iSCSI storage"] --> NIC0
    VMOT["vMotion migration"] --> NIC0
    NIC0 -->|"above 70% utilization"| SAT["Storage I/O latency rises"]

VM A’s bulk transfer, NFS storage I/O, and vMotion all converge on vmnic0. VM B uses vmnic1. The team has two uplinks, but three traffic classes share one of them. When vmnic0 crosses 70% utilization with storage traffic on it, latency becomes application-visible.

Where it shows up in production

vMotion saturating a shared link

vMotion uses VMkernel adapters to migrate VM memory between hosts. By default, a host uses one VMkernel adapter for vMotion, bound to one uplink. On 10GbE, a single vMotion can consume the entire link. If that uplink also carries NFS or iSCSI storage traffic, every VM on the host with storage on that network experiences elevated latency for the duration of the migration.

DRS-triggered vMotion storms amplify this. DRS evaluates placement every 5 minutes by default. When it decides to migrate multiple VMs, each migration competes for the same vMotion bandwidth. On a shared uplink, this can push storage I/O into latency cliffs that affect every VM with storage on that network simultaneously.

NFS and iSCSI on shared uplinks

NFS datastores are sensitive to network latency because the NFS client runs in the VMkernel. The storage I/O path includes the network. The “device” in DAVG (device average latency) for an NFS datastore is the NFS client in the VMkernel, not the storage array. When the uplink carrying NFS traffic exceeds 70% utilization, the VMkernel’s NFS queue begins to experience congestion. This shows up as elevated DAVG on NFS datastores even though the storage array itself is healthy.

iSCSI is similarly affected. The iSCSI initiator in the VMkernel sends SCSI commands over TCP/IP, so uplink congestion adds latency to every I/O. With multipathing (MPIO), iSCSI can use separate VMkernel adapters bound to separate uplinks, which helps if explicitly configured. Without explicit port binding, iSCSI traffic follows the same teaming policy as everything else.

vSAN on shared uplinks

vSAN uses VMkernel adapters for all storage traffic, including both front-end VM I/O and backend resync from rebuilds, rebalances, and policy changes. On converged uplinks, vSAN resync operations can consume significant bandwidth, competing with VM network traffic and other storage I/O.

The vSAN team recommends Active/Standby teaming for vSAN VMkernel ports, not LBT. In spine-leaf topologies where two uplinks connect to two separate leaf switches with no inter-switch link, LBT sends approximately half the vSAN traffic across the spine, adding latency to every vSAN I/O. Active/Standby keeps all vSAN traffic on one leaf until failover.

VCF currently defaults to LBT for all port groups including vSAN VMkernel ports, though the vSAN team recommends changing this to Active/Standby (Route based on originating virtual port ID) for optimal performance.

Failover: the hidden saturation risk

When one uplink in a team fails, all traffic shifts to the remaining uplink. A host that was running at 40% on each of two uplinks, well within budget, suddenly runs at 80% on one. If that uplink carries storage traffic, 80% is already past the 70% threshold where storage latency becomes visible. Failover events are when converged uplink designs bite hardest: the remaining link absorbs the failed link’s load instantly with no rebalancing.

Tradeoffs and when to use it

LBT for VM traffic, Active/Standby for VMkernel traffic

LBT is a reasonable default for VM port groups on a vDS. It redistributes VM flows when uplinks get unbalanced. But LBT is suboptimal for VMkernel traffic such as vMotion, vSAN, NFS, and iSCSI, because the periodic rebalancing adds unpredictability to latency-sensitive storage paths. Active/Standby keeps VMkernel traffic deterministic on one uplink, which matters for storage I/O consistency.

NIOC for converged uplinks

Network I/O Control (NIOC) on a vDS lets you allocate bandwidth per traffic type using shares, reservations, and limits. When multiple traffic classes contend for the same uplink, NIOC enforces minimum bandwidth guarantees. This is the primary tool for preventing vMotion from starving storage I/O on a shared link. The total bandwidth reserved among all system traffic types cannot exceed 75 percent of the bandwidth provided by the physical network adapter with the lowest capacity. This 75 percent ceiling applies to the combined reservations, not per traffic type, leaving the remaining 25 percent for unreserved traffic and overhead.

Physical separation

The simplest fix for storage-on-shared-wire problems is to stop sharing the wire. Dedicated uplinks for NFS, iSCSI, or vSAN eliminate the contention entirely. On hosts with limited physical ports, VLAN separation on converged links combined with NIOC reservations is the practical middle ground. The tradeoff is operational complexity: more port groups, more VMkernel adapters, more policies to maintain.

Signals to watch in production

The critical measurement discipline is per-vmnic, not per-team. A team of two 10GbE uplinks at an aggregate of 50% may have one link at 95% and one at 5%. Only per-vmnic metrics reveal this.

To collect these signals on an ESXi host:

# Per-uplink throughput in esxtop
esxtop
# Press 'n' for network view, look at MbRX/s and MbTX/s per vmnic

# Per-uplink detailed stats including drops
esxcli network nic stats get -n vmnic0
# Check rx_dropped and tx_dropped fields

# Dropped packets in esxtop network view
# Press 'n', look at %DRPTX and %DRPRX columns per port

Averaged metrics miss microbursts. Sub-second spikes that cause packet drops are invisible in 5-minute rollup averages. Realtime counters (20-second interval) kept for one hour are the minimum resolution for catching uplink saturation events, and even those can miss sub-second bursts.

How Netdata helps

• Per-vmnic throughput, not team aggregate: Netdata collects bytes Rx and Tx per physical uplink. This is the signal that reveals asymmetric saturation invisible in averaged team metrics.
• Correlating uplink saturation with storage latency: When an NFS-bearing uplink crosses 70%, per-second resolution lets you see DAVG rise on the datastore within the same timeframe, confirming the network as the bottleneck rather than the array.
• Dropped packet detection: Netdata surfaces dropped Rx and Tx per vNIC and pNIC, catching ring buffer exhaustion or CPU saturation before it becomes user-visible.
• vMotion impact correlation: During a vMotion, Netdata can show the vMotion uplink saturating, the storage uplink sharing the same wire degrading, and affected VMs experiencing elevated datastore latency, all in the same per-second view.
• Failover monitoring: When an uplink goes down, Netdata immediately shows the remaining uplink’s utilization spike and any resulting storage latency increase, giving early warning before the saturation cliff hits storage I/O.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap