vSphere vMotion slow or failing: memory dirty rate, bandwidth, and convergence

A vMotion task that should take minutes instead crawls for ten, twenty, or thirty minutes. The vSphere Client progress bar hangs in the high nineties. Sometimes the migration completes with a multi-second stun. Sometimes it fails outright: “The migration was cancelled because the amount of changing memory for the VM was greater than the available network bandwidth.”

Almost every slow or failing vMotion traces back to one relationship: the VM is dirtying memory faster than the vMotion network can transmit it. The iterative pre-copy loop cannot converge. ESXi either stuns the vCPUs to force convergence, hurting guest performance, or cancels the migration.

This article covers compute vMotion only: live migration of VM memory and CPU state between hosts. Storage vMotion moves VMDKs between datastores and has different failure modes. The convergence reasoning here does not apply to it.

What this means

vMotion does not copy memory once. It copies it iteratively. ESXi sends the entire memory contents of the VM across the vMotion network while the VM keeps running. While those pages are in flight, the guest keeps writing to memory, dirtying pages already sent. ESXi loops back and re-sends the dirty pages. This repeats until the remaining dirty set is small enough to copy within the switchover window.

The switchover target is approximately 500ms of VM stun time (Broadcom KB 436882 confirms 500ms as the vMotion goal). When the remaining dirty pages fit inside that window, ESXi stuns the VM, sends the final pages, transfers CPU and device state, and resumes the VM on the destination. From the guest perspective this is a brief pause.

Convergence is the whole game. If the dirty rate exceeds the transmit rate, the remaining dirty set never shrinks and the pre-copy loop runs indefinitely. ESXi has two responses:

• SDPS (Stun During Page Send), enabled by default, stuns the vCPUs in microsecond bursts to slow the dirty rate and force convergence. The guest sees latency spikes. On write-heavy database workloads such as SQL Server and AlwaysOn, the impact is noticeable.
• If SDPS cannot close the gap, the migration is cancelled with the bandwidth error above.

Long duration is the leading indicator. A vMotion taking more than 10 minutes for a moderately sized VM is a convergence problem. A single 1GbE link for a 128GB VM is untenable. Use 10GbE or faster with dedicated vMotion vmknics.

flowchart TD
  A[vMotion starts] --> B[Pre-copy memory over vMotion network]
  B --> C{Dirty rate lower than transmit rate?}
  C -- yes --> D[Iterate: resend dirty pages]
  D --> C
  C -- no --> E[SDPS stuns vCPUs to slow dirty rate]
  E --> F{Remaining set fits switchover window?}
  F -- yes --> G[Final stun, transfer state, resume on dest host]
  F -- no --> H[Migration cancelled: changing memory exceeds bandwidth]

Common causes

Quick checks

Run these on the source ESXi host unless noted. All are read-only.

# Verify jumbo frames end-to-end between vMotion vmks (8972 = 9000 MTU minus 28 bytes ICMP+IP headers)
vmkping -I vmk1 -d -s 8972 <destination-vmk-ip>

# Watch vMotion vmk bandwidth in real time. Press 'n' for network view in esxtop, find the vMotion vmk row.
esxtop

# Look for NIC driver or firmware errors that masquerade as bandwidth problems
grep -iE "bnxtnet|driver|firmware|link" /var/log/vmkernel.log | tail -50

# On the destination host, inspect stun time from the last completed migration
grep -i "vmDowntime" /var/log/hostd.log | tail

From vCenter or PowerCLI:

# Recent vMotion failures in the last 24 hours
Get-VIEvent -Types "VmFailedMigrateEvent" -Start (Get-Date).AddHours(-24)

How to diagnose it

1. Confirm it is a compute vMotion, not a storage vMotion. Storage vMotion moves VMDKs between datastores and is inherently slower. Check the task type in the recent tasks list.
2. Measure the actual transmit rate. In esxtop, watch the vMotion vmk during the migration. If it sits at roughly the speed of one uplink, the link is the bottleneck. A single vMotion stream achieves roughly 15 Gbps of average throughput on a capable NIC; operator testing on 25GbE NICs corroborates this ceiling.
3. Estimate the dirty rate. There is no single per-VM dirty-rate counter exposed through standard statistics levels or esxtop. The proxy is convergence behavior: if the migration runs long, hangs at a high percentage, then stuns for several seconds or fails, the dirty rate is outrunning the link.
4. Check for SDPS impact. If the VM is a write-heavy database and users report latency during migrations, SDPS is likely stunning vCPUs. Stun time is visible in hostd.log on the destination host via the vmDowntime field, reported in microseconds (Broadcom KB 436882). Normal stun is in the 100 to 500ms range, which appears as roughly 100000 to 500000 in the log. Seconds indicate convergence difficulty.
5. Verify the network path. Confirm the vMotion vmk is on a dedicated TCP/IP stack, that jumbo frames match end to end, and that no other high-bandwidth traffic shares the uplink without Network I/O Control enforcement.
6. Rule out driver and firmware problems. False bandwidth errors have been traced to NIC driver and firmware issues, particularly with bnxtnet. Check vmkernel.log around the failure timestamp.
7. Check destination host storage health. A migration that fails around 19% with “Operation Timed out” often correlates with destination datastore latency above 40ms.
8. Watch for concurrency limits. A migration stuck at 12% with “Resources currently in use by other operations. Waiting.” usually means the host has hit its concurrent vMotion cost limit. All hosts have a maximum cost per host of 8; each vMotion costs 2 units on the source and 2 on the destination.

Metrics and signals to monitor

Fixes

Bandwidth is the bottleneck

Move vMotion onto its own vmk bound to a 10GbE or faster uplink. One 1GbE link for a 128GB VM is not workable.

For 25GbE and faster, a single vMotion stream tops out around 15 Gbps of average throughput. To use the rest of the link, provision multiple vMotion vmks on the same subnet, or set the advanced option Migrate.VMotionStreamHelpers to a positive value to allow multiple parallel streams (a value of 0 means dynamic allocation of one stream per IP, per the VMware vMotion tuning guide).

If vMotion currently uses the Default TCP/IP stack, migrate it to a dedicated vMotion TCP/IP stack. The Default stack shares resources with management and backup traffic. On encrypted VMs this contention frequently triggers bandwidth timeouts. A dedicated stack also lets you set a distinct default gateway and route vMotion traffic predictably.

Enable Network I/O Control on the distributed switch and assign shares or limits to the vMotion traffic class so a backup storm cannot starve migrations.

Memory dirty rate outruns the link

For write-heavy databases and in-memory workloads, the workload itself is the problem. Options, in order of preference:

• Schedule migrations during low-write windows. A database sustaining high memory write rates during peak will not migrate cleanly on a 10GbE link.
• Quiesce the workload before migration. For databases this means draining connections or failing over to a standby.
• Increase vMotion bandwidth so the transmit rate exceeds the dirty rate.
• As a last resort, disable SDPS by setting Migrate.SdpsEnabled = 0 on the host. Warning: this removes the guest-visible stun but risks the migration failing outright if convergence is not reached. Use only when you have accepted the risk of migration failure.

Encrypted vMotion adds CPU overhead on the source host; VMware performance testing shows approximately one core per 10 Gbps of vMotion traffic on the source host (the destination uses less than half a core). If encrypted vMotion is slow and CPU headroom is low, encryption cost may be the limiter, not the link.

Cross-vSwitch and port group mismatch

A vMotion between hosts on different virtual switches fails if the port group configuration does not match. The VM’s network labels and VLAN IDs must exist on both source and destination. Verify naming, VLAN, and security policy on both sides before initiating.

NIC driver or firmware problems

If the bandwidth error appears on a link that should have ample capacity, suspect the driver. The bnxtnet driver has been linked to false “changing memory” cancellation errors. Check vmkernel.log for driver warnings around the failure time, confirm the driver and firmware versions against the VMware Compatibility Guide, and update if a fixed version is available.

Stuck or split VMs

A failed vMotion can, rarely, leave a VM registered on two hosts simultaneously. This is generally resolved by vCenter automation. If automation does not clean it up, do not manually delete the VM from either side without confirming which host is actually running it. Check the VM’s runtime host and power state through vCenter before taking action.

Prevention

• Dedicate a vmk and uplink to vMotion. 10GbE minimum for any VM larger than a few GB.
• For 25GbE and above, provision multiple vMotion vmks or enable stream helpers to use the full link.
• Put vMotion on a dedicated TCP/IP stack with its own gateway.
• Verify jumbo frame consistency on every vMotion path. An MTU mismatch is a top cause of migration failure. Test with vmkping -I vmk<#> -d -s 8972 <destination_IP>.
• Cap DRS aggressiveness on clusters with write-heavy workloads. Aggressive DRS can trigger migration storms that saturate the vMotion network and pile up at the per-host concurrency limit.
• Track vMotion duration and stun time as routine signals. A slow drift upward indicates either growing VMs or a shrinking bandwidth budget.
• For vGPU workloads, vSphere 8.0 Update 3 added zero-copy support that raises vMotion throughput up to 100 Gbps (confirmed in the 8.0 U3 release notes), and vSphere 9.0 extended zero-copy pre-copy for vGPU VMs (confirmed in the VCF 9.0 What’s New blog). Plan host upgrades around these if vGPU migration stun is a recurring pain point.

How Netdata helps

• Per-second visibility into host network utilization by vmk and vmnic, so you can see exactly when the vMotion link saturates and whether another traffic class is competing for the same uplink.
• Correlation of vMotion task duration and failure events with datastore latency on the destination host, separating bandwidth-bound failures from storage-bound failures that land around the 19% mark.
• CPU utilization tracking on the source host during encrypted vMotion, exposing encryption overhead as a separate signal from link saturation.
• Anomaly detection on physical uplink utilization that flags a single pinned uplink in a NIC team, since teaming does not aggregate bandwidth for a single flow.
• Trends on vMotion duration and stun time across weeks, surfacing the slow drift that precedes a convergence failure on a growing database.
• Composite cluster views that show concurrent migration counts against the per-host limit, useful when DRS storms pile tasks up at 12%.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap