vSphere 'Virtual machine disks consolidation is needed': clearing the warning without stunning the VM

The “Virtual machine disks consolidation is needed” warning means the VMkernel left delta VMDKs on the datastore after a snapshot delete that did not fully commit. The VM is still running, but its writes are going through delta files that were never meant to persist.

The warning is set by VirtualMachine.runtime.consolidationNeeded in the vCenter inventory. It is distinct from the Snapshot Manager view: a VM can have this flag set while showing zero snapshots in the manager, because the flag tracks orphaned files on the datastore, not the snapshot tree.

The headline risk is that consolidation is I/O-intensive and stuns the VM at the moment it commits the final helper delta. For large chains, the stun lasts seconds to minutes. The fix is mechanical, but it must be planned around the stun window and datastore free space.

What this means

When a snapshot delete succeeds, the VMkernel merges the delta into the parent and removes the delta file. When the delete fails or is interrupted, the snapshot metadata is updated (the snapshot “disappears” from the manager) but the delta file remains on the datastore. Every subsequent write still goes to that delta, and it grows indefinitely.

The consolidationNeeded flag flips to true when vCenter notices this discrepancy. The two failure shapes are:

1. The Snapshot Manager shows no snapshots, but *-delta.vmdk files exist on disk and the VM still writes to them.
2. The Snapshot Manager shows snapshots that you tried to delete, the delete task completed with a green check, but the deltas remain.

Both produce the same warning and the same risk: the delta grows with every write, datastore free space declines, and read latency climbs because every read must traverse the chain.

For a powered-on VM, ESXi creates an additional “Consolidate Helper” delta to absorb writes during the merge, then commits that helper at the end. The commit is the stun. ESXi predetermines the stun duration; if it estimates the stun will exceed 15 seconds, it waits for another merge iteration, up to 10 iterations. On the 10th iteration it stuns for the full required time.

Common causes

Quick checks

These are read-only and safe to run during business hours.

# PowerCLI: list every VM that needs consolidation
Get-VM | Where-Object {$_.ExtensionData.Runtime.ConsolidationNeeded} | Select Name

# PowerCLI: show snapshots (does not catch orphaned deltas)
Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB

# PowerCLI: snapshot tree depth per VM
Get-VM | Where-Object {$_.ExtensionData.Snapshot} | ForEach-Object {
  $tree = $_.ExtensionData.Snapshot.RootSnapshotList
  [PSCustomObject]@{ VM=$_.Name; RootSnapshots=$tree.Count }
}

# ESXi shell: list delta files for a specific VM folder
ls -lh /vmfs/volumes/<datastore>/<vm-folder>/*-delta.vmdk
ls -lh /vmfs/volumes/<datastore>/<vm-folder>/*-00000*.vmdk

# ESXi shell: datastore free space (consolidation needs ~1.5x total delta size)
# Note: ESXi's busybox df has limited output; use 'vdf' for VMFS volume detail.
df -h /vmfs/volumes/<datastore>
esxcli storage filesystem list

# ESXi shell: identify the lock owner on a VM disk
vmkfstools -D /vmfs/volumes/<datastore>/<vm-folder>/<disk>.vmdk
# Output includes the MAC of the lock owner; cross-reference with host vmnics

The vmkfstools -D output is the fastest way to identify a backup proxy that hot-added a disk and never released it. The owner MAC corresponds to whichever host still holds the lock.

How to diagnose it

1. Confirm the warning is real. Re-run the PowerCLI ConsolidationNeeded query. If true, move on.
2. Identify every VM with the flag. A single failed backup job can leave deltas on dozens of VMs simultaneously. The blast radius is the backup job, not a single VM.
3. For each VM, list the delta files and note their size. The size is the merge workload. A 10 GB delta merges in seconds on all-flash; a 500 GB delta can take many minutes and stun for tens of seconds.
4. Check datastore free space. Consolidation needs approximately 1.5x the total size of all delta files in free space. Below that threshold, consolidation fails with “File too large.”
5. Check for file locks before clicking Consolidate. If a backup proxy holds a lock, the task will fail. Resolving the lock means hot-removing the disk from the proxy VM (do not delete from datastore), or restarting the proxy’s VDDK service.
6. For vSphere 8.0 prior to U3e on NFSv3 datastores, expect a locking race that can fail consolidation and potentially power off the VM. Patch to U3e, or set the advanced option workaround described below.
7. Decide whether to consolidate while the VM is running or to schedule a power-off window. Powered-off consolidation is faster and safer: no helper delta is created, no stun occurs, and no extra space is needed beyond the delta size.

flowchart TD
    A["ConsolidationNeeded = true"] --> B{"Snapshot Manager shows snapshots?"}
    B -- Yes --> C["Try Delete All first"]
    B -- No --> D["Orphaned deltas on disk"]
    C --> E{"Delete succeeds?"}
    E -- Yes --> F["Flag clears"]
    E -- No --> D
    D --> G{"Lock held by backup proxy?"}
    G -- Yes --> H["Hot-remove disk from proxy"]
    G -- No --> I{"Datastore free >= 1.5x delta?"}
    H --> I
    I -- No --> J["Free space, then retry"]
    I -- Yes --> K{"Safe to stun VM?"}
    J --> K
    K -- Yes --> L["Consolidate (powered on)"]
    K -- No --> M["Schedule power-off window"]
    L --> N["Flag clears"]
    M --> O["Power off, consolidate, power on"]
    O --> N

Metrics and signals to monitor

Fixes

Resolve file locks held by backup proxies

The single most common cause of failed consolidation is a backup proxy VM that hot-added a disk via VDDK HotAdd transport and never released it. Do not delete files from the datastore. Edit the backup proxy VM’s settings, note the hot-added disks (named after the protected VM’s VMDKs), and remove them from the proxy using “Remove from virtual machine” only, not “Delete from disk”. After the proxy releases the disks, retry consolidation.

If the proxy does not release the lock, restarting the backup service on the proxy (or rebooting the proxy VM) usually clears it.

Free space before consolidating

Consolidation fails with “File too large” when there is not enough room to hold the merged disk alongside the originals until the swap-in completes. Free up approximately 1.5x the total delta size before retrying. Quick wins: delete unneeded snapshots on other VMs, migrate a VM off the datastore, or run vmkfstools -U on confirmed orphaned VMDKs.

WARNING: vmkfstools -U permanently deletes the VMDK. Confirm the file is truly orphaned (not referenced by any VM or snapshot chain) before running it. If in doubt, rename the file instead and observe for a week.

Set the NFSv3 locking workaround on affected builds

For vSphere 8.0 prior to U3e, a locking race on NFSv3 datastores causes consolidation to fail and can power off the VM. The workaround is setting consolidate.upgradeNFS3Locks = "FALSE" in the VMX (per-VM) or /etc/vmware/config (per-host). This is a workaround, not a permanent fix. Patch to U3e or later when possible.

Consolidate during a power-off window when the delta is large

Powered-off consolidation is faster, does not stun, and does not need the helper delta. For a large delta chain (hundreds of GB), schedule a maintenance window, shut the VM down, consolidate, and power it back on. The tradeoff is favorable when the stun estimate exceeds the application’s stun tolerance.

Recover from more than 32 snapshots

VMware supports at most 32 snapshots in a linear branch. Past that, standard consolidation fails and a manual vmkfstools procedure is required.

Set ctkEnabled = FALSE on the VM before starting; CBT must be disabled for this recovery path. The procedure applies to traditional storage and vSAN Original Storage Architecture. vSAN Express Storage Architecture uses Native Snapshots managed differently.

Corrupt .vmsd metadata

When the Snapshot Manager shows no snapshots but the warning persists and delta files exist, the .vmsd file is likely corrupt. The recovery sequence: move .vmsd, -ctk.vmdk, and .vmsn files to a temporary directory, reload the VMX (unregister and register, or vim-cmd vmsvc/reload <vmid>), create a new non-memory snapshot, then Delete All and Consolidate. This forces vCenter to rebuild the snapshot metadata from the actual on-disk state. Test in a lab first.

Distinguish Delete All from Consolidate

These are not the same operation. Delete All commits every snapshot in the chain directly to the base disk, sequentially, and can take hours on long chains. Consolidate merges only redundant delta disks left after a failed delete. If the Snapshot Manager shows snapshots, use Delete All first. If the manager is empty but the warning persists, use Consolidate. Consolidation cannot be stopped once started.

Prevention

• Monitor the consolidationNeeded flag daily. Treat any true value as a ticket. The longer it stays true, the larger the delta and the longer the eventual stun.
• Alert on snapshot age and chain depth. Any snapshot older than 72 hours, or any chain deeper than three, should generate a ticket.
• Correlate backup job success with snapshot state. A failed backup job is the most common root cause. When a backup job fails, the next monitoring cycle should show whether deltas were left behind.
• Watch datastore free space and reserve snapshot headroom. Keep at least 20 to 25 percent free, or enough to absorb 24 hours of snapshot growth across all VMs.
• Patch ESXi. Consolidation bugs are version-specific. The NFSv3 locking race that can fail consolidation and power off VMs is fixed in ESXi 8.0 Update 3e (KB 402937). The StorageFPIN heap exhaustion bug that caused “File too large” errors during consolidation even when datastore space was sufficient is fixed in 8.0 Update 3e (P05) (KB 410950). The 30-second NFS stun from a shared-to-exclusive lock upgrade race is fixed in 8.0 Update 2b (8.0.2 P03) (KB 320319).
• Stagger backup jobs. Backup proxies that hot-add many disks at once are the source of most lock conflicts. Stagger jobs so no single proxy holds disks from many VMs simultaneously.
• Keep chains shallow. VMware recommends no more than two to three snapshots per chain. Configure backup products that chain incremental snapshots to run synthetic fulls on a schedule that keeps the chain short.

How Netdata helps

• Netdata surfaces the consolidationNeeded boolean as a per-VM state, so you do not need to run PowerCLI every morning to find orphans. The flag turning true is the earliest point at which you can intervene before the delta grows.
• Correlate the flag with datastore free space trends. A datastore with declining free space and a VM that needs consolidation is on a clock. Per-second collection makes the rate of decline visible, so you can estimate when the datastore will hit the 1.5x headroom threshold.
• Correlate with datastore latency (GAVG, DAVG, KAVG) and per-virtual-disk read latency. The snapshot chain penalty shows up as elevated latency on the affected VM’s disks while peers on the same datastore remain normal. This separates a chain problem from an array problem.
• Track snapshot age and chain depth alongside the consolidation flag. The combination of old snapshot, deep chain, and flag true is the worst case and the one most likely to produce a long stun.
• Tie backup job success and failure to snapshot state. If you ingest backup job events as metrics or logs, Netdata’s anomaly detection can flag a pattern of failed jobs followed by new consolidationNeeded true values.
• During consolidation itself, watch VM CPU ready time, datastore latency, and task duration. The merge is I/O-heavy; an uptick in KAVG and host device queue depth during a consolidation window is expected and confirms the operation is progressing.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap
• vSphere memory reclamation cascade: balloon to compress to swap in minutes