vSphere thin-provisioned VMDK growth: space that never comes back without UNMAP

A thin-provisioned VMDK starts small and grows as the guest writes data. It never shrinks on its own. When a guest deletes a 100 GB database dump, the space inside the guest filesystem becomes free, but the VMDK file on the datastore stays at its high-water mark. Over months, the datastore fills with blocks the guest considers empty. The symptom is a datastore that creeps toward full while the guests report plenty of free space inside.

VMFS does not know which blocks the guest has freed. The guest filesystem marks blocks as free in its own metadata, but nothing automatically tells the hypervisor or the storage array. Reclaiming that space requires a deliberate signal chain: the guest issues a TRIM or UNMAP command, the hypervisor translates that into a VAAI UNMAP operation, and the array releases the underlying blocks. Break any link and the space stays stranded.

This article covers the reclamation chain end to end, VMFS-5 versus VMFS-6 differences, NFS complications, and the signals that reveal silent over-commitment before a datastore fills.

Why thin VMDKs only grow

A thin VMDK is a sparse file. VMFS allocates blocks on demand as the guest writes to previously unwritten regions. Once allocated, those blocks belong to the VMDK. The guest can later delete files, and its filesystem will mark those blocks as free internally, but VMFS has no view into the guest filesystem. From the datastore’s perspective, the block is still in use.

The high-water mark behavior is intentional. It avoids probing guest intent and keeps the write path fast. The tradeoff is that space consumption is monotonic unless something actively tells VMFS that blocks are reclaimable.

That something is the SCSI UNMAP command, also called TRIM in ATA terminology. The guest must issue it, and the hypervisor must propagate it. Without both, thin VMDKs grow but never shrink.

How the reclamation chain works

Space reclamation requires cooperation across three layers: the guest OS, the VMFS datastore, and the storage array. Each layer can silently drop the signal.

flowchart TD
  A[Guest deletes file] --> B{Guest issues TRIM/UNMAP?}
  B -- No --> C[VMDK stays at high-water mark]
  B -- Yes --> D{Datastore type?}
  D -- NFS --> E[No UNMAP path
array plugin only]
  D -- VMFS-5 --> F[EnableBlockDelete=1 for in-guest
or manual esxcli unmap]
  D -- VMFS-6 --> G[Automatic background UNMAP]
  F --> H[VAAI UNMAP to array]
  G --> H
  H --> I[Array releases blocks]

The guest side is the first gate. On Linux, the fstrim command, or the discard mount option, issues UNMAP for freed blocks. On Windows, Optimize-Volume -ReTrim does the equivalent. Many distributions do not run fstrim on a schedule by default, and Windows Server does not always detect the virtual disk as thin-provisioned. If the guest never issues the command, nothing downstream happens.

The VMFS side is the second gate. VMFS-5 and VMFS-6 handle reclamation very differently, and mixing them up is one of the most common reasons operators believe UNMAP is broken.

VMFS-5: manual UNMAP only

VMFS-5 has no automatic background reclamation. To reclaim space, you must run a manual UNMAP against the datastore.

# Reclaim dead blocks on a VMFS-5 datastore
# WARNING: Generates sustained UNMAP I/O that can saturate array ports.
# Schedule during a maintenance window and monitor array latency.
esxcli storage vmfs unmap -l <datastore-name>
# The reclaim-unit (default 200) controls batch size per iteration.
# Lower it on arrays with limited UNMAP throughput.

For in-guest UNMAP to propagate through VMFS-5 to the array, the VM must have the EnableBlockDelete advanced parameter set to 1. It is disabled (0) by default. Without it, fstrim or Optimize-Volume inside the guest appears to succeed from the guest’s perspective, but VMFS-5 does not forward the reclamation to the array.

Thick-provisioned VMDKs (lazy zeroed or eager zeroed) do not support UNMAP at all. If you need reclamation on a thick disk, the supported path is converting to thin via Storage vMotion. After that conversion, Windows Server may not detect the disk as thin until a guest reboot, so Optimize-Volume -ReTrim can fail with “not supported” even though the VMDK descriptor shows ddb.thinProvisioned = "1".

VMFS-6: automatic background UNMAP

VMFS-6 introduced asynchronous background space reclamation. By default it runs at low priority, typically reclaiming at 25 to 50 MB/s. No manual intervention is required for array-side reclamation of blocks VMFS already knows are dead.

On VMFS-6, EnableBlockDelete is ignored. The auto-UNMAP mechanism handles array reclamation directly. However, the guest still must issue TRIM or UNMAP for the background process to have anything to reclaim. VMFS-6 only learns about dead blocks when the guest tells it.

Two granularity limits govern whether VMFS-6 reclamation actually works:

• Array granularity: VMFS-6 auto-UNMAP requires the array’s unmap granularity to be 1 MB or less. If the array page or chunk size exceeds 1 MB, automatic reclamation silently does nothing. This is a common root cause for “UNMAP is configured but space is not coming back.”
• Guest UNMAP granularity: VMFS-6 processes guest UNMAP requests only when the space to reclaim equals 1 MB or a multiple of 1 MB. Sub-1 MB or misaligned requests are dropped silently.

Auto-UNMAP is also slow by design. After a VM is deleted or a large VMDK is removed, reclamation can take 12 to 24 hours to complete. In-guest UNMAP, running fstrim or Optimize-Volume, triggers reclamation within minutes because it pushes the signal directly rather than waiting for the background sweep.

A historical note: ESXi 6.7 had a bug where the unmap rate calculation was incorrect, causing excessive UNMAP throughput even when priority was set to Low. If you run 6.7 and your array reports UNMAP storms, patch before chasing array-side causes.

If automatic reclamation is set to None, Broadcom KB 323112 still lists esxcli storage vmfs unmap for a manually reclaimed VMFS-6 datastore.

Snapshots block reclamation

Snapshots change the reclamation story. When a VM has snapshots, writes go to a delta disk in SEsparse format. Automatic space reclamation on VMFS-6 for snapshot delta files works only on ESXi 6.7 and later, and only on the top snapshot while the VM is powered on. Older ESXi versions, or VMs with deep snapshot chains, may not reclaim space from the delta layer at all.

This matters because environments with long-lived snapshots, often from backup jobs that failed to clean up, accumulate space in deltas that cannot be reclaimed until the snapshots are consolidated. A VM with a 200 GB base disk and an active snapshot can grow the delta to 200 GB, and none of that space is reclaimable while the snapshot exists.

NFS: the second layer of thin provisioning

NFS datastores add a complication. SCSI UNMAP is not supported on NFS datastores. Guest TRIM and DEALLOCATE commands do not propagate through the NFS client to the array. If your VMDKs live on NFS, the only reclamation path is whatever the array vendor provides through a vSphere plugin or a separate management interface.

NFS also introduces a second layer of thin provisioning. The NFS server reports free space to ESXi based on its own view of the LUN or volume, which may itself be thin-provisioned on the array. The datastore can report plenty of free space while the underlying array volume is near its physical limit. The guest thinks it has space, the NFS client thinks it has space, and the array is the only one that knows the truth.

A known reporting quirk: thin-provisioned VMDKs on NFS may display as thick in the vSphere Client because the NFS server returns metadata that ESXi misinterprets. Do not rely on the GUI provisioning type column for NFS datastores.

Over-commitment: provisioned vs consumed

The deeper operational risk with thin provisioning is over-commitment. Thin VMDKs let you provision more capacity than the datastore physically holds, on the assumption that not every guest will write to its full allocation. This is reasonable until it is not.

Two numbers matter, and they diverge over time:

• Provisioned capacity: the sum of all VMDK sizes as configured. This can be many times the datastore’s physical size.
• Consumed capacity: the actual blocks written. This is what the datastore physically holds.

When consumed capacity approaches the physical limit, the datastore is full regardless of what provisioned capacity says. The standard failure is a VM that tries to grow its thin VMDK, create a swap file, or extend a snapshot delta, and finds no space. All VMs on that datastore halt writes, not just the one that triggered the allocation.

The trap is that consumed capacity grows monotonically when reclamation is not happening. A datastore can sit at 40% consumed for months, then climb to 95% in a week if a workload churns through large temporary files. Without reclamation, those temporary files leave permanent residue.

Where reclamation silently breaks

When a datastore keeps filling despite guests deleting data, the problem is almost always a broken link in the reclamation chain. Check each layer:

• Guest not issuing TRIM: Run fstrim -v / on Linux or Optimize-Volume -DriveLetter C -ReTrim -Verbose on Windows. If the command is unsupported or reclaims nothing, the guest is the gap. Confirm the disk is seen as thin and that the filesystem supports discard.
• VMFS-5 without EnableBlockDelete: In-guest UNMAP appears to work but nothing reaches the array. Set EnableBlockDelete=1 or rely on manual esxcli storage vmfs unmap.
• VMFS-6 array granularity above 1 MB: Auto-UNMAP runs but reclaims nothing. This is invisible from ESXi. Verify the array’s unmap page or chunk size through the array management interface.
• Snapshots in the chain: Deltas block reclamation below the top snapshot. Consolidate or delete snapshots before expecting space to return.
• NFS datastore: There is no guest-driven UNMAP path. Use the array vendor’s vSphere plugin or accept that space reclamation requires array-side action.
• Thick VMDKs: Thick disks, including those converted from VMFS-3 datastores or created manually, do not support UNMAP. Convert to thin via Storage vMotion.
• VAAI UNMAP silently unsupported: If the LUN partition table was created manually rather than via the vSphere Client, or if the datastore was upgraded from VMFS-3, VAAI UNMAP may not function even when everything else looks correct.

Signals to watch in production

The datastore free space thresholds use AND, not OR. A 50 TB datastore at 5% free still has 2.5 TB. A 200 GB datastore at 5% free has 10 GB, which is an emergency. Use both percentage and absolute numbers.

How Netdata helps

Netdata’s VMware vSphere integration surfaces the datastore signals that reveal thin-provisioning drift before it becomes an outage.

• Per-second datastore free space: catch the moment a datastore crosses a percentage or absolute threshold, not a rolled-up average hours later.
• Provisioned vs consumed capacity correlation: see over-commitment ratios change as workloads churn, which is the earliest sign that reclamation has stopped working.
• Datastore latency alongside free space: when a datastore nears full, latency often rises first as the array struggles with allocation. Correlating the two distinguishes a space problem from a performance problem.
• Snapshot age and delta growth tracking: snapshots are the most common accelerant for datastore pressure. Netdata exposes snapshot age per VM, letting you catch deltas that block reclamation.
• Anomaly detection on consumption rate: a sudden change in the daily free-space slope, flagged by ML, is often the first indicator that a workload is leaking space into an un-reclaimed thin VMDK.
• Cross-layer correlation: correlate datastore free space with host swap activity, VM disk latency, and backup job timing to identify whether a backup storm or memory pressure cascade is driving the growth.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap