vSphere VMware Tools heartbeat red: guest crash vs extreme CPU starvation

A red guestHeartbeatStatus on a powered-on vSphere VM means VMware Tools has stopped sending heartbeats to the VMkernel. The common assumption is a guest OS crash. The actual cause may be extreme CPU starvation, a hung guest, a crashed Tools process, or a transient vCenter API flap.

The risk is the automation attached to the signal. With vSphere HA VM Monitoring enabled, a sustained red heartbeat plus no I/O causes HA to reset the VM. If the root cause is CPU starvation, the reset does not fix it: the VM returns, gets descheduled again, loses heartbeats again, and HA resets again. A misread cascades into restart loops.

Always correlate heartbeat red with CPU ready time (%RDY), co-stop (%CSTP), and max-limited (%MLMTD) before declaring a crash. Starvation produces a VM that is alive but cannot run the Tools process fast enough to send heartbeats. That distinction decides whether you restart the VM or rebalance the host.

What this means

VMware Tools (or open-vm-tools) runs a process inside the guest that sends periodic heartbeats to the VMkernel via hostd. The API exposes this as guestHeartbeatStatus with four values:

• green: heartbeat received normally
• yellow: intermittent heartbeat, often tied to guest load
• red: no heartbeat received; guest stopped responding or Tools failed to send
• gray: Tools not installed or status unknown

Red on a VM that vCenter believes is powered on is the ambiguous case. The hypervisor cannot tell, from the heartbeat alone, whether the guest panicked, hung, or simply cannot get enough CPU time to run the Tools process.

VM Monitoring (the HA feature, distinct from host-level HA) does not reset on heartbeat loss alone. It also checks for disk and network I/O activity over an interval controlled by das.iostatsInterval, which defaults to 120 seconds. Only when both heartbeats and I/O are absent for the configured failure interval does HA reset the VM. The failure interval depends on monitoring sensitivity: High is roughly 30 seconds, Medium 60 seconds, Low 120 seconds.

The I/O guard is the single most important differentiator in practice. A CPU-starved VM that is still doing I/O will not be reset, even if its heartbeat is red. A completely stalled VM, with no heartbeats and no I/O, will be reset whether the cause is a crash or a hard hang.

Gray is an instrumentation gap, not an outage. A VM without Tools cannot be heartbeat-monitored, and VM Monitoring has no signal to act on. Treat gray as a Tools installation or startup problem, not as a crash.

Common causes

Quick checks

# Heartbeat and Tools status per VM (read-only)
Get-VM | Select Name,
  @{N='PowerState';E={$_.PowerState}},
  @{N='ToolsStatus';E={$_.ExtensionData.Guest.ToolsRunningStatus}},
  @{N='Heartbeat';E={$_.ExtensionData.GuestHeartbeatStatus}}

# CPU ready time per VM. Convert raw ms to a percentage:
#   value_ms / (20000 * vCPU_count) * 100 = %RDY
Get-Stat -Entity (Get-VM "myvm") -Stat cpu.ready.summation -Realtime

# On the host running the VM, open the CPU view and read
# %RDY, %CSTP, %MLMTD per VM world
esxtop
#   press 'c' for CPU view

# HA VM Monitoring reset events in the last 24 hours
Get-VIEvent -Types "VmResettingEvent" -Start (Get-Date).AddHours(-24)

# vCenter version and build
$global:DefaultVIServer | Select Version, Build

The das.iostatsInterval cluster setting is not exposed by a single stock cmdlet in every PowerCLI build. Verify it in the vSphere Client under Cluster, Configure, vSphere HA, Advanced Options, and confirm it is 120, not 0. A value of 0 disables the I/O activity guard entirely. Broadcom KB 419168 also says any value below 50 is not recommended because it can cause HA to misinterpret normal resource changes as a Tools heartbeat failure.

The vCenter-side transient gray behavior after a vpxd restart is release-dependent. Treat it as a hypothesis to confirm against the exact vCenter release notes and build; do not assume a fixed version range.

How to diagnose it

flowchart TD
    A["heartbeat red"] --> B{"VM powered on?"}
    B -- no --> C["host failure or power-off"]
    B -- yes --> D{"%RDY / %CSTP / %MLMTD high?"}
    D -- yes --> E["CPU starvation"]
    D -- no --> F{"disk or net I/O continuing?"}
    F -- yes --> G["Tools process issue"]
    F -- no --> H["guest crash or hang"]
    E --> I{"I/O continuing?"}
    I -- yes --> J["HA will not reset"]
    I -- no --> K["HA may reset"]

1. Confirm the VM is actually powered on. If not, the red is a downstream symptom of a host failure or power-off, not a guest crash.
2. Pull heartbeat and Tools status. If Tools status shows Tools not running, the red may be a Tools problem, not a guest crash.
3. SSH to the host running the VM and read %RDY, %CSTP, %MLMTD in esxtop. Elevated ready time with a red heartbeat points to starvation.
4. Check whether the VM is still generating I/O. If %RDY is high but I/O continues, VM Monitoring will not reset the VM, and the red is a starvation artifact, not a crash.
5. If %RDY is low and I/O has stopped, the guest has likely crashed or hung. Check the console screenshot HA saved to the VM’s datastore folder for forensic context.
6. Look for recent VmResettingEvent entries. These tell you whether HA already acted, and when.
7. Check the vCenter version and whether a vpxd restart preceded the red. Transient red or gray after a restart is often vCenter-side, not guest-side.
8. Verify das.iostatsInterval. A value of 0 disables the I/O guard and makes HA reset on heartbeat loss alone.

There is no Broadcom-documented CPU ready percentage at which heartbeat delivery is guaranteed to fail. The threshold is environment-dependent, driven by vCPU count, host contention pattern, and how the guest scheduler prioritizes the Tools process. Correlate %RDY spikes with the heartbeat red timestamps in your own environment rather than trusting a fixed cutoff.

Metrics and signals to monitor

Fixes

If the cause is CPU starvation

Reduce %RDY directly rather than tuning around it.

• vMotion the VM to a less contended host, or reduce contention on the current host by moving other VMs off.
• Right-size vCPUs. An oversized SMP VM accumulates co-stop and ready time because the scheduler must find many free pCPUs at once. Fewer vCPUs schedule more easily and often run faster.
• Remove forgotten CPU limits when %MLMTD is non-zero. Limits throttle identically to contention but are invisible to the guest.
• Moderate HA sensitivity if resets are firing on load spikes while you fix the host. This does not cure starvation, but it stops the restart loop.

If the cause is a guest crash or hang

• Review the console screenshot HA saved and the in-guest crash logs (Windows BSOD dump, Linux kernel panic).
• Treat the crash as an application or OS problem, not a vSphere problem.
• If crashes repeat, capture a memory dump before restarting so the forensic state is preserved.

If the cause is a Tools process problem

• Restart the Tools service inside the guest, or reinstall Tools.
• Confirm the status moves back to green. A VM stuck on gray is unmonitorable by VM Monitoring.

If the cause is HA over-sensitivity

• Verify das.iostatsInterval is 120, not 0. A value of 0 disables the I/O guard and makes HA reset on heartbeat loss alone.
• Lower monitoring sensitivity if resets fire on normal load spikes. High, at roughly 30 seconds, is aggressive for loaded VMs that briefly stall the Tools process.

Prevention

• Monitor %RDY, %CSTP, %MLMTD per VM alongside heartbeat. Starvation is the preventable cause of false red. If you only watch heartbeat, you cannot tell a crash from a scheduling problem.

• Track das.iostatsInterval as configuration drift, especially in VCF-managed clusters. Broadcom KB 419168 documents VCF 5.2 deployments in which the setting is automatically configured to 0 and recommends restoring the default 120.

• Keep vCenter patched. If heartbeat-gray follows a vpxd restart, check the vCenter release notes and fixed-build status before treating it as guest-side.

• Close gray gaps. A VM without Tools is invisible to VM Monitoring. Treat missing Tools as an inventory problem.

• Watch storage latency too. A VM stalled by extreme datastore latency can look like a heartbeat loss, and if I/O also stops, HA may reset it. See the storage latency guides below.

How Netdata helps

• Correlate heartbeat red with per-second CPU ready, co-stop, and max-limited on the same view, so starvation is visible next to the symptom rather than in a separate tool.
• Per-host CPU utilization and per-VM ready time show which host is oversubscribed and needs a vMotion, turning a vague “VM keeps resetting” into a concrete placement decision.
• HA reset events and configuration drift such as das.iostatsInterval changes surface as events you can alert on.
• vCenter service health, NTP offset, and version context help separate transient API flaps from genuine guest problems.

Related guides

• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere active vs consumed vs granted memory: why the percentage lies
• How vSphere actually works in production: a mental model for operators