vSphere host 'Not Responding': dead, isolated, or is hostd hung?

When vCenter shows an ESXi host as “Not Responding”, it has stopped receiving heartbeats from that host and cannot reach it on the management plane. This is distinct from “Disconnected” (a deliberate state set by an admin, or caused by license expiry) and from “Maintenance” (an intentional state for patching). The greyed-out host icon is one of the most paged-on vSphere symptoms because the same UI state covers at least four different underlying conditions.

The single most important triage question is scope. If one host is Not Responding, the problem almost always lives on that host: its management network, its hostd/vpxa agents, or its hardware. If many hosts flip to Not Responding at the same time, the hosts are probably fine and the problem is either vCenter itself (a vpxd restart, certificate push, or overload) or the management network between vCenter and the cluster. Treating a vCenter-side outage as a cluster-wide host failure is a classic way to make things worse.

One frequent false positive sits inside that scope question: restarting vpxd causes all hosts to briefly appear Not Responding while vCenter re-establishes host connections, typically for 5 to 15 minutes in large environments. If your page fired during a known vpxd restart or VCSA boot, wait before escalating. The hosts are running; vCenter just has not reconnected yet.

Finally, distinguish “Not Responding” from “dead”. A Not Responding host may still be serving VMs perfectly well. It is unmanageable through vCenter, but the workloads keep running. Many Not Responding incidents are management-plane outages, not workload outages, and the right first move is often to leave VMs alone and fix the path to the host.

What this means

The Not Responding state is vCenter’s verdict that heartbeat traffic from the host has stopped. vCenter and ESXi exchange heartbeats over the management network, and ESXi also signals liveness through datastore heartbeats that HA uses to disambiguate network partitions from real host death. When network heartbeats stop, vCenter waits for the heartbeat timeout window (default 120 seconds, configurable via config.vpxd.heartbeat.notRespondingTimeout) and then marks the host Not Responding. If datastore heartbeats are still updating, the host is alive and HA will treat it as “isolated”, not “dead”, and follow the configured isolation response.

The reverse matters too: if both network and datastore heartbeats stop, the host is effectively dead from vCenter’s perspective and HA will restart its VMs on surviving hosts. So the same UI label can mean “I cannot talk to a healthy host” or “the host is gone and I am recovering its VMs.” Always confirm which before acting.

The mechanism behind a real single-host Not Responding is usually one of three things: the management network path is broken (cable, VLAN, firewall, pNIC, switch port security), the hostd agent on the ESXi host has crashed or hung (so vCenter’s vpxa cannot proxy commands to it), or the host itself has failed (PSOD, hardware fault, power loss). On the vCenter side, the same UI state appears when vpxd cannot keep up with heartbeat processing, when certificates break the trust chain, or when vpxd has just restarted.

flowchart TD
    A["Host shows Not Responding"] --> B{"How many hosts?"}
    B -->|"One host"| C["Host-side fault"]
    B -->|"Many hosts at once"| D["vCenter or network"]
    C --> E{"VMs still running?"}
    E -->|"Yes"| F["hostd hung or mgmt net down"]
    E -->|"No"| G["Host dead: PSOD or HW"]
    D --> H{"vpxd restart in last 15 min?"}
    H -->|"Yes"| I["False positive: wait"]
    H -->|"No"| J["vpxd overload, certs, or net"]
    F --> K{"Datastore heartbeat updating?"}
    K -->|"Yes"| L["Mgmt network partition"]
    K -->|"No"| M["hostd/vpxa crash on host"]

Common causes

Quick checks

Run these before touching anything. They are read-only.

# On vCenter (VCSA SSH): is vpxd actually up, and how long has VCSA been booted?
service-control --status vpxd
uptime

# How many hosts are Not Responding right now? (PowerCLI)
Get-VMHost | Group-Object ConnectionState

# Recent host connection state changes in vpxd.log
grep -i "NO_RESPONSE\|notResponding\|FixNotRespondingHost" /var/log/vmware/vpxd/vpxd.log | tail -30

# On the suspect ESXi host (SSH or DCUI): is hostd alive?
/etc/init.d/hostd status
/etc/init.d/vpxa status

# Has vpxa lost contact with hostd? Look for the smoking gun
grep -i "Can't connect to hostd\|hostd.*shut" /var/log/vpxa.log | tail -20

# Management vmk interface up and on the right network?
esxcli network ip interface list
esxcli network ip connection list | grep -E "902|443"

# Is the host's clock sane? Time skew breaks cert and heartbeat validation.
# ESXi does not ship ntpq/chronyc; list configured NTP servers and rely on
# monitoring for offset rather than a host-side offset query.
esxcli system time get
esxcli system ntp get

# Host certificate still valid?
openssl x509 -in /etc/vmware/ssl/rui.crt -noout -dates

# HA view of the host from the host side
vim-cmd hostsvc/runtimeinfo | grep -i connectionstate
# FDM agent: verify VIB is installed and process is running
esxcli software vib list | grep -i fdm
ps -T | grep fdm

These commands collapse the diagnostic tree quickly. If service-control --status vpxd shows vpxd STOPPED or restarting, or VCSA uptime is only a few minutes, the page is almost certainly a vCenter-side false positive. If only one host is affected and /etc/init.d/hostd status on that host reports problems, the host is the patient, not vCenter.

How to diagnose it

1. Establish scope first. Run Get-VMHost | Group-Object ConnectionState. If only one host is Not Responding, work the host path. If many are, treat it as a vCenter or network incident and do not touch the hosts individually.
2. Rule out the vpxd restart false positive. Check VCSA uptime, service-control --status vpxd, and the recent vpxd.log for restart markers. If vpxd restarted in the last 15 minutes, suppress the page and wait. In large environments reconnect can take longer.
3. Decide dead vs isolated for a single host. From another host in the same cluster, check whether datastore heartbeats from the suspect host are still updating. If HA marks the host “isolated”, VMs are running and the issue is the management network. If both heartbeats stopped, expect HA restarts and treat the host as failed.
4. Confirm the host is reachable out of band. Log into the BMC/iLO/iDRAC, the DCUI, or SSH. A host unreachable on every channel, including BMC, is almost certainly a hardware or PSOD event. Check vmkernel.log for MCE or PSOD signatures.
5. If SSH works but vCenter does not, focus on hostd and vpxa. The host is alive but its management agents are not. vpxa.log errors of the form Can't connect to hostd indicate hostd is the failing component. hostd.log will usually show why: a memory hard-limit panic, a stuck storage callback, a corrupted /productLocker/vmtools/, or certificate/TLS errors.
6. If many hosts are affected but vCenter is not freshly restarted, check the path from VCSA to the hosts. Verify DNS resolution both directions, routing, and that TCP 443, TCP 902, and UDP 902 are not blocked. Heartbeats from vpxa to vpxd ride UDP 902; TCP 902 carries NFC (provisioning, cloning, VMDK upload); TCP 443 carries HTTPS/API traffic. Check vpxd CPU and task queue: a vpxd overload cascade drops heartbeat processing while hosts are healthy.
7. Check certificates when hosts flap. Not Responding that flaps back to Connected often points to STS or machine certificate issues rather than host failure. Inspect /var/log/vmware/sso/ and run the STS certificate check. NTP drift of more than a few minutes will cause the same flapping because token validation fails intermittently.
8. Check the host’s managed IP if it changed recently. If the host management IP was readdressed but vCenter’s database still references the old IP, vpxa cannot bind the heartbeat socket. The vpxa.log shows Failed to bind heartbeat socket with Cannot assign requested address because the stored IP no longer matches any interface on the host (VCDB vpx_host.ip_address).

Metrics and signals to monitor

Fixes

One host: management network partition

Confirm the host is alive through datastore heartbeats or BMC before changing anything. Do not let HA act on isolation if the host is healthy and you can fix the network first. Restore the path: check the physical switch port, VLAN tagging, port security on the switch (which can suppress the host’s MAC), pNIC teaming state, and the management vmkernel interface. If HA has an aggressive isolation response, weigh whether you have time to fix the network before HA shuts VMs down. Once the network path is back, the host returns to Connected automatically.

One host: hostd crash or hang

Restarting hostd is the usual recovery, but understand why it failed first if logs are accessible. Warning: restarting hostd drops vCenter management of that host for a short window and interrupts any in-flight task on the host; VMs keep running but do not trigger it reflexively or during a known storage outage.

Common causes include the ESXi 7.0 hostd memory hard-limit panic (preceded by SetVigorNotificationTime: Not connected entries in hostd.log), the ESXi 6.5/6.7 crash from a corrupted /productLocker/vmtools/ folder (copy it from a working host as the workaround), and storage-induced hangs where hostd is blocked on I/O after an array outage. If the underlying cause is storage, restarting hostd before the storage path recovers will not help. For repeated hostd panics, capture /var/log/hostd.log and engage support rather than bouncing the service on a loop.

Many hosts: vpxd overload

Identify the load source. Misbehaving SDK clients (backup solutions running snapshot storms, scripts opening hundreds of concurrent sessions, DRS thrash after maintenance) saturate the vpxd thread pool and heartbeat processing falls behind. Throttle or pause the offending client, set DRS to manual if it is the source, and let vpxd catch up. Do not restart vpxd reflexively: it drops all in-flight tasks and requires several minutes to rebuild the inventory cache, during which all hosts appear Not Responding.

Many hosts: vpxd restart or VCSA boot

Usually no action is needed. Wait for vpxd to re-establish host connections, which can take 5 to 15 minutes in large environments. If reconnect does not complete, check whether a certificate or STS problem is preventing hosts from re-establishing trust.

Many hosts: certificate or STS failure

If machine SSL, STS signing, or solution user certificates expired or were pushed incorrectly, hosts will flap or fail to reconnect. After upgrading vCenter to 8.0 U2, a non-CA certificate in the TRUSTED_ROOTS store causes hostd to discard it and issue an ssl_reset, restarting vpxa in a loop (Broadcom KB 318860). Renew or correct the certificates through the documented procedure, then let hosts reconnect. Verify NTP on both sides first, since clock skew makes valid certificates look invalid.

Many hosts: vCenter-side network or DNS

If vCenter cannot resolve host names or reach TCP 902 / TCP 443, every host appears Not Responding while the hosts themselves are fine. Verify DNS from VCSA to the hosts and back, routing, and any firewall change that followed a recent change window. This is one of the easiest causes to miss because the vSphere UI looks like a cluster-wide host failure.

One host: hardware failure or PSOD

If the host is unreachable on every channel including BMC, treat it as failed. Let HA restart VMs on surviving hosts. Investigate vmkernel.log for MCE entries, check CIM hardware health for uncorrectable ECC or PSU faults, and do not bring VMs back onto the host until the hardware is verified.

Prevention

• Monitor host connection state per host, and alert on the count of hosts Not Responding simultaneously so a single-host event is distinguishable from a vCenter event.
• Treat a vpxd restart as a maintenance event in alerting. Suppress host Not Responding pages for 15 minutes after a known vpxd restart or VCSA boot.
• Track all certificate expiry dates, especially the STS signing certificate, with 60 to 90 day lead times. The STS cert is the one that takes down authentication silently.
• Monitor NTP offset on VCSA and on every ESXi host. Skew over a few minutes causes intermittent auth and heartbeat failures that look exactly like host flapping.
• Monitor vpxd CPU, task queue depth, and SDK response time. Overload-driven Not Responding is preceded by clear saturation signals.
• Monitor VCSA per-partition disk usage, not just root. Log bombs during an unrelated incident fill /storage/log and amplify the original problem.
• Track ESXi hardware health (CIM, ECC rates) and storage path state. APD/PDL events can hang hostd and turn a storage incident into a host Not Responding incident.
• Keep ESXi build versions at or below the vCenter version per the VMware Product Interoperability Matrix. An ESXi host newer than its vCenter is unsupported and can present as Not Responding.

How Netdata helps

• Correlate host connection state with VCSA vpxd restart events and VCSA uptime to turn the most common false positive, the post-restart mass Not Responding, into a non-page.
• Pair host connection state with vpxd CPU, task queue depth, and SDK response time to surface overload-driven Not Responding before operators blame the hosts.
• Use anomaly detection on NTP offset, certificate expiry countdowns, and per-partition VCSA disk utilization to flag the slow-burn precursors (clock skew, cert expiry, log partition filling) that later manifest as host flapping.
• Bring ESXi hardware health, storage path state, and VMkernel error-rate signals into the same timeline as host connection state so a real dead host is distinguishable from an isolated one before HA acts.
• Putting vCenter-side and host-side signals on one timeline removes the “is it the host or is it vCenter?” argument from the first minute of the incident.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM