vSphere ESXi hardware health: ECC errors, fan failure, and thermal throttling

Hardware faults do not surface through hypervisor performance counters. They appear either as a Purple Screen of Death (PSOD) that kills every VM on the host instantly, or as slow degradation that looks like software misconfiguration until someone checks temperature sensors. Every DIMM error, failed fan, and degraded disk is the hypervisor’s problem.

Three failure classes dominate production incidents on ESXi: ECC memory errors that progress from correctable to uncorrectable, fan failures that trigger thermal throttling and silently cap CPU frequency, and predictive disk failures that kick off RAID rebuilds consuming storage I/O. All three share a common operational trap: without the vendor CIM provider VIB installed, most sensors report “unknown” or are absent entirely, leaving you blind to the root cause.

The CIM stack is also changing. As of ESXi 8.0, CIM and SLP services are deprecated, with informational VOBs warning that removal is planned for the next major release. The sfcbd CIM server has been disabled by default on fresh installs since ESXi 6.5 and only starts automatically when a third-party CIM VIB is present. If you are running ESXi 8.0 Update 3 or later and relying on CIM sensor data, you need a plan for what replaces it.

What this means

The critical pattern with ECC errors is progression. Correctable errors (CE) are the leading indicator. Broadcom KB 414711 documents a three-stage failure: corrected memory errors appear in vmkernel.log, then CPU heartbeat failures occur (“PCPU didn’t have a heartbeat for X seconds”), then a PSOD follows with “NMI IPI: Panic requested by another PCPU.” The window between stage 1 and stage 3 can be minutes to hours. When you see 10 or more CE errors within a few minutes on the same memory channel, that DIMM is about to fail.

Thermal throttling is the stealthiest of the three. ESXi does not log CPU throttling events. The CPU simply runs at a reduced frequency. The only indirect indicators are temperature sensors reading high and elevated host CPU utilization without corresponding VM throughput. Teams chase CPU contention, check ready time, inspect limits, and find nothing wrong because the problem is physical, not scheduling.

flowchart TD
    A[Physical hardware fault] --> B{Sensor class}
    B -->|DIMM ECC| C[MCE in vmkernel.log]
    B -->|Fan failure| D[CPU temperature rising]
    B -->|Disk SMART| E[Predictive failure flag]
    C --> F{CE rate >10/hr same channel?}
    F -->|Yes| G[Heartbeat loss then PSOD]
    D --> H[Thermal throttling active]
    H --> I[Silent MHz cap, VMs slow]
    E --> J[RAID rebuild starts]
    J --> K[Datastore latency spike]

Common causes

Quick checks

All commands are safe and read-only. Run them over SSH or via the ESXi shell.

# Read all IPMI sensor values: temperatures, fan speeds, voltages, PSU status
esxcli hardware ipmi sdr list

# Check memory ECC error summary
esxcli hardware memory get

# Search vmkernel.log for Machine Check Exception entries
grep -i "MCE\|MCA\|Machine Check" /var/log/vmkernel.log | tail -50

# Search for PCPU heartbeat loss (PSOD precursor)
grep -i "heartbeat" /var/log/vmkernel.log | tail -20

# Get SMART data for a specific disk device
esxcli storage core device smart get -d naa.600508b1001c3d0

# Get hardware platform summary (vendor, model, BIOS version)
esxcli hardware platform get

# Check whether the CIM broker service is running
/etc/init.d/sfcbd-watchdog status

# List installed vendor CIM provider VIBs
esxcli software vib list | grep -i "cim\|openmanage\|ssa\|hp-health\|cimc"

If the IPMI SDR list returns nothing or reports “unknown” for every sensor, you are likely missing the vendor CIM provider VIB. Generic IPMI sensors (CPU temp, fan speed) may still appear via the /dev/ipmi driver that hostd uses directly, but storage controller health and vendor-specific sensors require the VIB.

How to diagnose it

1. Confirm sensor visibility first. Before investigating any hardware fault, verify that your host actually reports sensor data. Run esxcli hardware ipmi sdr list and check whether fan speeds, temperatures, and PSU status populate. If they do not, install the vendor CIM provider before proceeding. Without sensors, you are guessing. Also confirm the sfcbd-watchdog service status, since the CIM broker must be running for provider data to flow.

2. For suspected ECC memory issues: Search vmkernel.log for MCE entries. Correctable errors appear with a format like MCA: [PCPU]: CE Poll G[status] B[bank] S[status] A[address] M[misc] Memory Controller Read Error on Channel [X]. Count occurrences per channel. If you see 10 or more CE entries within a few minutes on the same channel, that DIMM is failing. Look for the progression to heartbeat failures, which is the immediate PSOD precursor.

3. For suspected thermal throttling: Check IPMI temperature sensors. If CPU temperature is near or above the vendor threshold, check fan speeds. A single failed fan with redundant cooling is a P3; schedule replacement. All fans degraded, or a single fan failure on a non-redundant platform, is an emergency. Reduce host load immediately. Cross-reference with host CPU utilization: if utilization is high but VM throughput is low and ready time is normal, thermal throttling is the likely cause. Also verify the host power policy is set to “High Performance” rather than a balanced or low-power policy, which can independently cap CPU frequency and mimic throttling symptoms. On builds where the shell reports the package frequency, compare esxcli hardware cpu global get Current Speed with Max Speed; verify it against BMC temperatures and fans.

4. For suspected disk degradation: Pull SMART data with esxcli storage core device smart get -d <device-id>. Look for climbing reallocated sector counts, pending sectors, or media errors. If a disk is flagged predictive failed by the RAID controller, a rebuild may already be running. Check storage latency during this period, as rebuild I/O competes with VM workloads.

5. Rule out false positives. On ESXi 6.5 and 6.7, some IPMI sensors intermittently flip from green to red, generating false hardware health alarms. This was resolved in ESXi 6.5 U3 and 6.7 U2. If you are on an affected version, use esxcfg-advcfg -s <sensor-ID>[,<sensor-ID>...] /UserVars/HardwareHealthIgnoredSensors to ignore specific sensors, using the Node-Sensor ID suffix from esxcli hardware ipmi sdr list. Separately, after upgrading to ESXi 8.x, sensor flapping between red and green is often caused by an unsupported BIOS version sending corrupted IPMI data. The fix is a BIOS update to the version listed in the VMware HCL.

Metrics and signals to monitor

Fixes

Correctable ECC errors climbing toward uncorrectable

When CE errors appear in vmkernel.log, identify the failing DIMM from the MCE address and channel information. The bank and address fields in the MCE entry map to a physical DIMM slot per your server vendor’s documentation. Schedule a DIMM replacement during a maintenance window.

If the CE rate exceeds 10 per hour on the same channel, evacuate VMs off the host immediately via vMotion and place the host in maintenance mode before the PSOD occurs. Once a PSOD happens, every VM on the host crashes simultaneously. Evacuating before the uncorrectable error is strictly better than recovering after.

Fan failure causing thermal throttling

Check the BMC directly (iDRAC, iLO, IMM, XCC) for fan status and any thermal alerts the ESXi CIM layer may not surface. If a single fan has failed on a platform with redundant cooling, you have time but should schedule replacement. If cooling is non-redundant or multiple fans are degraded, reduce host load immediately.

Thermal throttling is not something you fix in software. The CPU is protecting itself by reducing frequency to stay below the thermal limit. The fix is physical: replace the failed fan, clean dust from heatsinks and airflow paths, verify the chassis is not in an environment exceeding its intake temperature spec, and reseat heatsinks if thermal paste has degraded.

Predictive disk failure and RAID rebuild impact

If SMART data shows a disk heading toward failure, proactively replace it before the RAID controller marks it failed and initiates a rebuild. A rebuild on a busy datastore saturates IOPS and spikes latency for every VM on that LUN or disk group.

If a rebuild is already running, expect elevated storage latency (DAVG) across all VMs on the affected datastore until it completes. Consider scheduling the rebuild during off-peak hours if the controller supports it. Monitor datastore latency and queue depth during the rebuild window. See vSphere datastore latency high: reading GAVG, DAVG, and KAVG for latency decomposition.

Missing sensor visibility (no vendor CIM VIB)

Install the vendor-specific CIM provider VIB for your hardware. Dell OpenManage, HP SSACLI or hp-health, and Cisco CIMC are the common providers. Without the appropriate VIB, most sensors show “unknown” and you lose all hardware health visibility.

Note that Dell dropped CIM provider support for LSI-based PERC controllers in their ESXi 6.7 image. VMware also blocked the LSI CIM provider from loading. If you are running ESXi 6.7 or later on Dell hardware with local RAID, disk health monitoring for the RAID controller may be broken regardless of VIB installation. In that case, monitor disk health through the iDRAC or OpenManage Server Administrator instead of through ESXi CIM.

Prevention

• Install vendor CIM VIBs on every host at deploy time. Without them, sensor data is absent and hardware faults are invisible until PSOD.
• Monitor ECC CE error rate per channel. Alert when any channel exceeds 10 errors per hour. This gives you hours of warning before uncorrectable failure.
• Track CPU temperature trends, not just thresholds. A gradual upward trend over weeks indicates degrading cooling (dust, paste, fan wear) before any single threshold trips.
• Verify host power policy is “High Performance.” Balanced or low-power policies cap CPU frequency independently of thermal events and mimic throttling symptoms.
• Forward vmkernel.log to a central syslog. MCE entries and heartbeat failures are the earliest hardware failure signals. If logs live only on the host, a PSOD may erase them.
• Plan for CIM deprecation on ESXi 8.0+. CIM and SLP are deprecated with removal planned for the next major release. Evaluate vendor-specific monitoring paths (iDRAC, iLO, XCC) before CIM disappears.
• Patch BMC firmware proactively. A known BMC issue caused multiple hardware sensor alert floods on ESXi 8.0 U3; the ESXi-side alert-flood issue was resolved in 8.0 U3g, and update the BMC firmware as well. BMC firmware bugs produce sensor noise that erodes alert trust.

How Netdata helps

• Correlates hardware sensor data with VM performance counters. When CPU temperature spikes and host utilization climbs but VM throughput drops, Netdata’s per-second metrics let you see the thermal throttling signature in a single view rather than stitching together separate tools.
• Surfaces ECC error rate trends. Monitoring correctable error counts per channel over time catches the progression toward uncorrectable failure hours before a PSOD, giving you time to evacuate VMs.
• Detects anomalous CPU utilization patterns. ML-based anomaly detection flags the specific pattern of high utilization without corresponding throughput that indicates thermal or power-management throttling, distinguishing it from genuine CPU contention.
• Correlates predictive disk failures with storage latency. When a SMART attribute starts climbing and datastore latency spikes simultaneously, Netdata shows both signals aligned in time, confirming the rebuild I/O impact across all VMs on the datastore.
• Tracks sfcbd-watchdog resource usage. The CIM broker memory leak is a known cause of host swapping. Monitoring its RSS catches the leak before it degrades management services.

Related guides

• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators