vCenter certificate expired: the STS signing cert outage nobody saw coming

vCenter is down. Not “slow” or “degraded.” Down. The vSphere Client shows a white screen or a 503. PowerCLI sessions fail to connect. API calls return authentication errors. ESXi hosts show as disconnected in bulk. Every integration that depends on vCenter (NSX, vRA, SRM, backup products) has lost connectivity simultaneously. VMs on the hosts are still running, but you cannot manage, migrate, or orchestrate anything.

You check the browser certificate on the vCenter URL. It looks fine. Months left. You check NTP. Synchronized. You check disk space. Plenty. Nothing in your standard monitoring explains why the entire management plane went dark at once.

The culprit is the STS signing certificate. It is not the certificate you see in the browser. It is an internal certificate used by the Security Token Service to sign SAML tokens for all vCenter authentication. When it expires, STS cannot issue valid tokens. Every authentication attempt fails. Every service that depends on inter-service authentication begins failing. There is no graceful degradation: everything works until the moment of expiry, then everything breaks together.

What this means

The STS signing certificate is separate from the machine SSL certificate that a browser sees and has an independent lifecycle. Its validity cannot be inferred from the current vCenter version alone. Broadcom documents scenarios with a lifetime of around two years, including systems originally installed at 6.5 U2 and later upgraded, and manually replaced STS certificates. Inspect the certificate’s actual Not After date instead of assuming a version-wide default.

When the STS signing cert expires, the cascade is immediate and total:

flowchart TD
    A[STS signing cert expires] --> B[STS cannot sign valid SAML tokens]
    B --> C[All authentication fails at once]
    C --> D[vSphere Client: white screen or 503]
    C --> E[PowerCLI and API: auth rejected]
    C --> F[ESXi hosts: bulk disconnect]
    C --> G[Integrations drop: NSX, vRA, SRM, backup]
    B --> H[Inter-service auth fails]
    H --> I[Management plane goes dark]

VMs continue running on ESXi hosts. The hypervisor plane is unaffected. But the management plane is completely inaccessible. You cannot log in, cannot migrate, cannot provision, cannot reconfigure. HA continues to operate on the hosts because the FDM agents run independently of vCenter, but you cannot reconfigure HA or respond to host failures through vCenter.

Common causes

Quick checks

These are safe, read-only commands. Run them from an SSH session to the VCSA.

# List all certificate stores in VECS
/usr/lib/vmware-vmafd/bin/vecs-cli store list

# Check machine SSL certificate dates
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT \
  --alias __MACHINE_CERT | openssl x509 -noout -dates -subject

# The STS signing certificate is not stored in VECS.
# STS_INTERNAL_SSL_CERT is a non-default legacy store and may not exist.
# Use Broadcom's vCert tool to inspect the STS signing certificate.

# Check the externally-facing certificate
echo | openssl s_client -connect localhost:443 2>/dev/null | \
  openssl x509 -noout -dates -subject

# On vCenter 7.0-9.0, inspect the STS signing certificate with vCert:
# View Certificate Info -> STS signing certificates

# Check service health
service-control --status --all

# Look for certificate errors in vpxd log
grep -i "ssl\|certificate.*expir\|cert.*invalid" /var/log/vmware/vpxd/vpxd.log | tail -50

# Count SSO authentication failures
grep -c "Authentication attempt failed\|LOGIN_FAILED\|AuthenticationFailed" \
  /var/log/vmware/sso/vmware-sts-idmd.log

# Verify NTP synchronization (clock skew causes false cert validation failures)
chronyc tracking

How to diagnose it

1. Confirm the symptom pattern. If all authentication fails simultaneously (not gradually), the browser cert looks fine, and multiple hosts disconnected at the same time, suspect the STS signing cert. A gradual degradation or single-service failure points elsewhere.

2. Check the STS signing certificate. On vCenter 7.0 through 9.0, use Broadcom’s vCert tool and select View Certificate Info → STS signing certificates. checksts.py is deprecated. The STS signing certificate is not stored in VECS, and the non-default legacy STS_INTERNAL_SSL_CERT store is not the certificate that signs SAML tokens.

3. Check NTP synchronization. Clock skew between VCSA and ESXi hosts can cause certificate validation failures even when certificates are not expired. Run chronyc tracking on the VCSA. An offset of more than 5 minutes will break Kerberos authentication and likely SAML token validation. NTP synchronization is a prerequisite for certificate health.

4. Check service health. Run service-control --status --all. If vmware-stsd (the STS service) is STOPPED or in a crash loop, that confirms the STS layer is the problem. If vpxd is also down, it may be because it cannot authenticate to dependent services.

5. Check host connection states. If multiple hosts show notResponding simultaneously, the problem is vCenter-side, not host-side. This distinguishes a cert expiry cascade from individual host failures.

6. Check vpxd logs for certificate errors. Grep for SSL, certificate, and authentication errors. The specific error patterns confirm which certificate is causing the failure.

Metrics and signals to monitor

Fixes

Warning: certificate renewal requires service restarts and will interrupt vCenter management. Take snapshots of all affected nodes before proceeding. Schedule a maintenance window.

STS signing certificate expired

For vCenter 7.0 through 9.0, use Broadcom’s current vCert tool and select Manage Certificates → STS signing certificates. checksts.py is deprecated. For vCenter 6.x, use Broadcom’s version-specific recovery procedure.

Do not attempt to renew other certificates first. If the STS cert is expired, certificate regeneration for other cert types will fail because dependent services like vmware-stsd and vmware-vapi-endpoint cannot start without a valid token.

Machine SSL certificate expired

Use /usr/lib/vmware-vmca/bin/certificate-manager to renew. This is simpler than STS renewal but still requires service restarts. Plan for a maintenance window.

VMCA root certificate expired

Renewing the VMCA root requires re-issuing all dependent certificates. This is a major operation, not a quick fix. Begin renewal at 120 days to allow for planning and cascading renewal of all dependent certs.

Post-renewal service restarts

After certificate renewal, some services cache old certificates and require explicit restarts. A “certificate renewed but services still failing” scenario is common. Restart services in dependency order using service-control.

Enhanced Linked Mode considerations

In Enhanced Linked Mode (ELM) topologies, all vCenter/PSC nodes in the SSO domain must be snapshotted before STS cert operations. A single-node snapshot is insufficient. In vSphere 6.x with external PSC, certificates exist on both the PSC and the vCenter. Both must be monitored and renewed.

Prevention

• Plan STS replacement within six months of expiry. Broadcom recommends replacement when the STS signing certificate has less than six months remaining. vCenter 7.0 U1 and later also issue weekly notifications starting 90 days before expiry and daily notifications during the final week.
• Monitor all certificate types, not just machine SSL. The STS signing certificate is not visible in the browser or stored in VECS. Use vCert on vCenter 7.0 through 9.0 and the version-specific Broadcom procedure on vCenter 6.x.
• Verify NTP synchronization as a prerequisite. Clock skew between VCSA and ESXi hosts can cause certificate validation failures even when certificates are not expired.
• Snapshot all nodes in ELM before cert operations. In Enhanced Linked Mode, all vCenter/PSC nodes in the SSO domain must be snapshotted. A single-node snapshot is insufficient.
• Document the renewal procedure per version. VMware has changed the certificate architecture significantly across versions. The tools and stores differ between 6.5, 6.7, 7.0, and 8.0. Verify which tools work on your specific version before you need them in an emergency.

How Netdata helps

• Correlate certificate expiry with authentication failures. If you feed certificate expiry dates and SSO authentication failure rates into Netdata, the correlation between approaching STS cert expiry and auth failure spikes is immediately visible on a single dashboard.
• Per-second service health metrics. Netdata monitors VCSA systemd services (vmware-stsd, vpxd, vmware-vapi-endpoint) at per-second resolution. A service entering a crash loop is visible within seconds, not at the next 5-minute polling interval.
• NTP offset tracking. Netdata’s chrony collector tracks clock synchronization continuously. Clock skew that could cause false certificate validation failures is caught before it breaks authentication.
• ESXi host connection state visibility. When host connection states are collected (via SNMP or vSphere API integration), bulk disconnection events with multiple hosts going notResponding simultaneously are immediately distinguishable from individual host failures.
• Log-based early warning. If you forward vpxd.log to Netdata, certificate and SSL error patterns can surface as metric anomalies before the full authentication failure cascade.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong