vCenter vPostgres 'too many clients already': connection exhaustion

vCenter operations time out. Power-on tasks hang in “Running” state. The vSphere Client is sluggish or fails to load. PowerCLI sessions return errors. Backup jobs fail mid-run. SSH into the VCSA and tail the vpxd or vPostgres logs, and you find the string that anchors the incident: FATAL: sorry, too many clients already.

The embedded vPostgres database is rejecting new connections because it has reached max_connections. VMware tunes this value per deployment size, and the appliance’s pg_tuning script rewrites it from the allocated memory when vmware-vpostgres is restarted. At the limit, anything needing a database connection fails immediately. This is a cliff-edge failure, not graceful degradation.

The fix is not to raise max_connections. Find the client generating the sessions. The database is rarely the root cause.

What this means

vpxd, the core vCenter daemon, connects to vPostgres for every inventory read, task state write, event insert, and statistics rollup. Other VCSA services also open database connections. The total concurrent count is bounded by max_connections, which VMware sets based on VCSA memory allocation. When active connections hit the ceiling, the postmaster rejects new attempts with FATAL: sorry, too many clients already.

The rejection is binary. A service that needs a connection either gets one or does not. There is no queueing at the database layer. vpxd task processing stalls because it cannot persist task state. Stats rollup falls behind because it cannot write. Host heartbeat processing slows because inventory updates cannot be committed. The visible symptom is “vCenter is slow or broken,” but the mechanism is a saturated connection pool behind a fixed PostgreSQL limit.

The correlate that almost always appears alongside this error is a growing vpxd task queue. Tasks pile up in “Running” or “Queued” state because the threads that would execute them are blocked acquiring database connections. If you see the error string and task queue depth climbing simultaneously, you are looking at the same failure from two angles.

flowchart TD
    A[SDK clients open sessions] --> B[vpxd executes tasks]
    B --> C[Each task needs vPostgres connection]
    C --> D{Active connections below max?}
    D -- yes --> E[Task proceeds]
    D -- no --> F[FATAL: too many clients already]
    F --> G[Task stalls in queue]
    G --> H[vpxd CPU rises, SDK response degrades]
    H --> B

Peak connection count should stay below roughly 70% of max_connections. Above that, operational bursts, maintenance connections, and superuser reserved connections leave no headroom. Once you cross the line, the cliff is immediate.

Common causes

Quick checks

These are read-only and safe to run during an incident. Run psql as the postgres OS user (su - postgres) to avoid peer auth issues. If psql fails with a socket path error, set PGHOST=/var/run/vpostgres explicitly; on current VCSA builds the Unix socket is /var/run/vpostgres/.s.PGSQL.5432, while older builds may still default to /tmp/.

# Current max_connections as tuned by the appliance
/opt/vmware/vpostgres/current/bin/psql -U postgres -A -t -c "SHOW max_connections;"

# Active connection count and ratio against max
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT count(*) AS active_connections, (SELECT setting::int FROM pg_settings WHERE name='max_connections') AS max_connections FROM pg_stat_activity;"

# Long-running queries holding connections
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT pid, now() - query_start AS duration, state, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;"

# Long-running transactions that block autovacuum and hold the horizon
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT pid, now() - xact_start AS xact_duration, state, query FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start LIMIT 10;"

# VCSA service health
service-control --status --all

# vpxd errors related to database connectivity
grep -i "database.*connect\|postgres.*fail\|ODBC\|too many clients" /var/log/vmware/vpxd/vpxd.log | tail -50

# Offending SDK session if a monitoring tool is the cause
grep -i "queryAvailableMetric" /var/log/vmware/vpxd/vpxd-profiler.log | tail -20

# Database partition usage (full disk produces a different but related failure)
df -h /storage/db

# VCHA replication state, if configured
/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT client_addr, state, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes FROM pg_stat_replication;"

How to diagnose

1. Confirm the error is vPostgres-side. Grep the vpxd log for the exact string and for ODBC or database connectivity errors. If the string appears, you are in connection exhaustion.
2. Read max_connections and the active count from pg_stat_activity. Compute the ratio. Above 70% of max at peak is a headroom problem. At 100%, you are in the incident.
3. Identify what is holding connections. Sort pg_stat_activity by duration and by application name if available. Long-running queries suggest stats rollup or a misbehaving query. Many short-lived sessions from the same client suggest an SDK storm.
4. Cross-reference with the vpxd task queue. A queue growing while vpxd CPU is elevated and connections are near max confirms the overload cascade.
5. Identify the offending SDK client. Check SDK session count by client IP. Check vpxd-profiler.log for repeated calls to PerformanceManager.queryAvailableMetric from a single session UUID and ClientIP. That is the classic signature of a monitoring or automation tool generating sessions.
6. Rule out storage latency on the VCSA VM’s datastore. High DAVG or KAVG stretches transaction times, which holds connections longer and worsens exhaustion. The VCSA VM should not sit on an overcommitted or slow datastore.
7. If VCHA is configured, verify the passive node has memory identical to the active node. A passive node with less memory gets a lower max_connections from the tuning script, and the active node cannot open enough connections for PostgreSQL replication, producing “PostgreSQL replication is not in progress” errors that share the root cause.

Metrics and signals to monitor

Fixes

Do not raise max_connections by hand. It is tuned to the appliance’s resources, and the tuning script overwrites it on the next vmware-vpostgres restart. Treat the database limit as a fixed boundary and fix the load.

Throttle or disconnect the offending SDK client

If SDK session counts by client IP point to a single integration, that is the root cause. Throttle the client, pause the backup job, or disconnect the rogue session. For monitoring tools hammering PerformanceManager.queryAvailableMetric, the fix lives in the tool’s configuration: reduce collection frequency, reduce the metric scope, or upgrade the tool if a vendor fix exists.

Reduce the vpxd task load

If the queue is growing because of legitimate operational burst, reduce concurrency. For DRS storms, temporarily set DRS to manual. For post-maintenance bursts, stagger hosts exiting maintenance mode. For backup storms, spread backup jobs across wider windows so snapshot create/delete operations do not pile up.

Address stats rollup overlap

If stats rollup overlaps with heavy operational windows and consumes connections, check the statistics level. Levels 3 and 4 generate substantially more write load than levels 1 and 2. Reducing to level 1 or 2 lowers baseline connection pressure from rollup. In common VCDB schemas, the rollup stores are vpx_hist_stat1 through vpx_hist_stat4 with companion vpx_sample_time1 through vpx_sample_time4; confirm the table names in your vCenter build before querying them.

Clear long-running transactions

Long-running idle-in-transaction sessions hold connections and block autovacuum across all tables. Identify them with the long-running transaction query and work with the owning service or integration to close them. Do not kill PostgreSQL backends casually during an incident without understanding what vpxd or another VCSA service is doing.

Fix storage latency on the VCSA datastore

If DAVG or KAVG on the VCSA VM’s datastore is elevated, transactions take longer, connections are held longer, and the exhaustion cliff arrives sooner. Migrate the VCSA VM to faster or less contended storage. The VCSA should not sit on an overcommitted host or a noisy datastore.

Correct VCHA passive node sizing

If the passive node has less memory than the active, the tuning script assigns it a lower max_connections, replication breaks, and the active node accumulates WAL. Size the passive node identically to the active. Restarting vmware-vpostgres via vmon-cli --restart vmware-vpostgres reruns the tuning logic and applies the corrected value.

When restarting is unavoidable

Restarting vpxd drops all in-flight tasks and requires minutes to tens of minutes to rebuild the inventory cache from the database. Use it only when the offending client cannot be throttled and the queue is irrecoverable. Restarting vmware-vpostgres will not raise max_connections above what the appliance memory supports, so it is not a workaround for exhaustion.

Prevention

• Track active connections against max_connections continuously. Alert at 70% sustained. Treat 90% as a page.
• Monitor SDK session count by client IP. Baseline each integration. A new integration or a misconfigured collection interval is the most common recurring cause.
• Monitor vpxd task queue depth and average task duration. Sustained queue depth above 0 outside peak hours is an early signal.
• Keep statistics level at 1 or 2 unless a specific investigation requires more. Return to baseline after the investigation.
• Keep the VCSA VM on fast, dedicated storage. Monitor its datastore latency independently of the rest of the cluster.
• In VCHA configurations, size the passive node identically to the active. Monitor replication lag.
• Track vpxd log error rate. The “too many clients already” string is the terminal symptom. Earlier signals appear as ODBC warnings and database connectivity errors.
• Re-baseline after every new integration. Backup tools, monitoring platforms, orchestration, and custom scripts are each potential session sources.

How Netdata helps

• Per-second PostgreSQL connection metrics show the ramp toward max_connections before the error string appears in logs.
• Correlating active connection count with vpxd CPU and task queue depth distinguishes a connection-bound stall from a compute-bound stall, which changes the fix.
• Per-client SDK session tracking requires custom instrumentation (log parsing of vpxd-profiler.log), but once collected, it surfaces the misbehaving integration by IP so you can throttle the right client instead of restarting services.
• Anomaly detection on connection count catches the monitoring-tool-hammering pattern early, before vPostgres CPU pins at 100%.
• VCSA VM-level metrics from the hypervisor (CPU ready, datastore latency, memory balloon) reveal whether storage or host contention is lengthening transactions and compounding the exhaustion.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong