Elasticsearch CPU saturation: search, merges, GC, and hot-spotting

Search latency climbs, bulk indexing slows, and clients see EsRejectedExecutionException or HTTP 429. On data nodes, CPU is pinned above 90% and load average rises. Elasticsearch burns CPU in query execution, text analysis, segment merges, and JVM garbage collection. If saturation persists, search and write thread pool queues grow until the cluster rejects work.

The first trap is assuming all CPU belongs to Elasticsearch. os.cpu.percent includes every process on the host, co-located containers, and kernel work. In containers this diverges sharply from process.cpu.percent, which tracks only the Elasticsearch process. A node may show 89% OS CPU and 67% process CPU, while the container runtime reports a third figure. Attribute CPU correctly before tuning thread pools or adding nodes.

The second trap is cluster-wide averaging. Queries scatter to shards and gather results. One node with asymmetric load, a hot shard, or an expensive aggregation can bottleneck the entire request while other nodes idle. CPU saturation is a per-node problem that becomes a cluster-wide symptom.

What this means

Sustained high CPU on data nodes means the node cannot schedule threads fast enough. The search and write thread pools have bounded queues. As saturation persists, tasks wait longer. Once a queue fills, the pool rejects new tasks. Search queries fail; indexing drops documents unless the client retries.

On master-eligible nodes, sustained CPU above 80% is a different failure mode. The master must serialize and publish cluster state updates. CPU pressure delays this work, so pending cluster tasks back up. Shard allocation, mapping updates, and index creation stall. Master saturation surfaces as sluggish administrative operations and delayed recovery, not always as rejections.

Elasticsearch CPU falls into four main consumers:

• Search execution: scoring, aggregations, and script evaluation
• Indexing: text analysis, inversion, and ingest pipeline processing
• Segment merges: background Lucene operations that compact segments
• JVM garbage collection: young and old generation cycles that pause application threads

Common causes

Quick checks

# Rank nodes by OS CPU and load average
curl -s 'http://localhost:9200/_cat/nodes?v&h=name,node.role,cpu,load_1m&sort=cpu:desc'

# Compare OS CPU to ES process CPU per node
curl -s 'http://localhost:9200/_nodes/stats/os,process?filter_path=nodes.*.os.cpu.percent,nodes.*.process.cpu.percent'

# Identify what threads are consuming CPU right now
curl -s 'http://localhost:9200/_nodes/hot_threads'

# Check thread pool queues and rejections
curl -s 'http://localhost:9200/_cat/thread_pool/write,search?v&h=node_name,name,active,queue,rejected'

# Inspect JVM GC behavior and heap pressure
curl -s 'http://localhost:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.gc,nodes.*.jvm.mem.heap_used_percent'

# Check merge concurrency and segment count
curl -s 'http://localhost:9200/_cat/nodes?v&h=name,merges.current,segments.count,segments.memory'

# View per-processor ingest latency
curl -s 'http://localhost:9200/_nodes/stats/ingest?filter_path=nodes.*.ingest'

# Find the current master and assess its load
curl -s 'http://localhost:9200/_cat/master?v'

How to diagnose it

flowchart TD
    A[High CPU sustained] --> B{One node or many?}
    B -->|One node high| C[Check shard distribution and thread pool queues]
    B -->|Many nodes high| D{process.cpu vs os.cpu}
    D -->|os.cpu only| E[External process or host noise]
    D -->|Both high| F[Run hot_threads API]
    F -->|search frames| G[Expensive queries]
    F -->|ingest or grok frames| H[Pipeline bottleneck]
    F -->|merge or IndexWriter| I[Merge backlog]
    F -->|GC threads| J[GC pressure]
    J --> K[Check old GC frequency and heap floor]
    C --> L[Hot-spotting: reroute or rebalance shards]
    G --> M[Cancel slow tasks and optimize mappings]
    H --> N[Fix grok or dissect patterns, or scale ingest]
    I --> O[Reduce merge threads or force-merge cold indices]

1. Confirm the scope. Is CPU high cluster-wide or on specific nodes? Asymmetric CPU points to hot-spotting; uniform saturation points to workload limits or undersized hardware.
2. Distinguish os.cpu.percent from process.cpu.percent. If OS CPU is high but process CPU is moderate, investigate co-located processes or container CPU limits.
3. Run GET /_nodes/hot_threads. Look for dominant class prefixes: org.elasticsearch.search for query load, org.elasticsearch.ingest.Pipeline or org.elasticsearch.grok for ingest cost, GlobalOrdinalsBuilder for global ordinals buildup, and merge-related frames for Lucene merge overhead. GC threads also appear here when GC is consuming cores.
4. Correlate with thread pool state. If search or write queues are growing and rejections are rising, CPU saturation is already causing pushback.
5. Check JVM GC metrics. Rising old GC frequency or duration means CPU is being consumed by garbage collection, not application work. Look for log entries indicating overhead spent collecting in the last second.
6. Inspect segment count and merge activity. If merges.current is at the scheduler limit and segment count is climbing, merges are competing for CPU and I/O.
7. Check the master node separately. If the master is above 80% CPU, pending cluster tasks will grow even if data nodes look healthy.
8. Check thread pool configuration. The search pool default size is computed as int((# allocated processors * 3) / 2) + 1 (13 threads on an 8-core node) and has no hard cap. The write pool defaults to # allocated processors threads (8 on an 8-core node) with a documented hard maximum of 1 + # allocated processors for that pool. Both sizes are static: changing thread_pool.search.size or thread_pool.write.size requires a node restart.

Metrics and signals to monitor

Fixes

Reduce search CPU load

Cancel expensive queries via GET /_tasks?detailed=true&actions=*search*, then POST /_tasks/{task_id}/_cancel. For recurring patterns, optimize aggregations: avoid high-cardinality terms on text fields, replace regex or leading wildcards with keyword prefix queries, and add keyword sub-fields for sorting and aggregations. Reduce shard count per query to lower fan-out overhead on the coordinating node.

Throttle or reschedule merges

If merges dominate CPU, reduce index.merge.scheduler.max_thread_count to 1 on spinning disks; the default assumes SSDs. For indices no longer receiving writes, force-merge to 1 segment during a maintenance window with POST /<index>/_forcemerge?max_num_segments=1. Never force-merge live indices receiving writes. Temporarily increase refresh_interval on write-heavy indices to reduce segment creation rate.

Address GC-driven CPU

GC consuming CPU is a symptom, not a root cause. If old GC is frequent and the post-GC heap floor is rising, reduce long-lived heap pressure. Common fixes: lower shard count per node (segment metadata lives in heap), eliminate fielddata usage on text fields by using keyword doc_values, reduce aggregation cardinality, and ensure heap stays at or below 31GB to remain within compressed OOPs. Do not simply add heap above 31GB; this worsens hot-spotting and GC efficiency.

Fix hot-spotting

If one node carries most CPU while others idle, check shard distribution with GET /_cat/allocation. Since Elasticsearch 8.6, desired balancing considers ingest load, not just shard count, so _cat/allocation alone may not reflect write imbalance. Use GET /_cat/thread_pool/write,search to detect queue buildup on specific nodes. Manually reroute hot shards or add nodes. Ensure hardware is uniform; mixing instance sizes guarantees imbalance.

Relieve master CPU pressure

Isolate master-eligible nodes from data and ingest workloads. If master CPU is elevated, investigate cluster state size and pending tasks. Reduce cluster state churn: consolidate time-series indices with ILM rollover instead of per-minute indices, cap total field counts with index.mapping.total_fields.limit, and pause rapid index creation during incidents.

Prevention

Monitor process.cpu.percent per node, not cluster-wide averages. Alert on asymmetric CPU: any data node above 80% while the cluster median stays below 50%. Keep master nodes dedicated and monitor their CPU independently. Size thread pools consciously: the search pool defaults to int((# allocated processors * 3) / 2) + 1 threads and the write pool to # allocated processors threads (maximum 1 + # allocated processors); changes require a node restart. Maintain segment health with ILM force-merge on read-only indices, and avoid aggressive refresh_interval during bulk loads. Keep heap at or below 31GB and scale horizontally rather than vertically.

How Netdata helps

• Correlate elasticsearch.process.cpu.percent with system.cpu.utilization per node to spot co-located process interference or container CPU limits.
• Track per-node thread pool queue depth and rejection rates alongside CPU to distinguish saturation from external load.
• Monitor JVM heap usage and GC collection time alongside CPU to identify GC thrashing quickly.
• Alert when individual data nodes diverge from the cluster CPU median.
• Monitor master-node CPU separately from data nodes to catch master lag before cluster state tasks back up.

Related guides

• Elasticsearch all shards failed: diagnosing search_phase_execution_exception
• Elasticsearch authentication failures: audit logs, brute force, and credential drift
• Elasticsearch CircuitBreakingException: [parent] Data too large - causes and fixes
• Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained
• Elasticsearch cluster health red: unassigned primaries and how to recover
• Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks
• Elasticsearch cluster state too large: field count, index count, and per-node heap
• Elasticsearch coordinating node overload: aggregation merge, heap spikes, and 429s
• Elasticsearch disk full: emergency recovery and freeing space safely
• Elasticsearch disk I/O saturation: merges, fsync, and page-cache starvation
• Elasticsearch disk watermark cascade: from low watermark to cluster-wide read-only
• Elasticsearch document indexing failures: index_failed, bulk item errors, and version conflicts