$ guides / elasticsearch / elasticsearch-limit-of-total-fields-exceeded ▌

Operations Guides

Elasticsearch Limit of total fields [1000] in index has been exceeded — mapping explosion

Every write suddenly returns illegal_argument_exception: Limit of total fields [1000] in index [X] has been exceeded. Indexing stops. The temptation is to raise index.mapping.total_fields.limit and move on. Do not. This is a mapping explosion: dynamic mapping creates a new field for every unique key in your documents. The 1000-field limit is a guardrail, not the root cause. Runaway mappings bloat the cluster state, inflate heap on every node, and eventually destabilize the master. Diagnose the source, relieve pressure safely, and fix the data shape so it does not recur.

What this means

Elasticsearch counts every field, object property, multi-field, and field alias toward index.mapping.total_fields.limit, which defaults to 1000. With dynamic mapping enabled, an unrecognized key in a document triggers a mapping update. If the update would exceed the limit, Elasticsearch rejects the entire indexing operation with an illegal_argument_exception.

Every mapping change becomes cluster state that the master serializes and publishes to every node. An index with thousands of fields increases heap pressure across the entire cluster, not only on nodes holding its shards. Unstructured data plus dynamic mapping produces linear metadata growth that compounds into master instability and JVM heap exhaustion.

flowchart TD
    A[Unstructured JSON document] --> B[Dynamic mapping update]
    B --> C{Field count > limit?}
    C -->|Yes| D[illegal_argument_exception]
    C -->|No| E[Mapping grows]
    E --> F[Cluster state expands]
    F --> G[Heap pressure on all nodes]
    G --> H[Master pending tasks backlog]

Common causes

Cause	What it looks like	First thing to check
Variable-key objects ingested as nested properties	Field names look like values: UUIDs, timestamps, Kubernetes labels, or HTTP headers mapped as objects. Mapping size grows with every unique key.	`GET /<index>/_mapping` for objects with high-cardinality sub-field names.
Default dynamic mapping on high-cardinality data streams	`logs--` or similar data streams silently accumulate new fields until ingestion halts on rollover or the current backing index.	Index template settings for `index.mapping.total_fields.limit` and `dynamic`.
Application releases that change log structure	Field count jumps after a deployment, often with new nested error objects or contextual metadata.	Compare mapping generation timestamps with deployment events.
Missing depth limits allowing recursive objects	Deeply nested JSON from APIs or metrics platforms creates fields at multiple levels, multiplying the total count.	`index.mapping.depth.limit` in index settings (default 20).

Quick checks

# Confirm total field count and cluster-wide mapping scale
curl -s 'http://localhost:9200/_cluster/stats?filter_path=indices.mappings' | python3 -m json.tool

# Inspect the affected index mapping
curl -s "http://localhost:9200/<index>/_mapping?pretty"

# Estimate cluster state serialization size
# WARNING: Can be heavy on large clusters; run during low traffic if possible.
curl -s 'http://localhost:9200/_cluster/state' | wc -c

# Check master backlog from mapping updates
curl -s 'http://localhost:9200/_cluster/pending_tasks?pretty'

# View index-level mapping limits and dynamic behavior
curl -s "http://localhost:9200/<index>/_settings?filter_path=*.index.mapping" | python3 -m json.tool

# Assess heap and segment memory pressure per node
curl -s 'http://localhost:9200/_cat/nodes?v&h=name,heap.percent,segments.memory'

How to diagnose it

Confirm the exact exception. Look for illegal_argument_exception with the message Limit of total fields [1000] in index [X] has been exceeded (Elasticsearch 7.x wording; 8.x and newer print Limit of total fields [1000] has been exceeded without the index name) in client logs, ingest dead-letter queues, or Elasticsearch server logs.
Inspect the offending mapping. Run GET /<index>/_mapping. Identify objects where property names are data values rather than schema keys. A labels object containing keys like app.kubernetes.io/name, pod-template-hash, or UUIDs signals runaway dynamic mapping.
Trace the ingestion path. Determine which Beat, Logstash pipeline, or application emits the documents. Kubernetes metadata, APM agent context, and user-generated tags are common sources of unbounded keys.
Determine scope. Query _cluster/stats?filter_path=indices.mappings. A steadily climbing total field count means other indices are likely approaching the limit.
Measure cluster state impact. Check GET /_cluster/pending_tasks and GET /_cat/nodes?v&h=name,heap.percent. Growing pending tasks and elevated heap indicate master pressure from mapping updates.
Review templates. For data streams, inspect the index template. It defines the baseline index.mapping.total_fields.limit and dynamic mapping policy for future backing indices.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Total field count (`_cluster/stats`)	Leading indicator of cluster state bloat.	Steady week-over-week growth without index deletion.
Pending cluster tasks	Every mapping update generates a cluster task. A backlog means the master cannot keep up.	>20 tasks or any task older than 30 seconds.
JVM heap used percent	Field definitions live in cluster state and segment metadata on every node.	Sustained >75% or a rising post-GC floor.
Indexing errors (`index_failed`)	Documents rejected at the indexing layer, including mapping limit breaches.	`index_failed` delta increasing while indexing rate is flat.
Write thread pool rejections	Mapping backpressure eventually stalls the write path.	Sustained rejections >0 per minute with active ingest.
Segment memory per node	Large mappings increase per-segment metadata overhead.	`segments.memory` growing faster than stored data.

Fixes

Immediate relief: raise the limit

To restore writes while you fix the source, increase the limit on the affected index:

# WARNING: Bandage only. Do not leave elevated without fixing data shape.
curl -X PUT "http://localhost:9200/<index>/_settings" -H 'Content-Type: application/json' -d'
{
  "index.mapping.total_fields.limit": 2000
}'

Tradeoff: A higher limit delays the error but increases cluster state size and heap consumption. Use this only to buy time.

Fix the data shape

The durable fix is to stop sending unbounded keys.

Normalize variable-key objects. Convert { "labels": { "app": "x", "env": "y" } } into an array like { "labels": [ { "key": "app", "value": "x" } ] } or stringify the map into a single keyword field. Either keeps the mapped field count constant regardless of input variability.
Drop or flatten nested noise. Use an ingest pipeline or Logstash filter to remove high-cardinality objects whose keys are IDs, timestamps, or hashes before they reach Elasticsearch.
Enforce depth limits. If deep nesting multiplies field counts, lower index.mapping.depth.limit to prevent excessive recursion from creating new levels.

Restrict dynamic mapping

If the fields are not needed for search or aggregation, stop indexing them:

# New fields are ignored; they remain in _source but are not mapped
curl -X PUT "http://localhost:9200/<index>/_mapping" -H 'Content-Type: application/json' -d'
{
  "dynamic": "false"
}'

For stricter control, use dynamic: strict to reject unknown fields immediately, forcing explicit mappings.

Apply via index template

For managed indices and data streams, set the policy in the composable index template so new backing indices inherit the fix:

curl -X PUT "http://localhost:9200/_index_template/<template>" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.mapping.total_fields.limit": 1500
    },
    "mappings": {
      "dynamic": "strict"
    }
  }
}'

Composable index templates (_index_template) supersede the legacy _template API.

Prevention

Use explicit mappings. Define indices with dynamic: strict so unexpected fields cause visible client errors rather than silent mapping growth.
Normalize at the edge. Configure Beats, Logstash, or ingest pipelines to flatten or stringify objects with variable keys before they reach the cluster.
Monitor field count trend. Sample indices.mappings from _cluster/stats regularly. A steady climb is an early warning.
Set limits intentionally. If a legitimate use case needs more than 1000 fields, raise index.mapping.total_fields.limit in the index template after validating that master nodes and heap can support the larger cluster state.
Remember rollover does not fix the source. A new backing index starts with zero fields, but if the emitted documents still carry random keys, the limit will be breached again. Fix the emitter.

How Netdata helps

JVM heap: Correlate per-node heap utilization with cluster state changes. Rising heap across the cluster, especially without a matching rise in document volume, indicates mapping bloat.
Pending tasks: A growing backlog on the master often signals mapping updates are overwhelming coordination.
Indexing failures: A spike in index_failed reveals the first mapping rejections before the index is fully saturated.
Write rejections: Thread pool rejections alongside steady ingest traffic isolate mapping backpressure from generic overload.
OS metrics: Disk and CPU metrics rule out I/O saturation when indexing latency rises.

The Netdata solution

Elasticsearch monitoring with Netdata

Netdata monitors Elasticsearch with per-second metrics and ML anomaly detection. Correlate JVM heap pressure, shard counts, disk watermarks, mapping growth, and merge activity with cluster and node health in one view.

See Elasticsearch monitoring → Start monitoring free