OpenTelemetry

OpenTelemetry

Plugin: otel.plugin Module: otel

Overview

Receive OpenTelemetry metrics, logs, and traces from an OpenTelemetry Collector over OTLP/gRPC or OTLP/HTTP. The plugin converts supported metrics into Netdata charts, indexes logs for exploration in the Logs tab, and indexes traces for exploration in the Traces tab.

The plugin accepts gauges, sums, explicit-bucket histograms, and summaries. It does not currently ingest exponential histograms. Monotonic cumulative and delta sums are rate-normalized; non-monotonic cumulative sums behave as gauges.

For a maintained Collector configuration and end-to-end verification steps, see Ingest OpenTelemetry Metrics, Logs, and Traces. The separate OpenTelemetry Logs integration explains how received logs appear in Netdata, and Trace Storage and Retention explains how received traces are stored and kept.

The plugin starts automatically and listens for OTLP/gRPC on port 4317. It can also listen for OTLP/HTTP on port 4318, but this listener is off by default: set receivers.otlp.protocols.http.enabled to true to turn it on. Point the Collector’s otlp_grpc exporter, or its otlp_http exporter when the OTLP/HTTP listener is on, at the matching port. Keep the exporter’s default protobuf encoding and gzip compression: the plugin rejects other compression algorithms, and with encoding: json it rejects any request that carries a NaN or infinite metric value.

Each metric name becomes a Netdata chart context prefixed with otel.. YAML mapping rules can match a metric name and instrumentation scope, select a data point attribute for dimension names, and override chart timing. User mapping files take priority over the stock hostmetrics mappings compiled into the plugin. Metrics without a matching mapping still create charts using default settings.

Invalid user mapping files are logged and skipped without disabling the stock mappings. Charts expire after their configured inactivity period.

This collector is only supported on the following platforms:

• Linux
• macOS

This collector only supports collecting metrics from a single instance of this integration.

Default Behavior

Auto-Detection

The plugin starts automatically and listens on 127.0.0.1:4317 for OTLP/gRPC connections. When you turn on the OTLP/HTTP listener, it also listens on 127.0.0.1:4318 for OTLP/HTTP connections.

Limits

Each export request is limited to 4 MiB, both as received and after decompression. The plugin rejects a larger request as a whole. Each export request can create at most 100 new charts by default. This is a per-request throttle, not a total cardinality limit. By default, log timestamps are accepted from up to 24 hours in the past through 10 minutes in the future; records without a timestamp use their arrival time. A span is accepted when it started no more than 24 hours ago and ends no more than 10 minutes in the future. Log and trace retention are each bounded by whichever limit is reached first: retained file count, retained indexed-data size, or age.

Performance Impact

Resource use depends on the number and cardinality of received metrics, log and span volume, retention, and query activity. Keep unbounded attributes out of metric identities and dimension names.

Setup

Prerequisites

A Netdata installation that includes the OpenTelemetry plugin

Official Linux packages, static builds, containers, and supported macOS packages include the plugin. Linux source installs using netdata-installer.sh require a compatible Rust toolchain and --enable-plugin-otel; on macOS, the installer enables the plugin automatically when it finds a compatible Rust toolchain.

An OTLP data source

An OpenTelemetry Collector that exports metrics, logs, or traces to the Agent: the otlp_grpc exporter to port 4317, or the otlp_http exporter to port 4318 after you turn on the OTLP/HTTP listener. The default loopback endpoints accept only a Collector on the same host.

Configuration

Options

Edit otel.yaml with edit-config. A user file can contain only the fields that differ from the stock configuration. Configuration resolves in this order: stock file, user file, then environment variables. Restart the Netdata Agent after changing otel.yaml or any NETDATA_OTEL_CFG_* environment variable.

The absolute paths shown below are the defaults for standard Linux packages. The installed stock configuration resolves its configuration and log directories for the installation layout, so macOS and custom-prefix installations use different paths. Use edit-config with otel.yaml to inspect the installed values.

Environment overrides use the NETDATA_OTEL_CFG_ prefix. Names are the uppercase option path with dots replaced by underscores. For example, receivers.otlp.protocols.grpc.tls.cert_file becomes NETDATA_OTEL_CFG_RECEIVERS_OTLP_PROTOCOLS_GRPC_TLS_CERT_FILE, and receivers.otlp.protocols.http.enabled becomes NETDATA_OTEL_CFG_RECEIVERS_OTLP_PROTOCOLS_HTTP_ENABLED. For the default rotation and retention entries, omit the default path segment: logs.retention.default.max_age becomes NETDATA_OTEL_CFG_LOGS_RETENTION_MAX_AGE, and traces.retention.default.max_age becomes NETDATA_OTEL_CFG_TRACES_RETENTION_MAX_AGE.

Configuration is strict. Unknown YAML fields, unknown NETDATA_OTEL_CFG_* variables, malformed values, and the former experimental schema prevent the plugin from starting. The effective configuration is logged at startup; remote_storage.uri is redacted in that log.

In a YAML file, an option set to null keeps its current value. To turn a listener on or off, set its enabled option. The OTLP/gRPC listener is on and the OTLP/HTTP listener is off by default. At least one of the two listeners must stay enabled.

Older files configure the OTLP/gRPC listener in an endpoint: section: endpoint.path, endpoint.tls_cert_path, endpoint.tls_key_path, and endpoint.tls_ca_cert_path, and the matching NETDATA_OTEL_CFG_ENDPOINT_* variables. These names still work and log a deprecation warning. If a file or the environment sets an option under both names, the plugin logs a warning and uses the receivers name.

receivers.otlp.protocols.grpc.endpoint

The default IPv4 loopback address accepts only local senders. To accept remote senders, bind a non-loopback address and protect the listener with TLS or mutual TLS and network access controls.

receivers.otlp.protocols.grpc.tls.client_ca_file

Setting a CA certificate enables mutual TLS and therefore also requires the server certificate and key. Clients must present a certificate signed by this CA.

receivers.otlp.protocols.http.endpoint

The default IPv4 loopback address accepts only local senders. To accept remote senders, bind a non-loopback address and protect the listener with TLS or mutual TLS and network access controls.

receivers.otlp.protocols.http.tls.client_ca_file

Setting a CA certificate enables mutual TLS and therefore also requires the server certificate and key. Clients must present a certificate signed by this CA.

metrics.chart_configs_dir

User mappings take priority over the stock mappings compiled into the plugin. Files are processed in filename order. An invalid file is logged and skipped.

metrics.interval_secs

Timing must satisfy 0 < interval <= 3600, interval < grace, and grace <= expiry. Because the stock configuration explicitly sets all three values, set compatible values together when changing the global timing. An invalid resolved timing configuration logs a warning and falls back to the plugin’s hardcoded chart defaults.

metrics.max_new_charts_per_request

This limits a single request’s cardinality burst. It does not cap the total number of charts that successive requests can create.

base_dir

Retention limits govern retained indexed files. Write-ahead logs, catalogs, and the download cache for offloaded data (in the remote-read subdirectory under your configured base directory) can make total usage below this directory exceed the retention.*.max_total_size limits.

remote_storage.enabled

Sealed indexed files are uploaded in addition to being kept under local retention. When a query needs a remotely cataloged file that is no longer local, the plugin downloads it into the bounded download cache shared by logs and traces. Each download may take 30 seconds plus one second per MiB, at most 5 minutes; a file that takes longer or cannot be served is left out (traces report the answer as partial). A query needing more data than the cache holds fails with a message to narrow its time range.

remote_storage.uri

Put non-secret backend options in the query string. Never put credentials in this file or URI. For S3, use the standard AWS environment, credentials file, or instance-role mechanisms available to the Netdata service account.

auth.enabled

This is tenant selection, not credential authentication. When disabled, all logs and traces belong to the default tenant. When enabled, trust the header only behind TLS or mutual TLS and suitable network controls. Metrics are not tenant-scoped.

logs.retention.default.max_total_size

This is not a cap on all plugin disk use. Write-ahead logs, catalogs, and cache are additional.

traces.retention.default.max_total_size

This is not a cap on all plugin disk use. Write-ahead logs, catalogs, and cache are additional.

via File

The configuration file name for this integration is otel.yaml.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config otel.yaml

Examples

Accept remote senders with mutual TLS

Bind beyond loopback only when the endpoint is protected. This example requires each sender to present a certificate signed by the configured CA. Each listener has its own TLS settings. If no sender uses one of the listeners, set its enabled option to false instead.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
        tls:
          cert_file: /etc/netdata/ssl/server-cert.pem
          key_file: /etc/netdata/ssl/server-key.pem
          client_ca_file: /etc/netdata/ssl/client-ca.pem
      http:
        enabled: true
        endpoint: "0.0.0.0:4318"
        tls:
          cert_file: /etc/netdata/ssl/server-cert.pem
          key_file: /etc/netdata/ssl/server-key.pem
          client_ca_file: /etc/netdata/ssl/client-ca.pem

Configure log and trace retention

A user file needs only the fields that change; omitted fields keep their stock values. Logs and traces have separate settings, each tenant entry inherits omitted fields from default, and a per-tenant key must match the sender’s X-Scope-OrgID value when auth.enabled is true.

logs:
  retention:
    default:
      max_total_size: "10GB"
      max_age: "30 days"
traces:
  retention:
    default:
      max_total_size: "10GB"
      max_age: "30 days"

Metric mapping file

Without a mapping, all data point attributes contribute to chart identity and the dimension is named value. A mapping selects one string attribute as the dimension name and removes it from chart identity, grouping data points that differ only by that attribute into one multi-dimension chart. Place user mapping files in metrics.chart_configs_dir; omit instrumentation_scope when the metric name alone is unambiguous.

metrics:
  "system.cpu.time":
    - instrumentation_scope:
        name: .*hostmetricsreceiver.*cpuscraper$
      dimension_attribute_key: state
      interval_secs: 5
  "system.network.io":
    - instrumentation_scope:
        name: .*hostmetricsreceiver.*networkscraper$
      dimension_attribute_key: direction
  "redis.cpu.time":
    - dimension_attribute_key: state

Troubleshooting

Known Errors

failed to bind OTLP/gRPC endpoint 127.0.0.1:4317: Address already in use (os error 98)

When

Logged in the Agent journal at startup by the otel-plugin/ingestor worker, followed by exited with error code 1 and haven't collected any data. Disabling it. from the Agent. A conflict on the OTLP/HTTP listener fails the same strict way: failed to bind OTLP/HTTP endpoint 127.0.0.1:4318: Address already in use (os error 98).

Cause

Another process already listens on the configured address. Typical sources are a standalone OpenTelemetry Collector on the same host — its own OTLP receivers default to the same conventional ports, 4317 for gRPC and 4318 for HTTP — or several Netdata Agents running in containers that share the host network namespace (--network host), where only one of them can own a given port. The plugin binds both listeners before it advertises anything, so on a conflict it exits once and the Agent disables it until the next Agent restart.

Fix

Find the owner with ss -ltnp 'sport = :4317' (or sport = :4318 for the OTLP/HTTP listener). Then stop that listener, or give this Agent its own addresses in otel.yaml and point every OTLP sender at the new ports. You can also turn off a listener that no sender uses:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:14317
      http:
        endpoint: 127.0.0.1:14318
        # or keep the OTLP/HTTP listener off:
        # enabled: false

When several Agents share one network namespace and only one needs to receive OTLP, disable the plugin on the others in netdata.conf:

[plugins]
  otel = no

Other Problems

The plugin does not start

Check the Agent journal for otel-plugin configuration errors. User YAML and NETDATA_OTEL_CFG_* variables are validated strictly, so a typo or an option from the former experimental schema stops startup. For a source install, confirm that a compatible Rust toolchain is available; Linux installs using netdata-installer.sh also require --enable-plugin-otel.

The endpoint is reachable but no data appears

A successful TCP connection proves only that something is listening. Check that the Collector’s exporter uses the right port: 4317 for otlp_grpc, 4318 for otlp_http. The otlp_http exporter needs the OTLP/HTTP listener, which is off by default. With the default endpoints, use 127.0.0.1, not localhost, which can resolve to IPv6. Then look for rejected exports in the Collector log, and verify the resulting chart, log entry, or trace.

A metric does not create the expected chart

Exponential histograms are not currently ingested. For other supported metrics, inspect the Agent journal for rejected user mapping files and verify the metric name, instrumentation scope, and dimension_attribute_key. The resulting chart context is otel.<metric-name>.

Large exports are rejected

The plugin rejects an export request larger than 4 MiB as a whole, and the Collector drops that batch. Make the batches smaller. On the exporter, cap them by size with sending_queue: { batch: { sizer: bytes, max_size: 3000000 } }. Alternatively, lower send_batch_max_size on the batch processor; it counts items (spans, data points, log records), not bytes.

Some exported logs are missing

By default, the plugin rejects log records timestamped more than 24 hours in the past or more than 10 minutes in the future. The Collector logs a Partial success response warning for rejected records. Check the clock of the host that produced the logs, how old the backfilled records are, the Collector log, and the Agent journal.

Some exported spans are missing

By default, the plugin rejects a span that started more than 24 hours ago or ends more than 10 minutes in the future; a span without an end time is judged by its start. The Collector logs a Partial success response warning for rejected spans, and the plugin logs a warning in the Agent journal. Check the clock of the instrumented application’s host, how old the backfilled spans are, the Collector log, and the Agent journal.

Logs from the former experimental plugin are not visible

A former-schema otel.yaml stops the current plugin and prints a migration guide. Replace it with a partial configuration based on the current stock file. The legacy logs.journal_dir key is accepted only to locate the former plugin’s read-only journals; it is not part of the current storage layout.