Nvidia Data Center GPU Manager (DCGM)

Nvidia Data Center GPU Manager (DCGM) documentation

Nvidia Data Center GPU Manager (DCGM)

Plugin: go.d.plugin Module: dcgm

Overview

This collector gathers NVIDIA GPU telemetry from a dcgm-exporter endpoint. Known fields use source-defined units and chart groups. Other numeric fields remain available as raw measurements.

It collects metrics by periodically scraping the exporter Prometheus endpoint over HTTP.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration.

Default Behavior

Auto-Detection

This integration requires a configured dcgm-exporter endpoint.

Limits

The collector applies global and per-metric time series limits to prevent excessive cardinality. Some exporter versions round floating-point bit error ratios to six decimal places. Enable the packed *_BER_RAW (or legacy COUNT_*_BER) field when available to preserve small ratios; float-only values retain the exporter’s precision limit.

Performance Impact

The impact depends on dcgm-exporter field selection and resulting series cardinality.

Setup

You can configure the dcgm collector in two ways:

Method	Best for	How to
UI	Fast setup without editing files	Go to Nodes → Configure this node → Collectors → Jobs, search for dcgm, then click + to add a job.
File	If you prefer configuring via file, or need to automate deployments (e.g., with Ansible)	Edit `go.d/dcgm.conf` and add a job.

Important
UI configuration requires paid Netdata Cloud plan.

Prerequisites

Run dcgm-exporter

Install DCGM and run dcgm-exporter so that a Prometheus endpoint is available (default :9400/metrics).

Configure exporter field list

The default exporter profile exposes a small subset of fields. Use the Netdata recommended profile: dcgm-exporter-netdata.csv (raw download: https://raw.githubusercontent.com/netdata/netdata/master/src/go/plugin/go.d/collector/dcgm/dcgm-exporter-netdata.csv).

The Netdata profile enables 121 fields by default and lists optional fields from its source dataset as commented entries. To customize beyond the baseline, uncomment the field you need and comment one currently enabled field.

Runtime validation artifact: src/go/plugin/go.d/collector/dcgm/runtime-validation-driver-590.48.01-dcgm-exporter-4.4.1-4.5.2.md and src/go/plugin/go.d/collector/dcgm/runtime-validation-driver-590.48.01-dcgm-exporter-4.4.1-4.5.2.json

Validation is primarily version-scoped (NVIDIA driver + DCGM/DCGM-exporter versions), so treat it as a strong baseline rather than universal compatibility.

Example: dcgm-exporter -f /path/to/dcgm-exporter-netdata.csv

Keep collection intervals aligned

Set Netdata update_every to the same value as dcgm-exporter collection interval (default 30 seconds). Example exporter interval: dcgm-exporter -c 30000 and Netdata update_every: 30.

Enable profiling capabilities (optional)

Profiling fields may require additional privileges/capabilities in your runtime environment.

Configuration

Options

The following options can be defined globally: update_every, autodetection_retry.

Group	Option	Description	Default	Required
Collection	update_every	Data collection interval (seconds). Keep this aligned with dcgm-exporter collection interval.	30	no
	autodetection_retry	Autodetection retry interval (seconds). Set 0 to disable.	0	no
Target	url	DCGM exporter metrics endpoint URL.	http://127.0.0.1:9400/metrics	yes
	timeout	HTTP request timeout (seconds).	10	no
Limits	max_time_series	Global time series limit. If exceeded, collection is skipped for this cycle.	2000	no
	max_time_series_per_metric	Per-metric time series limit. Metrics above this limit are skipped.	200	no
HTTP Auth	username	Username for Basic HTTP authentication.		no
	password	Password for Basic HTTP authentication.		no
	bearer_token_file	Path to a file containing a bearer token.		no
TLS	tls_skip_verify	Skip TLS certificate and hostname verification (insecure).	no	no
	tls_ca	Path to CA bundle used to validate the server certificate.		no
	tls_cert	Path to client TLS certificate (for mTLS).		no
	tls_key	Path to client TLS private key (for mTLS).		no
Proxy	proxy_url	HTTP proxy URL.		no
	proxy_username	Username for proxy authentication.		no
	proxy_password	Password for proxy authentication.		no
Request	headers	Additional HTTP headers to include in the request.		no
	method	HTTP method.	GET	no
	body	HTTP request body.		no
	not_follow_redirects	Do not follow HTTP redirects.	no	no
	force_http2	Force HTTP/2 (including h2c over TCP).	no	no
Virtual Node	vnode	Associate this job with a Virtual Node.		no

via UI

Configure the dcgm collector from the Netdata web interface:

Go to Nodes.
Select the node where you want the dcgm data-collection job to run and click the :gear: (Configure this node). That node will run the data collection.
The Collectors → Jobs view opens by default.
In the Search box, type dcgm (or scroll the list) to locate the dcgm collector.
Click the + next to the dcgm collector to add a new job.
Fill in the job fields, then click Test to verify the configuration and Submit to save.
- Test validates the provided settings and checks the collector’s startup prerequisites. Successful validation does not guarantee that every metric will be available during collection.
- If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/dcgm.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/dcgm.conf

Examples

Local exporter

Collect metrics from a local dcgm-exporter endpoint.

jobs:
  - name: local
    url: http://127.0.0.1:9400/metrics
    update_every: 30

TLS endpoint

Collect metrics over HTTPS with custom CA certificate.

jobs:
  - name: secure
    url: https://dcgm-exporter.example.com:9400/metrics
    update_every: 30
    tls_ca: /etc/netdata/certs/dcgm-ca.crt

Increased cardinality limits

Increase limits when collecting large field sets and multiple entities.

jobs:
  - name: dcgm_large
    url: http://127.0.0.1:9400/metrics
    update_every: 30
    max_time_series: 10000
    max_time_series_per_metric: 2000

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Charts appear only for fields supplied by the exporter; availability depends on hardware, driver and exporter versions. Missing, unsupported and invalid samples leave gaps. Labels identifying health watches, event reasons and peers distinguish dimensions; event window lengths distinguish chart instances.

See metric semantics and chart migration for interpretation, source selection and changes to existing charts.

Per host

The GPU inventory reported by one exporter host.

This scope has no labels.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.host.inventory.gpu_count	Host GPU Count	gpus	GPUs

Per gpu

A physical GPU identified by the exporter device and UUID labels.

Labels:

Label	Description
gpu	GPU device index.
uuid	GPU UUID.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.gpu.compute.utilization	GPU and Memory Busy Time	gpu, memory_copy	percentage
dcgm.gpu.compute.engine_activity	GPU Graphics and Compute Engine Activity	activity	percentage
dcgm.gpu.compute.sm.utilization	GPU Streaming Multiprocessor Utilization	activity, occupancy	percentage
dcgm.gpu.compute.resource_activity	GPU Tensor and Memory Interface Activity	dram, tensor	percentage
dcgm.gpu.compute.pipe.activity	GPU Arithmetic Pipeline Activity	fp16, fp32, fp64, integer	percentage
dcgm.gpu.compute.tensor.activity	GPU Tensor Instruction Activity	tensor_dfma, tensor_hmma, tensor_imma	percentage
dcgm.gpu.compute.media.utilization	GPU Video Engine Utilization	decoder, encoder	percentage
dcgm.gpu.compute.media.activity	GPU Media Engine Activity	nvdec0, nvdec1, nvdec2, nvdec3, nvdec4, nvdec5, nvdec6, nvdec7, nvjpg0, nvjpg1, nvjpg2, nvjpg3, nvjpg4, nvjpg5, nvjpg6, nvjpg7, nvofa0, nvofa1	percentage
dcgm.gpu.compute.cache.host	GPU Host Memory Cache Efficiency	hit, miss	percentage
dcgm.gpu.compute.cache.peer	GPU Peer Memory Cache Efficiency	hit, miss	percentage
dcgm.gpu.memory.utilization	GPU VRAM Capacity Used	used_percent	percentage
dcgm.gpu.memory.usage	GPU VRAM Allocation	free, reserved, used	bytes
dcgm.gpu.memory.capacity	GPU VRAM Capacity	total	bytes
dcgm.gpu.memory.bar1_usage	GPU BAR1 Mapping Usage	free, used	bytes
dcgm.gpu.memory.bar1_capacity	GPU BAR1 Mapping Capacity	total	bytes
dcgm.gpu.memory.ecc_mode	GPU ECC Mode	current, pending	state
dcgm.gpu.memory.ecc_volatile	GPU Volatile ECC Errors	dbe, sbe	errors
dcgm.gpu.memory.ecc_aggregate	GPU Persistent ECC Errors	dbe, sbe	errors
dcgm.gpu.memory.ecc_volatile_rate	GPU Volatile ECC Error Rate	dbe, sbe	errors/s
dcgm.gpu.memory.ecc_aggregate_rate	GPU Persistent ECC Error Rate	dbe, sbe	errors/s
dcgm.gpu.memory.ecc_volatile_detail	GPU Volatile ECC Errors by Location	dbe_cbu, dbe_dev, dbe_l1, dbe_l2, dbe_reg, dbe_shm, dbe_srm, dbe_tex, sbe_cbu, sbe_dev, sbe_l1, sbe_l2, sbe_reg, sbe_shm, sbe_srm, sbe_tex	errors
dcgm.gpu.memory.ecc_aggregate_detail	GPU Persistent ECC Errors by Location	dbe_cbu, dbe_dev, dbe_l1, dbe_l2, dbe_reg, dbe_shm, dbe_srm, dbe_tex, sbe_cbu, sbe_dev, sbe_l1, sbe_l2, sbe_reg, sbe_shm, sbe_srm, sbe_tex	errors
dcgm.gpu.memory.page_retirements	GPU Memory Page Retirement Rate	dbe, sbe	pages/s
dcgm.gpu.memory.retired_pages	GPU Retired Memory Pages	dbe, sbe	pages
dcgm.gpu.reliability.page_retirement_status	GPU Memory Page Retirement Pending	pending	state
dcgm.gpu.reliability.row_remap_status	GPU Row Remap Status	row_remap_failure, row_remap_pending	state
dcgm.gpu.reliability.row_remap_events	GPU Row Remap Rate	correctable_remapped_rows, uncorrectable_remapped_rows	rows/s
dcgm.gpu.reliability.remapped_rows	GPU Remapped Memory Rows	correctable_remapped_rows, uncorrectable_remapped_rows	rows
dcgm.gpu.reliability.remap_banks	GPU Memory Bank Remap Availability	high, low, max, none, partial	banks
dcgm.gpu.reliability.memory_health	GPU Memory Health Flags	sram_threshold_exceeded, unrepairable	state
dcgm.gpu.reliability.recovery_action	GPU Recovery Action	action	state
dcgm.gpu.clock.sm.frequency	GPU SM Clock Frequency	application, current, maximum	MHz
dcgm.gpu.clock.memory.frequency	GPU Memory Clock Frequency	application, current, maximum	MHz
dcgm.gpu.clock.video.frequency	GPU Video Clock Frequency	current, maximum	MHz
dcgm.gpu.throttle.reasons	GPU Clock Event Reasons	application_clocks, display_clocks, gpu_idle, hardware_power_brake, hardware_slowdown, hardware_thermal, power_cap, software_thermal, sync_boost, unknown	state
dcgm.gpu.throttle.duration	GPU Clock Limiting Duration	board_limit, hw_power_brake_slowdown, hw_therm_slowdown, low_utilization, power_violation, reliability, sw_therm_slowdown, sync_boost, thermal_violation	percentage
dcgm.gpu.throttle.event_samples	GPU Clock Event Samples	value_clock_event=*	samples
dcgm.gpu.throttle.event_rate	GPU Clock Event Rate	value_clock_event=*	events/s
dcgm.gpu.thermal.temperature	GPU Temperature and Limits	gpu, gpu_max_operating, gpu_shutdown, gpu_slowdown, memory, memory_max_operating	Celsius
dcgm.gpu.thermal.headroom	GPU Thermal Headroom	gpu	Celsius
dcgm.gpu.thermal.fan_speed	GPU Reported Fan Speed	fan_speed	percentage
dcgm.gpu.power.usage	GPU Power Draw and Limit	configured_limit, draw, enforced_limit, instantaneous	Watts
dcgm.gpu.power.limits	GPU Configurable Power Limits	default, maximum, minimum	Watts
dcgm.gpu.power.energy_rate	GPU Energy-Derived Power	power	Watts
dcgm.gpu.power.smoothing.bounds	GPU Power Smoothing Bounds	applied_tmp_ceil, applied_tmp_floor	Watts
dcgm.gpu.power.smoothing.floor	GPU Power Smoothing Floor	admin_override_percent_tmp_floor, max_percent_tmp_floor_setting, min_percent_tmp_floor_setting, profile_percent_tmp_floor	percentage
dcgm.gpu.power.smoothing.lifetime	GPU Power Smoothing Circuit Lifetime Remaining	hw_circuitry_percent_lifetime_remaining	percentage
dcgm.gpu.power.smoothing.ramp	GPU Power Smoothing Ramp Rate	admin_override_ramp_down_rate, admin_override_ramp_up_rate, profile_ramp_down_rate, profile_ramp_up_rate	Watts/s
dcgm.gpu.power.smoothing.hysteresis	GPU Power Smoothing Ramp Hysteresis	admin_override_ramp_down_hyst_val, profile_ramp_down_hyst_val	milliseconds
dcgm.gpu.power.smoothing.state	GPU Power Smoothing Enablement	enabled, imm_ramp_down_enabled	state
dcgm.gpu.power.smoothing.privilege	GPU Power Smoothing Privilege Level	priv_lvl	level
dcgm.gpu.power.smoothing.profile	GPU Active Power Smoothing Profile	active_preset_profile	profile
dcgm.gpu.power.smoothing.profile_capacity	GPU Power Smoothing Profile Capacity	max_num_preset_profiles	profiles
dcgm.gpu.interconnect.total.throughput	GPU Observed Interconnect Throughput	nvlink, pcie	bytes/s
dcgm.gpu.interconnect.pcie.throughput	GPU PCIe Throughput	pcie_rx, pcie_tx	bytes/s
dcgm.gpu.interconnect.nvlink.throughput	GPU NVLink Throughput	nvlink_rx, nvlink_tx	bytes/s
dcgm.gpu.interconnect.pcie.link.generation	GPU PCIe Link Generation	link_gen, max_link_gen	generation
dcgm.gpu.interconnect.pcie.link.width	GPU PCIe Link Width	link_width, max_link_width	lanes
dcgm.gpu.interconnect.pcie.error_rate	GPU PCIe Error and Replay Rate	pcie_replay	errors/s
dcgm.gpu.interconnect.nvlink.error_rate	GPU NVLink Error Rate	nvlink_crc_data_error, nvlink_crc_flit_error, nvlink_ecc_data_error, nvlink_error_dl_crc, nvlink_error_dl_recovery, nvlink_error_dl_replay, nvlink_recovery_error, nvlink_replay_error	errors/s
dcgm.gpu.interconnect.nvlink.traffic	GPU NVLink Packet Rate	rx, tx	packets/s
dcgm.gpu.interconnect.nvlink.ber	GPU NVLink Bit Error Ratio	effective, symbol	errors per trillion bits
dcgm.gpu.interconnect.fabric	GPU Fabric Manager State	failure, in_progress, not_started, not_supported, nvml_too_old, success, unknown, unrecognized	state
dcgm.gpu.interconnect.fabric_error	GPU Fabric Manager Error	error	code
dcgm.gpu.interconnect.p2p	GPU Peer Access State	value_peer_gpu=*	state
dcgm.gpu.interconnect.c2c.capacity	GPU C2C Link Capacity	maximum	bytes/s
dcgm.gpu.interconnect.c2c.links	GPU C2C Link Count	links	links
dcgm.gpu.interconnect.c2c.state	GPU C2C Link State	power_state, status	state
dcgm.gpu.interconnect.c2c.throughput	GPU C2C All Traffic	rx, tx	bytes/s
dcgm.gpu.interconnect.c2c.payload	GPU C2C Payload Traffic	rx, tx	bytes/s
dcgm.gpu.interconnect.c2c.error_rate	GPU C2C Error Rate	intr, replay, replay_b2b	errors/s
dcgm.gpu.reliability.xid	GPU XID Error Code	xid	code
dcgm.gpu.reliability.xid_samples	GPU XID Samples	value_xid=*	samples
dcgm.gpu.reliability.xid_rate	GPU XID Record Rate	value_xid=*	records/s
dcgm.gpu.health.status	GPU Health Status by Watch	value_health_watch=*	state
dcgm.gpu.state.performance	GPU Performance State	P0, P1, P10, P11, P12, P13, P14, P15, P2, P3, P4, P5, P6, P7, P8, P9, unknown	state
dcgm.gpu.state.virtualization	GPU Virtualization Mode	mig_enabled, virtualization_bare_metal, virtualization_host_vgpu, virtualization_host_vsga, virtualization_passthrough, virtualization_unknown, virtualization_vgpu	state
dcgm.gpu.state.configuration	GPU Compute and Persistence Modes	autoboost_enabled, compute_default, compute_exclusive_process, compute_exclusive_thread, compute_prohibited, compute_unknown, persistence_enabled	state
dcgm.gpu.virtualization.vgpu.license	GPU vGPU License State	licensed	state
dcgm.gpu.virtualization.vgpu.memory	GPU vGPU Memory Usage	used	bytes
dcgm.gpu.virtualization.vgpu.frame_rate	GPU vGPU Frame Rate Limit	limit	frames/s
dcgm.gpu.capability.mig_slices	GPU Maximum MIG Slices	maximum	slices

Per mig

A GPU instance exposed when MIG monitoring is available.

Labels:

Label	Description
gpu	GPU device index.
uuid	GPU UUID.
gpu_i_id	MIG GPU instance identifier.
gpu_i_profile	MIG profile.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.mig.compute.utilization	MIG GPU and Memory Busy Time	gpu, memory_copy	percentage
dcgm.mig.compute.engine_activity	MIG Graphics and Compute Engine Activity	activity	percentage
dcgm.mig.compute.sm.utilization	MIG Streaming Multiprocessor Utilization	activity, occupancy	percentage
dcgm.mig.compute.resource_activity	MIG Tensor and Memory Interface Activity	dram, tensor	percentage
dcgm.mig.compute.pipe.activity	MIG Arithmetic Pipeline Activity	fp16, fp32, fp64, integer	percentage
dcgm.mig.compute.tensor.activity	MIG Tensor Instruction Activity	tensor_dfma, tensor_hmma, tensor_imma	percentage
dcgm.mig.compute.media.utilization	MIG Video Engine Utilization	decoder, encoder	percentage
dcgm.mig.compute.media.activity	MIG Media Engine Activity	nvdec0, nvdec1, nvdec2, nvdec3, nvdec4, nvdec5, nvdec6, nvdec7, nvjpg0, nvjpg1, nvjpg2, nvjpg3, nvjpg4, nvjpg5, nvjpg6, nvjpg7, nvofa0, nvofa1	percentage
dcgm.mig.compute.cache.host	MIG Host Memory Cache Efficiency	hit, miss	percentage
dcgm.mig.compute.cache.peer	MIG Peer Memory Cache Efficiency	hit, miss	percentage
dcgm.mig.memory.utilization	MIG VRAM Capacity Used	used_percent	percentage
dcgm.mig.memory.usage	MIG VRAM Allocation	free, reserved, used	bytes
dcgm.mig.memory.capacity	MIG VRAM Capacity	total	bytes
dcgm.mig.memory.bar1_usage	MIG BAR1 Mapping Usage	free, used	bytes
dcgm.mig.memory.bar1_capacity	MIG BAR1 Mapping Capacity	total	bytes
dcgm.mig.memory.ecc_mode	MIG ECC Mode	current, pending	state
dcgm.mig.memory.ecc_volatile	MIG Volatile ECC Errors	dbe, sbe	errors
dcgm.mig.memory.ecc_aggregate	MIG Persistent ECC Errors	dbe, sbe	errors
dcgm.mig.memory.ecc_volatile_rate	MIG Volatile ECC Error Rate	dbe, sbe	errors/s
dcgm.mig.memory.ecc_aggregate_rate	MIG Persistent ECC Error Rate	dbe, sbe	errors/s
dcgm.mig.memory.ecc_volatile_detail	MIG Volatile ECC Errors by Location	dbe_cbu, dbe_dev, dbe_l1, dbe_l2, dbe_reg, dbe_shm, dbe_srm, dbe_tex, sbe_cbu, sbe_dev, sbe_l1, sbe_l2, sbe_reg, sbe_shm, sbe_srm, sbe_tex	errors
dcgm.mig.memory.ecc_aggregate_detail	MIG Persistent ECC Errors by Location	dbe_cbu, dbe_dev, dbe_l1, dbe_l2, dbe_reg, dbe_shm, dbe_srm, dbe_tex, sbe_cbu, sbe_dev, sbe_l1, sbe_l2, sbe_reg, sbe_shm, sbe_srm, sbe_tex	errors
dcgm.mig.memory.page_retirements	MIG Memory Page Retirement Rate	dbe, sbe	pages/s
dcgm.mig.memory.retired_pages	MIG Retired Memory Pages	dbe, sbe	pages
dcgm.mig.reliability.page_retirement_status	MIG Memory Page Retirement Pending	pending	state
dcgm.mig.reliability.row_remap_status	MIG Row Remap Status	row_remap_failure, row_remap_pending	state
dcgm.mig.reliability.row_remap_events	MIG Row Remap Rate	correctable_remapped_rows, uncorrectable_remapped_rows	rows/s
dcgm.mig.reliability.remapped_rows	MIG Remapped Memory Rows	correctable_remapped_rows, uncorrectable_remapped_rows	rows
dcgm.mig.reliability.remap_banks	MIG Memory Bank Remap Availability	high, low, max, none, partial	banks
dcgm.mig.reliability.memory_health	MIG Memory Health Flags	sram_threshold_exceeded, unrepairable	state
dcgm.mig.reliability.recovery_action	MIG Recovery Action	action	state
dcgm.mig.clock.sm.frequency	MIG SM Clock Frequency	application, current, maximum	MHz
dcgm.mig.clock.memory.frequency	MIG Memory Clock Frequency	application, current, maximum	MHz
dcgm.mig.clock.video.frequency	MIG Video Clock Frequency	current, maximum	MHz
dcgm.mig.throttle.reasons	MIG Clock Event Reasons	application_clocks, display_clocks, gpu_idle, hardware_power_brake, hardware_slowdown, hardware_thermal, power_cap, software_thermal, sync_boost, unknown	state
dcgm.mig.throttle.duration	MIG Clock Limiting Duration	board_limit, hw_power_brake_slowdown, hw_therm_slowdown, low_utilization, power_violation, reliability, sw_therm_slowdown, sync_boost, thermal_violation	percentage
dcgm.mig.throttle.event_samples	MIG Clock Event Samples	value_clock_event=*	samples
dcgm.mig.throttle.event_rate	MIG Clock Event Rate	value_clock_event=*	events/s
dcgm.mig.thermal.temperature	MIG Temperature and Limits	gpu, gpu_max_operating, gpu_shutdown, gpu_slowdown, memory, memory_max_operating	Celsius
dcgm.mig.thermal.headroom	MIG Thermal Headroom	gpu	Celsius
dcgm.mig.thermal.fan_speed	MIG Reported Fan Speed	fan_speed	percentage
dcgm.mig.power.usage	MIG Power Draw and Limit	configured_limit, draw, enforced_limit, instantaneous	Watts
dcgm.mig.power.limits	MIG Configurable Power Limits	default, maximum, minimum	Watts
dcgm.mig.power.energy_rate	MIG Energy-Derived Power	power	Watts
dcgm.mig.power.smoothing.bounds	MIG Power Smoothing Bounds	applied_tmp_ceil, applied_tmp_floor	Watts
dcgm.mig.power.smoothing.floor	MIG Power Smoothing Floor	admin_override_percent_tmp_floor, max_percent_tmp_floor_setting, min_percent_tmp_floor_setting, profile_percent_tmp_floor	percentage
dcgm.mig.power.smoothing.lifetime	MIG Power Smoothing Circuit Lifetime Remaining	hw_circuitry_percent_lifetime_remaining	percentage
dcgm.mig.power.smoothing.ramp	MIG Power Smoothing Ramp Rate	admin_override_ramp_down_rate, admin_override_ramp_up_rate, profile_ramp_down_rate, profile_ramp_up_rate	Watts/s
dcgm.mig.power.smoothing.hysteresis	MIG Power Smoothing Ramp Hysteresis	admin_override_ramp_down_hyst_val, profile_ramp_down_hyst_val	milliseconds
dcgm.mig.power.smoothing.state	MIG Power Smoothing Enablement	enabled, imm_ramp_down_enabled	state
dcgm.mig.power.smoothing.privilege	MIG Power Smoothing Privilege Level	priv_lvl	level
dcgm.mig.power.smoothing.profile	MIG Active Power Smoothing Profile	active_preset_profile	profile
dcgm.mig.power.smoothing.profile_capacity	MIG Power Smoothing Profile Capacity	max_num_preset_profiles	profiles
dcgm.mig.interconnect.total.throughput	MIG Observed Interconnect Throughput	nvlink, pcie	bytes/s
dcgm.mig.interconnect.pcie.throughput	MIG PCIe Throughput	pcie_rx, pcie_tx	bytes/s
dcgm.mig.interconnect.nvlink.throughput	MIG NVLink Throughput	nvlink_rx, nvlink_tx	bytes/s
dcgm.mig.interconnect.pcie.link.generation	MIG PCIe Link Generation	link_gen, max_link_gen	generation
dcgm.mig.interconnect.pcie.link.width	MIG PCIe Link Width	link_width, max_link_width	lanes
dcgm.mig.interconnect.pcie.error_rate	MIG PCIe Error and Replay Rate	pcie_replay	errors/s
dcgm.mig.interconnect.nvlink.error_rate	MIG NVLink Error Rate	nvlink_crc_data_error, nvlink_crc_flit_error, nvlink_ecc_data_error, nvlink_error_dl_crc, nvlink_error_dl_recovery, nvlink_error_dl_replay, nvlink_recovery_error, nvlink_replay_error	errors/s
dcgm.mig.interconnect.nvlink.traffic	MIG NVLink Packet Rate	rx, tx	packets/s
dcgm.mig.interconnect.nvlink.ber	MIG NVLink Bit Error Ratio	effective, symbol	errors per trillion bits
dcgm.mig.interconnect.fabric	MIG Fabric Manager State	failure, in_progress, not_started, not_supported, nvml_too_old, success, unknown, unrecognized	state
dcgm.mig.interconnect.fabric_error	MIG Fabric Manager Error	error	code
dcgm.mig.interconnect.p2p	MIG Peer Access State	value_peer_gpu=*	state
dcgm.mig.interconnect.c2c.capacity	MIG C2C Link Capacity	maximum	bytes/s
dcgm.mig.interconnect.c2c.links	MIG C2C Link Count	links	links
dcgm.mig.interconnect.c2c.state	MIG C2C Link State	power_state, status	state
dcgm.mig.interconnect.c2c.throughput	MIG C2C All Traffic	rx, tx	bytes/s
dcgm.mig.interconnect.c2c.payload	MIG C2C Payload Traffic	rx, tx	bytes/s
dcgm.mig.interconnect.c2c.error_rate	MIG C2C Error Rate	intr, replay, replay_b2b	errors/s
dcgm.mig.reliability.xid	MIG XID Error Code	xid	code
dcgm.mig.reliability.xid_samples	MIG XID Samples	value_xid=*	samples
dcgm.mig.reliability.xid_rate	MIG XID Record Rate	value_xid=*	records/s
dcgm.mig.health.status	MIG Health Status by Watch	value_health_watch=*	state
dcgm.mig.state.performance	MIG Performance State	P0, P1, P10, P11, P12, P13, P14, P15, P2, P3, P4, P5, P6, P7, P8, P9, unknown	state
dcgm.mig.state.virtualization	MIG Virtualization Mode	mig_enabled, virtualization_bare_metal, virtualization_host_vgpu, virtualization_host_vsga, virtualization_passthrough, virtualization_unknown, virtualization_vgpu	state
dcgm.mig.state.configuration	MIG Compute and Persistence Modes	autoboost_enabled, compute_default, compute_exclusive_process, compute_exclusive_thread, compute_prohibited, compute_unknown, persistence_enabled	state
dcgm.mig.virtualization.vgpu.license	MIG vGPU License State	licensed	state
dcgm.mig.virtualization.vgpu.memory	MIG vGPU Memory Usage	used	bytes
dcgm.mig.virtualization.vgpu.frame_rate	MIG vGPU Frame Rate Limit	limit	frames/s
dcgm.mig.capability.mig_slices	MIG Maximum MIG Slices	maximum	slices

Per nvlink

An individual NVLink with its GPU, MIG or switch parent identity.

Labels:

Label	Description
gpu	GPU device index.
gpu_uuid	Parent GPU UUID.
gpu_i_id	MIG GPU instance identifier.
nvswitch	NVSwitch identifier.
nvlink	NVLink index within its parent.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.nvlink.interconnect.combined.throughput	NVLink Combined Link Throughput	nvlink_total	bytes/s
dcgm.nvlink.interconnect.throughput	NVLink Link Throughput	nvlink_rx, nvlink_tx	bytes/s
dcgm.nvlink.interconnect.error_rate	NVLink Link Error Rate	nvlink_crc_data_error, nvlink_crc_flit_error, nvlink_recovery_error, nvlink_replay_error	errors/s
dcgm.nvlink.interconnect.nvswitch.link_sxid	NVLink NVSwitch Link Error Code	fatal, non_fatal	code
dcgm.nvlink.interconnect.nvswitch.ecc_lanes	NVLink NVSwitch ECC Error Rate by Lane	lane0, lane1, lane2, lane3, lane4, lane5, lane6, lane7	errors/s
dcgm.nvlink.interconnect.nvswitch.errors	NVLink NVSwitch Error Rate	crc, ecc, recovery, replay	errors/s

Per nvswitch

An NVSwitch exposed by a supported fabric monitoring backend.

Labels:

Label	Description
nvswitch	NVSwitch identifier.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.nvswitch.interconnect.nvswitch.sxid	NVSwitch Error Code	fatal, non_fatal	code
dcgm.nvswitch.interconnect.nvswitch.temperature	NVSwitch Temperature	current	Celsius

Per cpu

A CPU exposed when DCGM CPU monitoring is available.

Labels:

Label	Description
cpu	CPU identifier.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.cpu.clock.frequency	CPU Clock Frequency	current	MHz
dcgm.cpu.cpu.utilization	CPU Busy Time	total	percentage
dcgm.cpu.cpu.utilization_modes	CPU Activity by Mode	irq, nice, sys, user	percentage
dcgm.cpu.cpu.temperature	CPU Temperature	critical, current, warning	Celsius
dcgm.cpu.cpu.power	CPU Power	current, limit	Watts

Per cpu_core

An individual CPU core exposed when DCGM CPU monitoring is available.

Labels:

Label	Description
cpu	CPU identifier.
cpucore	CPU core identifier.

Metrics:

Metric	Description	Dimensions	Unit
dcgm.cpu_core.clock.frequency	CPU Core Clock Frequency	current	MHz
dcgm.cpu_core.cpu.utilization	CPU Core CPU Busy Time	total	percentage
dcgm.cpu_core.cpu.utilization_modes	CPU Core CPU Activity by Mode	irq, nice, sys, user	percentage
dcgm.cpu_core.cpu.temperature	CPU Core CPU Temperature	critical, current, warning	Celsius
dcgm.cpu_core.cpu.power	CPU Core CPU Power	current, limit	Watts

Alerts

The following alerts are available:

Alert name	On metric	Description
dcgm_gpu_xid_errors	dcgm.gpu.reliability.xid	NVIDIA driver reported a GPU XID error (metric ${label:chart_context}).
dcgm_gpu_row_remap_failure	dcgm.gpu.reliability.row_remap_status	Row remapping has failed, indicating a persistent memory reliability problem.
dcgm_gpu_uncorrectable_remapped_rows	dcgm.gpu.reliability.row_remap_events	New uncorrectable row remap events were detected in the last 5 minutes.
dcgm_gpu_power_violation	dcgm.gpu.throttle.duration	The GPU was power-throttled during the last 5 minutes.
dcgm_gpu_thermal_violation	dcgm.gpu.throttle.duration	The GPU was thermally throttled during the last 5 minutes.

Troubleshooting

Diagnostics

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the dcgm collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn’t working.

Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that’s not the case on your system, open netdata.conf and look for the plugins setting under [directories].
```
cd /usr/libexec/netdata/plugins.d/
```
Switch to the netdata user.
```
sudo -u netdata -s
```
Run the go.d.plugin to debug the collector:
```
./go.d.plugin -d -m dcgm
```
To debug a specific job:
```
./go.d.plugin -d -m dcgm -j jobName
```

Getting Logs

If you’re encountering problems with the dcgm collector, follow these steps to retrieve logs and identify potential issues:

Run the command specific to your system (systemd, non-systemd, or Docker container).
Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep dcgm

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector’s name:

grep dcgm /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named “netdata” (replace if different), use this command:

docker logs netdata 2>&1 | grep dcgm

The observability platform companies need to succeed

Want a personalised demo of Netdata for your use case?

Contact Sales