NIC RSS misconfiguration: one CPU core silently dropping your telemetry

Your flow collector has 16 CPU cores, but one is pinned at 100% while the other 15 sit idle. NIC receive drop counters are climbing. UDP socket buffer errors (Udp_RcvbufErrors) are incrementing. Your bandwidth charts show traffic declining during what is actually a traffic spike. The box looks underpowered, so you start sizing a bigger one. The real problem: Receive Side Scaling (RSS) is funneling every inbound packet to a single receive queue serviced by a single CPU core. No amount of additional cores or RAM fixes this until RSS distributes interrupts across them.

This failure is silent in the specific way that hurts most: aggregate CPU utilization looks healthy, the collector process is up, and dashboards render without errors. The data is just wrong. NetFlow records, sFlow samples, syslog messages, and SNMP traps that arrive at the NIC but cannot be drained from the kernel ring buffer in time are dropped before the application ever sees them. Charts trend downward because received-but-dropped packets never reach the TSDB.

The canonical symptom pair: a single core at 100% (mostly in softirq) while other cores are idle, combined with incrementing rx_missed_errors or per-queue drop counters on queue 0 only.

What RSS does and what breaks it

RSS hashes incoming packet headers and distributes the resulting flows across multiple hardware receive queues. Each queue has its own interrupt (IRQ), and each IRQ is bound to a specific CPU core. When RSS is disabled, misconfigured, or unsupported by the driver, all traffic lands on queue 0, and the single CPU core servicing that queue becomes the bottleneck for the entire receive path.

The bottleneck manifests as kernel ring buffer overflow. Packets arrive at the NIC faster than the one bound core can drain the ring, so the NIC drops them. The application-layer collector (nfdump, pmacct, Suricata, rsyslog) never sees these packets. Its “packets received” counter looks normal or even healthy because it only counts what made it past the kernel. The drop happens below the application, in the gap between the NIC hardware and the socket layer.

flowchart TD
    A[Inbound packets from exporters] --> B{RSS distributing?}
    B -- No / misconfigured --> C[All packets to queue 0]
    C --> D[Single CPU core handles all IRQs]
    D --> E[Core saturates in softirq]
    E --> F[NIC ring buffer overflows]
    F --> G[rx_missed_errors increment]
    G --> H[Socket buffer starved]
    H --> I[Udp_RcvbufErrors increment]
    I --> J[Silent telemetry loss]
    B -- Yes --> K[Spread across N queues]
    K --> L[IRQs distributed across cores]
    L --> M[Ring buffer drains fast]
    M --> N[No drops]

Common causes

Quick checks

Read-only commands. None of them modify system state.

# Check aggregate and per-core CPU; look for one core at 100% in %soft
mpstat -P ALL 1 5

# Check IRQ distribution across CPU cores for the NIC
grep -E 'CPU|<iface>' /proc/interrupts

# Check softirq rates; NET_RX should be spread, not concentrated
cat /proc/softirqs | grep -E 'NET_RX|NET_TX'

# Check kernel packet processing backpressure
cat /proc/net/softnet_stat

# Check NIC ring buffer current and max settings
ethtool -g <iface>

# Check per-queue drop counters
ethtool -S <iface> | grep -iE 'drop|miss'

# Check number of combined channels (queues)
ethtool -l <iface>

# Check whether receive-hashing is enabled
ethtool -k <iface> | grep receive-hashing

# Check the RSS indirection table
ethtool -x <iface>

# Check UDP socket buffer drops (the downstream symptom)
cat /proc/net/snmp | grep '^Udp:'

# Check current socket buffer fill for a specific listener
ss -lun '( sport = :2055 )' -m

How to diagnose it

1. Confirm the single-core bottleneck. Run mpstat -P ALL 1 5 and look at the %soft column. If one core shows sustained high softirq time (often above 80-90%) while other cores are near zero, packet receive processing is concentrated on that core. Aggregate CPU may look fine.

2. Verify IRQ distribution. Run grep -E 'CPU|<iface>' /proc/interrupts. Each receive queue should have its own interrupt line, and those lines should show counts spread across different CPU columns. If all queue interrupts are concentrated in one CPU column, RSS is not distributing or IRQ affinity is misconfigured.

3. Check the number of configured queues. Run ethtool -l <iface>. The output shows Combined (or separate RX/TX) channel counts. If Combined: 1, the NIC is using a single queue and RSS cannot distribute regardless of hashing settings. Production collectors should have the queue count set to match or approach the number of physical CPU cores available for receive processing.

4. Verify RSS hashing is active. Run ethtool -k <iface> | grep receive-hashing. If receive-hashing: off, the NIC is not computing RSS hashes and all traffic defaults to queue 0.

5. Inspect the indirection table. Run ethtool -x <iface>. This shows which RSS hash buckets map to which queues. If the table is heavily skewed toward queue 0, traffic distribution is uneven even with multiple queues configured. A healthy table distributes buckets roughly evenly across all available queues.

6. Check for hardware-level drops. Run ethtool -S <iface> | grep -iE 'drop|miss'. Counter names are driver-specific. Look for rx_missed_errors, rx_no_dma_resources, or per-queue counters like rx_queue_0_drops. Nonzero values on queue 0 while other queue counters are zero confirms the concentration problem.

7. Confirm the downstream impact. Run cat /proc/net/snmp | grep '^Udp:' and check the RcvbufErrors column. If it is incrementing, packets that made it past the NIC ring are being dropped at the socket buffer because the bound core cannot drain the socket fast enough.

8. Check softnet_stat for kernel backpressure. In /proc/net/softnet_stat, the second column counts packets dropped at the softirq level. The third column (time_squeeze) indicates the softirq exhausted its budget without necessarily dropping; sustained nonzero time_squeeze is a precursor to drops.

Metrics and signals to monitor

Fixes

Enable RSS and set the queue count

If ethtool -l shows a single combined queue and the NIC supports more, increase the channel count. This briefly interrupts traffic on the interface.

# Check maximum supported channels
ethtool -l <iface>

# Set combined queues to match available cores (N <= physical core count)
ethtool -L <iface> combined <N>

On some Intel adapters, configuring 8 or more queues requires a system reboot to reprogram the indirection table. Check the Maximum value before setting.

Enable receive hashing

If receive-hashing is off, enable it:

ethtool -k <iface> | grep receive-hashing
# If off, enable via the feature name
ethtool -K <iface> rxhash on

Fix IRQ affinity

Even with correct RSS, if IRQ affinity is manually pinned to one core, distribution fails. On most modern distributions with irqbalance running, IRQ affinity is managed automatically. If irqbalance is disabled or overridden, verify that each queue’s IRQ is assigned to a different core. Check /proc/irq/<irq>/smp_affinity for each NIC receive queue interrupt.

Increase the ring buffer

RSS fixes distribution, but an undersized ring buffer still drops packets during bursts. Changing the ring briefly interrupts traffic on the interface.

# Check current and maximum ring settings
ethtool -g <iface>

# Increase RX ring to maximum supported
ethtool -G <iface> rx 4096

This is a mitigation, not a replacement for RSS. A larger ring buffer on a single-queue configuration delays the drop rather than preventing it under sustained load.

Handle NICs without RSS support

Some NICs lack functional RSS in their Linux drivers. Realtek RTL8125 and RTL8168 chips are common examples: the upstream r8169 kernel driver does not expose RSS queues. The software fallback is RPS (Receive Packet Steering), configured via /sys/class/net/<dev>/queues/rx-<n>/rps_cpus. RPS distributes packet processing across cores in software, with higher overhead than hardware RSS. For production telemetry collectors, replacing the NIC with one that has hardware RSS support (Intel X710, Broadcom NetXtreme, Mellanox ConnectX) is the durable fix.

Check for the DPDK X710 regression

If you are running a DPDK-based collector on an Intel X710 with the i40e PMD, DPDK 20.11.3 introduced a regression where all packets land on queue 0 regardless of the rss_queues setting. Downgrade to DPDK 20.11.2 or upgrade past the affected version.

Persist settings across reboots

ethtool settings do not survive reboot by default. Persist them via a udev rule or a systemd service that applies the configuration after the interface comes up. The common pattern is a oneshot systemd unit running after network.target.

Prevention

• Monitor per-core CPU, not just aggregate. Aggregate CPU hides single-core saturation. Dashboards should show per-core utilization with attention to the %soft column.
• Verify RSS on every collector deployment. Add ethtool -l, ethtool -k | grep receive-hashing, and IRQ distribution checks to your provisioning checklist. RSS is set once and forgotten, which is exactly why it drifts.
• Alert on rx_missed_errors. Any nonzero increment on a flow-ingress NIC is a data-loss event.
• Alert on Udp_RcvbufErrors. This confirms the single-core bottleneck is causing actual telemetry loss.
• Baseline the indirection table. After correct configuration, record the expected ethtool -x output. A skewed table after a driver update or firmware change is a leading indicator.
• Compare collector inbound rate against device-side export counts. Device-side flow export counters (for example, Cisco cfnFlowsExported at .1.3.6.1.4.1.9.9.367.1.4) versus collector inbound rate is the only reliable end-to-end loss detection. A gap means silent loss somewhere in the path.

How Netdata helps

Netdata collects the signals needed to detect this failure before data loss:

• Per-core CPU utilization with softirq breakdown exposes the single-core saturation pattern that aggregate CPU hides. Correlate a pinned core with rising NIC drops in the same dashboard.
• NIC RX/TX drops from /proc/net/dev and ethtool -S are collected automatically, including rx_missed_errors where exposed by the driver.
• Softirq statistics from /proc/softirqs and /proc/net/softnet_stat show kernel-level packet processing backpressure.
• UDP socket buffer errors (Udp_RcvbufErrors) from /proc/net/snmp let you correlate NIC-level drops with socket-level drops.
• Per-interrupt CPU distribution from /proc/interrupts reveals whether NIC IRQs are spread across cores or concentrated on one.
• Disk and write queue metrics on the TSDB volume help distinguish RSS-induced drops from downstream storage backpressure.

The value is correlation: a single chart showing per-core CPU, NIC drop counters, and Udp_RcvbufErrors together makes the RSS diagnosis obvious in seconds instead of hours.

Related guides

• ARP cache staleness: when IP-to-MAC mapping goes bad
• Asymmetric routing: why your path and latency measurements lie
• Audit log gaps: detecting syslog/trap tampering or loss
• BGP flapping: why a peer keeps resetting and how to find the cause
• BGP NOTIFICATION and Cease messages: what each subcode is telling you
• BGP RIB and FIB growth: monitoring route-table size before it bites
• BGP route leak and hijack: the detection signals and alerts that matter
• BGP session Established but stale: detecting silent route loss
• Cold-start topology: why your map is incomplete after a collector restart
• Locating endpoints behind NAT and wireless: the positioning problem
• Stale FDB/MAC tables: why endpoint location is wrong
• NetFlow storage sizing: how much disk your flow collector really needs