SNMP runs over UDP port 161, a transport with no delivery guarantee. When your monitoring platform reports that devices are down, the first question is not “why is the network broken” but “is this actually a network problem, or is my polling stack the problem.” SNMP timeout and retry behavior is one of the most common causes of false-positive “device down” alerts, and it is also one of the most misdiagnosed.
The classic symptom: a dashboard lights up with dozens of devices showing as unreachable, but when you SSH into them or ping their management IPs, they respond fine. The SNMP agent is alive, the network path is intact, and the device is forwarding traffic. What failed was the SNMP poll, not the device.
This article covers how to distinguish a real outage from an SNMP timeout cascade, how to identify whether the bottleneck is on the collector side or the device side, and how to tune timeout and retry parameters so they stop producing false alarms without masking real failures.
What this means
When an SNMP poll times out, the collector marks the device as unreachable for that poll cycle. If retries are configured, the collector sends additional requests before declaring the device down. Each retry consumes a worker thread for the duration of the timeout window. The cumulative effect across many devices can starve the poller pool, causing other devices to time out in turn. This is the SNMP polling storm cascade: a feedback loop where retries amplify load on both the collector and the monitored devices.
Because SNMP over UDP provides no transport-layer acknowledgment or retransmission, a dropped, congested, or delayed request packet is indistinguishable from a genuinely unreachable device at the application layer. The retry mechanism is the only recovery path, and its tuning determines whether your monitoring produces signal or noise.
A single missed poll is normal under transient packet loss. The standard practice is to wait for 3 consecutive failed polls at the configured interval before declaring a device down. Sustained SNMP failure with healthy ICMP and syslog means an agent-specific problem, not a device outage, and should be demoted in severity.
flowchart TD
A[SNMP poll starts] --> B{Response within timeout?}
B -- Yes --> C[Device marked UP]
B -- No --> D[Timeout fires]
D --> E{Retries remaining?}
E -- Yes --> F[Send retry, hold worker]
F --> B
E -- No --> G[Device marked DOWN]
G --> H[Worker released late]
H --> I[Other polls delayed]
I --> J[More timeouts fleet-wide]
J --> K[False-positive cascade]Common causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| Collector overload | Many devices time out simultaneously; ICMP to those devices succeeds | Collector CPU, poll cycle duration vs. configured interval |
| Device control-plane CPU saturation | One device times out intermittently; device CPU is elevated during poll windows | Device CPU via SNMP or CLI |
| Aggressive bulk MIB walks | A specific OID walk consistently exceeds timeout | Per-device SNMP latency; which OID was being polled |
| Management network congestion | ICMP RTT to device is elevated during polling windows | ICMP latency baseline vs. SNMP poll latency |
| Incorrect community string or credentials | Device never responds; poll always reaches full timeout | SNMP authentication failure counters |
| SNMPv3 auth/priv overhead | SNMPv3 polls are slower than SNMPv2c on the same device | Compare v2c vs. v3 latency on same OID |
| Timeout too low for WAN or SD-WAN paths | Remote branch devices time out intermittently; local devices are fine | ICMP RTT to the remote device |
Quick checks
# Check if the device responds to SNMP at all, with a generous timeout and no retries
time snmpget -v2c -c <community> -t 5 -r 0 <device> .1.3.6.1.2.1.1.3.0
# Compare SNMP reachability against ICMP reachability
ping -c 5 -i 0.2 <device>
# Check device control-plane CPU (Cisco cpmCPUTotal5secRev; index .1 is the first CPU entry)
snmpget -v2c -c <community> <device> .1.3.6.1.4.1.9.9.109.1.1.1.1.7.1
# If the index is unknown, walk the table instead:
snmpwalk -v2c -c <community> <device> .1.3.6.1.4.1.9.9.109.1.1.1.1.7
# Check SNMP authentication failure counter (snmpInBadCommunityNames)
snmpget -v2c -c <community> <device> .1.3.6.1.2.1.11.4.0
# Measure SNMP poll latency variance over 10 polls (requires /usr/bin/time, not the shell builtin)
for i in $(seq 1 10); do
/usr/bin/time -f "%e" snmpget -v2c -c <community> -t 5 <device> .1.3.6.1.2.1.1.3.0 2>&1 | tail -1
done
# Check collector CPU per core (look for single-core saturation from RSS misconfiguration)
mpstat -P ALL 1 5
# Check whether the Netdata SNMP collector is behind on its poll cycle
curl -s http://localhost:19999/api/v1/allmetrics | grep -iE 'snmp.*timeout|poll.*duration|collector.*duration'
How to diagnose it
Corroborate with ICMP. Ping the devices showing as down. If ICMP succeeds but SNMP does not, the device is alive and the problem is SNMP-specific: agent overload, ACL blocking port 161, credential mismatch, or a collector-side issue. If both fail, the problem is the network path or the device itself.
Determine the blast radius. Check whether the timeouts affect one device, a group of devices sharing the same network path, or the entire fleet. One device points to a device-side issue. A group sharing a path points to management-network congestion. The entire fleet points to the collector.
Check the poll cycle duration. If the collector takes longer to complete a full poll cycle than the configured interval, polls are drifting. Data goes stale before it is collected. This is the most under-monitored meta-signal in network monitoring: when the poller falls behind, all other signals become unreliable.
Identify the slowest device. Look at per-device SNMP latency. A single device with a 10-second bulk walk can stall a worker thread and cascade delays across the schedule. Large table walks such as
cdpCacheTableordot1dTpFdbTableon a large switch routinely exceed default timeouts.Check device control-plane CPU. SNMP processing consumes device CPU. If CPU is above 70 percent during poll windows, the SNMP agent is being starved. Walking a large FDB table on a data center switch with tens of thousands of MAC entries can spike device CPU for seconds and degrade other control-plane functions like BGP hold-time processing.
Check for authentication failures. A wrong community string or expired SNMPv3 credential causes the device to silently drop the request. The poll waits the full timeout before failing. Check the authentication failure counter at
.1.3.6.1.2.1.11.4.0for SNMPv2c or the USM stats OIDs for SNMPv3.Check ICMP RTT to the device. If RTT to the management IP is elevated, the management path is congested. SNMP requests and replies traverse that same path. SNMPv3 with auth and privacy adds processing overhead compared to SNMPv2c, compounding the delay on high-latency paths.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
| SNMP timeout rate | Direct measure of polling reliability | Above 5 percent sustained is early warning; above 30 percent means data is unreliable for alerting |
| SNMP poll response latency | Indicates agent or path degradation | Latency above 1 second on a normally fast device |
| Poll cycle duration vs. interval | Detects schedule drift before it causes false downs | Cycle duration exceeding configured interval |
| Device control-plane CPU | SNMP starvation under CPU pressure | Above 70 percent sustained; above 90 percent warrants paging |
| ICMP reachability and RTT | Distinguishes network path issues from SNMP issues | SNMP down with ICMP up equals agent or collector problem |
| Collector CPU per core | Single-core saturation from RSS or thread contention | One core at 100 percent while others are idle |
| SNMP authentication failures | Credential mismatch or scanning | Any nonzero in production |
| Poller worker queue depth | Growing queue means collector cannot keep up | Queue depth above baseline or growing without bound |
Fixes
Collector-side: scheduler and worker pool
If many devices time out simultaneously and ICMP to those devices succeeds, the collector is the bottleneck. Reduce poller concurrency so the collector does not oversubscribe its own CPU or the management network. A smaller concurrency limit with a well-tuned timeout produces more reliable results than a large pool with aggressive timeouts.
Check the poll cycle duration against the configured interval. If the cycle takes longer than the interval, the collector is already behind. Either reduce the number of devices per collector, increase the poll interval, or distribute devices across additional collectors.
Timeout and retry tuning
The net-snmp CLI defaults are documented as 1 second timeout and 5 retries. For LAN-attached devices with fast agents, these defaults are often adequate. For WAN, SD-WAN, or cloud-managed devices, they are frequently too aggressive.
The key principle: set the timeout to accommodate the worst-case legitimate response time, then set retries conservatively. A single timeout should be long enough to absorb network RTT plus agent processing time plus a safety margin. If ICMP RTT to the device is 50ms, a 1-second timeout is generous. If the device is across an SD-WAN overlay with 200ms RTT and the agent takes 500ms to respond to a bulk walk, a 1-second timeout will fail intermittently.
A practical rule from operator consensus: ensure the cumulative worst-case timeout (per-request timeout multiplied by retries plus one) is shorter than the poll interval. This prevents a single stalled device from overlapping into the next scheduled poll.
Do not increase both timeout and retries simultaneously. Some NMS implementations use exponential or doubling backoff per retry. An initial timeout of 8 seconds with 5 retries can yield a cumulative timeout exceeding several minutes for a single device, which ties up a worker thread and accelerates schedule drift.
Slow device vs. packet loss: opposite tuning
The tuning direction depends on the failure pattern:
- Scattered individual timeouts suggest slow device response. Increase the timeout by roughly 50 percent and reduce retries by one. This gives each request more time to succeed without compounding the cumulative delay.
- Entire subnet or path timing out suggests packet loss on the management network. Increase retries by one and reduce the timeout by roughly 50 percent. Additional retry packets improve the probability of delivery over a lossy path. A shorter timeout means each retry fires sooner.
Device-side: control-plane CPU
If the device times out only during specific operations such as large bulk walks or FDB table queries, exclude those OID walks from the default polling schedule. Walk them on a separate, longer-interval schedule, or use targeted polling for specific OIDs instead of full table walks.
Check whether the device has a control-plane policer (CoPP) that is rate-limiting SNMP. On some platforms, SNMP is classified as a low-priority control-plane service and is dropped first under CPU pressure.
Credential and ACL issues
If a device never responds to SNMP but ICMP succeeds, verify the community string or SNMPv3 credentials. An incorrect community string causes the device to silently drop the request, and the poll waits the full timeout before failing. Check the SNMP authentication failure counter on the device. Also verify that the collector’s source IP is permitted by the device’s SNMP ACL on UDP port 161.
Prevention
- Monitor the SNMP timeout rate as a first-class signal. A timeout rate above 5 percent sustained is the early-warning threshold. Above 30 percent, polling data is unreliable for alerting and capacity decisions.
- Monitor poll cycle duration against the configured interval. If the cycle duration exceeds the interval, alerts based on stale data will produce false positives.
- Baseline SNMP latency per device. A device that was responding in 20ms and is now responding in 800ms is degrading. Catching this before it crosses the timeout threshold prevents a false-down alert.
- Separate bulk walks from standard polling. Large table walks (FDB, CDP cache, ARP) should run on their own schedule with their own timeout, not in the same worker pool as device-liveness polls.
- Size the worker pool to the collector’s capacity, not to the device count. A collector with 500 devices and a small fixed worker pool will fall behind if any device stalls.
- Correlate SNMP status with ICMP status in alerting. Suppress “SNMP device down” alerts when ICMP to the same device is healthy. SNMP failure with healthy ICMP is an agent or collector problem, not a device outage, and should be opened as a lower-severity ticket rather than a page.
How Netdata helps
- Netdata’s SNMP collector exposes per-device poll latency and timeout metrics, letting you identify the slow device before its delays cascade to the rest of the schedule.
- Per-core CPU metrics on the collector host catch single-core saturation from RSS misconfiguration or thread contention before it causes a polling cascade.
- Correlating SNMP agent reachability with ICMP reachability and syslog activity in a single dashboard makes the distinction between “agent down,” “device down,” and “network down” immediate.
- Netdata’s anomaly detection on SNMP response latency surfaces gradual degradation before it crosses the timeout threshold and triggers a false alert.
- Built-in alerting can require multiple corroborating signals before paging, reducing false-positive device-down alerts without missing real outages.