Vendor API latency and pagination: monitoring pull-mode collection

For SD-WAN controllers, cloud-managed networking, and modern firewall platforms, vendor API pull-mode collection is now the primary telemetry path. Operators depend on HTTPS calls to Meraki, Cato, PAN-OS, RESTCONF, and gRPC endpoints to learn tunnel state, license validity, session counts, and topology. Each API has its own authentication model, rate-limit budget, and pagination semantics. A collector that ignores these constraints will silently lose data, get throttled, or report healthy when the payload is empty.

The dominant failure mode is not the API going down. It is the collector treating HTTP 200 as success without validating the payload, exhausting a shared rate-limit budget, or stalling on paginated responses that never complete. These are silent failures: dashboards flat-line, no error is logged at INFO, and the first signal is a user complaint hours later.

What it is and why it matters

Pull-mode collection means the monitoring platform initiates a request to the vendor API and waits for a response. This is distinct from push-mode telemetry such as streaming gNMI, YANG Push (RFC 8641), or syslog traps, where the device or controller sends data unsolicited.

For API-first and controller-managed platforms, pull-mode is often the only option. Meraki, Aruba Central, Juniper Mist, and Cisco DNA/Catalyst Center expose most operational state through their APIs, not through SNMP. SD-WAN overlays like Cato, Viptela, Versa, and Fortinet make tunnel state, SLA metrics, and application-aware routing decisions available via the orchestrator API. For these platforms, the API is the telemetry source.

The shift from SNMP to vendor APIs changes the failure surface. SNMP is UDP-based with no session overhead; a request and response are a single round-trip. Vendor APIs run over HTTPS, incurring TCP connection setup and TLS handshake cost on every request unless the collector pools connections. Authentication adds another layer: API keys expire, tokens refresh on schedules, and SAML/SSO administrators on some platforms (Meraki) cannot generate API keys at all.

How it works

A pull-mode collection cycle has four stages, each with its own failure mode.

flowchart LR
  S[Scheduler] --> A[Auth / token refresh]
  A --> R[API request]
  R --> RL{Rate limit remaining?}
  RL -->|Exhausted| T[HTTP 429 + Retry-After]
  RL -->|OK| P{Paginated response?}
  P -->|Yes| PG[Fetch next page via cursor or offset]
  P -->|No| V[Validate payload]
  PG --> V
  V --> E{Schema valid?}
  E -->|No| G[Silent gap: HTTP 200 + empty or error body]
  E -->|Yes| W[Write to TSDB]

Stage 1: Authentication. The collector presents credentials to the vendor API. PAN-OS uses an API key requested via /api/?type=keygen&user=<user>&password=<password>, then passed as an X-PAN-KEY header or key= query parameter on subsequent calls. Meraki uses a bearer token in the Authorization header. Cato uses an x-api-key header against a GraphQL endpoint at https://api.catonetworks.com/api/v1/graphql2. Key expiration and rotation are the leading cause of silent collection failure: a key that worked yesterday returns 401 today (or, for Meraki, 404 by design to avoid leaking resource existence).

# Check PAN-OS API key validity and response payload
# WARNING: the key appears in the URL and will be visible in shell history
# and process listings. Use this only for debugging, not in production scripts.
curl -sk "https://<fw>/api/?type=op&cmd=<show><system><info></info></system></show>&key=<apikey>"

# Check Meraki API reachability and HTTP status separately
curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $KEY" \
  https://api.meraki.com/api/v1/organizations

# Isolate DNS resolution time from total API latency
curl -s -o /dev/null -w "dns:%{time_namelookup} total:%{time_total}\n" \
  https://api.meraki.com/api/v1/organizations

Stage 2: Rate-limit budget. Every vendor imposes per-token, per-minute, or per-query limits. Meraki allows 10 requests per second per organization, with a burst of 30 in 2 seconds, and 100 per second per source IP. Cato enforces per-query, per-account limits: 120/min general, but accountSnapshot at 1/sec, accountMetrics at 15/min, and eventsFeed at 100/min. Multiple collectors sharing the same API key share the same budget. When the budget is exhausted, the API returns HTTP 429, and collection stops until the window resets. Meraki returns a Retry-After header. Cato does not formally publish response header names for rate limiting, so verify empirically.

Stage 3: Pagination. Large responses (FDB tables, event feeds, license inventories) are delivered across multiple pages. The IETF is standardizing pagination for RESTCONF and NETCONF via two active drafts: draft-ietf-netconf-list-pagination and draft-ietf-netconf-list-pagination-rc. These define query parameters including:

The defined processing order is: where, then sort-by, then direction, then (offset or cursor), then limit. Offset and cursor are mutually exclusive per the draft specification. Cursor values are opaque and ephemeral; collectors must not cache or reuse them across requests. Before these drafts existed, vendors implemented proprietary parameters (such as ?page=1&size=50), and operators had to discover behavior per vendor empirically.

The drafts explicitly warn that retrieving all entries without pagination “can lead to inefficiencies (e.g., long loading time, memory overconsuming, or crash) in the server, the client, and the network in between.” A paginated response that stalls partway through (due to a timeout, cursor expiry, or rate-limit hit mid-walk) produces a partial dataset, and the collector may not detect the incompleteness.

Stage 4: Response validation. The collector must verify not just the HTTP status code but the payload contents. PAN-OS returns <response status="error"> inside an HTTP 200. A collector that checks only the HTTP status will treat this as success and record the error payload as valid telemetry. Schema validation catches vendor-side API changes that break the adapter without changing the HTTP status.

Where it shows up in production

Five deployment variants depend heavily on vendor API pull-mode collection:

• API-first / controller-managed (Meraki, Aruba Central, Juniper Mist, Cisco DNA/Catalyst Center). Many operational facts live in the controller API, not on the device. Topology is often controller-asserted rather than inferred from CDP/LLDP. SNMP may be entirely absent.
• SD-WAN overlay (Cato, Viptela, Versa, Fortinet). The tunnel is the unit of interest, not the physical interface. Tunnel up/down state, SLA probe latency, jitter, loss, and application-aware routing decisions all come from the orchestrator API. Underlay versus overlay confusion is the most common misdiagnosis.
• Cloud-native (AWS VPC Flow Logs, Azure NSG Flow Logs, GCP VPC Flow Logs). No SNMP, no CDP, no LLDP. Flow records arrive via object storage poll, not UDP. Delivery lag is measured in minutes, not seconds. Sampling is implicit in the cloud provider’s collection mechanism.
• Modern firewalls (PAN-OS, FortiGate). Session counts, NAT translation tables, license state, and threat logs are available via XML or JSON-RPC APIs. These typically supplement, not replace, SNMP for interface counters.
• Hybrid / multi-vendor. Most enterprises. Requires a normalization layer to translate vendor-specific API schemas into a common data model, which is the highest operational complexity for topology inference and flow normalization.

Tradeoffs and when to use it

Pull-mode vendor APIs solve real problems that SNMP cannot: structured, model-driven payloads, controller-asserted topology, and access to state that exists only in the management cloud. But they introduce failure modes that SNMP does not have.

Connection overhead. Each RESTCONF or vendor HTTPS request incurs TCP and TLS session cost absent from SNMP’s UDP model. Connection pooling is essential for high-volume polling. Without it, per-request handshake overhead dominates latency measurements and reduces effective throughput.

Rate-limit contention. SNMP has no equivalent of a shared per-token quota. When multiple tools (NMS, security scanner, automation script, ad-hoc dashboard) share one Meraki or Cato API key, a single aggressive consumer can exhaust the budget and blind every other consumer. Track consumption against the published limits and watch for Retry-After headers before throttling occurs.

The 200-with-empty-payload trap. This is the most operationally damaging failure mode. The API is up, the network path is fine, ICMP to the vendor cloud is healthy, but the payload is empty or schema-mismatched. This happens during vendor maintenance windows, after an API schema change without a version bump, or after a vendor-side incident. Collectors that validate only HTTP status record silence as success. The signal that catches it is tracking data freshness (time since last valid payload) independently of HTTP status.

Streaming as an alternative. Where available, streaming telemetry (gNMI Subscribe RPC in STREAM mode, YANG Push per RFC 8641) removes the polling bottleneck entirely. gNMI also supports a POLL mode for collector-initiated snapshots. But streaming coverage is uneven across vendors and device generations; older devices require polling regardless. The choice between pull and push is often dictated by what the platform supports, not by operator preference.

Retry and backoff discipline. When a vendor API returns 429 or 5xx, the collector must back off. The standard pattern is exponential backoff with jitter: start at 1 to 2 seconds, double each attempt, add random jitter, and cap retries at 3 to 5. Without jitter, synchronized retry storms from multiple collectors can amplify load on the vendor endpoint.

Signals to watch in production

How Netdata helps

Netdata instruments per-API-call latency, HTTP status distribution, and response validation as first-class metrics, and correlates them alongside SNMP, flow, and syslog signals on a unified timeline:

• Correlate vendor API latency spikes with SNMP timeout rates and device control-plane CPU on the same dashboard to distinguish collector-side issues from device-side problems.
• Track HTTP 429 counts and rate-limit consumption trends alongside API-sourced data freshness to catch throttling before it causes a silent data gap.
• Alert on payload validation failures (HTTP 200 with error or empty body) as a signal distinct from transport-level errors, so the 200-with-empty-payload trap does not go unnoticed.
• Monitor collector CPU, DNS resolution time, and connection pool behavior to detect collector-side bottlenecks in the pull pipeline.
• Combine vendor API health with ICMP reachability to the vendor cloud endpoint to separate local network path issues from vendor-side outages.

Related guides

• ARP cache staleness: when IP-to-MAC mapping goes bad
• Asymmetric routing: why your path and latency measurements lie
• Audit log gaps: detecting syslog/trap tampering or loss
• BGP flapping: why a peer keeps resetting and how to find the cause
• BGP NOTIFICATION and Cease messages: what each subcode is telling you
• BGP RIB and FIB growth: monitoring route-table size before it bites
• BGP route leak and hijack: the detection signals and alerts that matter
• BGP session Established but stale: detecting silent route loss
• Cold-start topology: why your map is incomplete after a collector restart
• Locating endpoints behind NAT and wireless: the positioning problem
• Stale FDB/MAC tables: why endpoint location is wrong
• NetFlow storage sizing: how much disk your flow collector really needs