Ceph RGW failed requests: aborted request rate and what it actually counts

ceph_rgw_failed_req is one of the most commonly misread RADOS Gateway metrics. Operators see “failed” in the name and assume it counts HTTP 4xx and 5xx responses. It does not. It counts requests where the client connection was aborted before the response completed. That distinction changes who you page, where you look, and which signals you correlate with.

What it is and why it matters

ceph_rgw_failed_req is a per-instance counter exposed by each RGW daemon, scraped from the admin socket and labeled with instance_id (alongside ceph_daemon, instance, job). Its sibling counter is ceph_rgw_req, the total request count. The operational signal is the ratio between the two: a sustained failed-request rate above 5% of total traffic for more than 5 minutes is the playbook’s TICKET condition.

RGW is stateless, horizontally scalable, and almost always behind a load balancer. A single gateway dying does not produce a rising failed ratio; the LB routes around it. A rising ratio across multiple instance_id labels means clients are giving up on requests in flight, which means the gateways are taking too long to respond. The most common reasons for that are load-balancer timeouts and RADOS slowness, not RGW itself.

The misread is treating ceph_rgw_failed_req as an HTTP error counter. A request that completes with a 403, 404, or 500 increments ceph_rgw_req and does not touch failed_req. For HTTP status breakdown you need the RGW access log, not this metric.

How it works

The counter lives in the rgw perf counter section and is described in the schema as “Aborted requests.” It increments once per request that is aborted before a completed response: the RGW frontend is writing the response back to the socket, the write fails because the peer has gone away, and the request is accounted as aborted. In current Ceph the increment happens in abort_early() in rgw_rest.cc (the counter itself is registered in rgw_perf_counters.cc).

Two things follow. First, the increment is connection-level, not status-level. There is no notion of “the request was going to be a 500” in this counter. Second, the counter increments even when the RGW is still working on the request. If RADOS takes 60 seconds to satisfy a GET and the client (or its load balancer) gives up at 30 seconds, the RGW eventually notices the dead socket and increments failed_req.

flowchart TD
  A[Client opens HTTP connection] --> B[LB forwards to RGW]
  B --> C[RGW processes request]
  C --> D[RGW issues RADOS op]
  D --> E{Client still connected
when response is ready?}
  E -->|Yes| F[RGW sends response
2xx, 4xx, or 5xx]
  F --> G["ceph_rgw_req++
failed_req NOT touched"]
  E -->|No, client closed first| H[Socket write fails]
  H --> I["ceph_rgw_req++
ceph_rgw_failed_req++"]

failed_req is a “client gave up” signal, not a “RGW returned an error” signal. The cause can be anywhere on the path: the load balancer’s timeout, the network between LB and RGW, the RGW’s own processing, or the RADOS op sitting in an OSD queue for 40 seconds.

What does NOT increment the counter is just as important:

• HTTP 4xx and 5xx responses. A request that completes inside the op handler (a 404 for a missing key, a handler-level 500) is a completed response: it increments ceph_rgw_req only and does not touch failed_req. Requests aborted before execution (auth failures, rate limiting, init errors) do count against failed_req.
• Rate-limited requests. When rgw_max_concurrent_requests (default 1024) is exceeded, scheduling fails and RGW returns a 503 with an internal ERR_RATE_LIMITED (errno 2218, exposed as -2218). Because this aborts the request before execution, it does count against failed_req.
• RADOS timeouts that hang the RGW. There is a reported failure mode where a get_obj operation times out against RADOS (op status -110, ETIMEDOUT) and the RGW reports http_status=200 while sending no body. The connection was not aborted by the client, so this does not show up in failed_req until the client eventually gives up. If clients have very long or no timeouts, this class of bug can hide entirely from the ratio alert.

Where it shows up in production

The dominant cause of an elevated failed_req ratio is load-balancer timeouts. HAProxy, nginx, F5, or cloud LBs with timeout server or timeout client settings shorter than the worst-case RGW response time will abort connections whenever the backend is slow. The RGW logs the failed socket write (for example ERROR: abort_early: send_body() returned err=..., errno -5/EPIPE) in these cases. From the RGW’s perspective, this looks identical to a real client disconnect.

The second most common cause is genuine RADOS slowness pushing RGW response times past client or LB timeouts. The failed ratio rises on RGW, but the cause is a hot OSD, a BlueStore compaction stall, a nearfull cluster throttling backfill, or an OMAP storm on a bucket index. The RGW is the messenger.

The third pattern is RGW-local saturation. The SimpleThrottler (beast/dmclock frontend) bounds outstanding requests by rgw_max_concurrent_requests (default 1024), and there have been reports of RGWs reaching that limit and failing to recover, causing all new connections to be aborted. This is rarer but worth checking when the failed ratio is concentrated on a single instance_id while sibling gateways are healthy.

A diagnostic trap: RGW does not always log the actual HTTP return code when a client disconnects mid-request. The access log may show a misleading status instead of an explicit abort marker, which makes pure log analysis unreliable for explaining a failed_req spike. Trust the metric for the count; use the log for the request shape, not the outcome.

Tradeoffs and common misuses

Alerting on absolute rate instead of ratio. A flat threshold like rate(ceph_rgw_failed_req[5m]) > 10 fires differently on a 100 RPS site and a 10,000 RPS site. The ratio alert (> 0.05 sustained for 300 seconds) normalizes for traffic volume and is the form the playbook specifies. Reserve absolute-rate alerts for capacity-style ceilings you have tuned to your site.

Treating the ratio as an HTTP error rate. If you report “RGW error rate is 4%” based on failed_req / req, you are reporting the abort rate, not the error rate. For HTTP errors you need access-log-derived metrics, grouped by status code class.

Aggregating across instance_id blindly. Summing failed_req across all RGW instances is fine for a cluster-wide ticket, but it hides single-gateway failures. A misbehaving gateway behind a round-robin LB will show as a moderate cluster-wide ratio when it is really one bad instance. Always break the ratio down by instance_id during investigation.

Alerting without correlation. A failed_req ticket with no OSD, capacity, or OMAP context attached sends engineers to the RGW hosts first. Most of the time the RGW hosts are fine. Wire the alert to carry (or link to) the concurrent state of ceph_osd_commit_latency_ms, ceph_healthcheck_slow_ops, nearfull status, and LARGE_OMAP_OBJECTS.

Ignoring cold-start noise. A rolling RGW restart will briefly spike failed_req as in-flight requests lose their backend. The 300-second sustain window in the ticket condition is partly there to absorb this.

Signals to watch in production

How Netdata helps

• Netdata scrapes ceph_rgw_req and ceph_rgw_failed_req per second and preserves the instance_id label, so the ratio alert fires on the right gateway and the per-instance breakdown is one click away.
• Per-second granularity catches the LB-timeout cascade pattern, where aborts cluster in tight bursts that minute-bucketed scrapers smear into noise.
• RGW metrics sit alongside OSD latency, slow-ops, capacity, and OMAP health signals in the same node view, so the natural first move on a failed_req ticket is to check RADOS rather than SSH into the RGW host.
• ML anomaly detection on the failed ratio and on RGW queue depth flags slow drift that precedes a hard ticket, typical of the capacity-and-RADOS-slowness class where the ratio creeps up over hours.
• Correlating RGW abort rate with LB-side connection metrics on the load-balancer hosts distinguishes “LB gave up first” from “RGW never responded” without requiring log scraping.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph FS_DEGRADED: standby MDS failed to take over a rank
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault