HAProxy DroppedLogs: losing log lines exactly when you need them

You are mid-incident. Traffic is spiking, 5xx rates are climbing, and you go to the HAProxy logs for per-request timing and termination codes. The logs are not there. Whole seconds of traffic are missing, right at the peak. Then you notice DroppedLogs in show info has been incrementing for hours and nobody was watching it.

DroppedLogs counts log messages HAProxy tried to emit but could not deliver. The cause is almost never HAProxy itself: it is the log pipeline downstream. A syslog daemon that cannot keep up, a socket buffer that overflowed, a log target that stalled. There is no error in the HAProxy log, because the line that would have told you is one of the ones that got dropped.

The failure mode has a cruel symmetry: log volume scales with traffic, so the pipeline is most likely to saturate exactly when traffic is highest, which is when you need per-request logs most. Any nonzero increment of DroppedLogs deserves a ticket. Not because dropped logs hurt traffic (they do not), but because they destroy your ability to diagnose the next problem.

What this means

HAProxy emits a log line when a proxied session ends. Under load this is thousands of lines per second, going out over whatever log target you configured: a local syslog socket, a UDP syslog endpoint, a file descriptor, or a ring buffer. HAProxy emits a single log per transaction, once every item in the log format can be satisfied; in practice that means after the end of the response for HTTP or at the end of the connection for TCP, so in HTTP mode one line can cover several keep-alive requests. option logasap emits the line as soon as the response headers are sent, and log-steps selects finer-grained origins (accept, connect, request, response, close).

When the delivery path cannot absorb the volume, HAProxy does not block the event loop waiting for the log sink. It drops the message and increments DroppedLogs. That is the right design for a proxy (logging must never stall traffic), but it means the log sink’s capacity problem becomes your observability problem.

flowchart LR
  S[Sessions close] --> E[HAProxy emits log line]
  E --> B{Log sink accepts?}
  B -->|yes| L[syslog / file / ring]
  B -->|no, buffer full or sink slow| D[DroppedLogs counter +1]
  L --> W[written to disk or forwarded]
  D -.->|silent: no log line| X[message lost forever]

Two properties matter operationally:

• Dropped messages are gone. HAProxy does not retry or spool them. The counter is the only evidence they existed.
• The counter only sees local delivery failures. If HAProxy hands a UDP datagram to the kernel and the network loses it, or the remote syslog drops it after receipt, DroppedLogs does not move. The counter reflects “HAProxy could not send,” not “the log line made it to disk.”

One nuance: the HAProxy documentation ties this drop-and-count behavior to unbuffered file descriptor targets (fd@<n>, stdout, stderr), where a failed writev() increments the counter. With UDP syslog targets, the kernel socket buffer can also overflow silently. Operators have reported the counter moving with local syslog socket targets too, so treat any increment as real regardless of target type, but do not assume a zero counter means every line reached your log store.

Common causes

Quick checks

All of these are read-only. Adjust the socket path if your build puts it elsewhere (commonly /var/lib/haproxy/stats).

# 1. Read the counter itself
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^DroppedLogs:"

# 2. Take two samples 60s apart to get a drop rate, not just a lifetime count
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^DroppedLogs:"
sleep 60
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^DroppedLogs:"

# 3. Confirm current traffic and log volume pressure
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(SessRate|ConnRate|CurrConns|Uptime_sec):"

# 4. See what log targets HAProxy is configured with
grep -E "^\s*log " /etc/haproxy/haproxy.cfg

# 5. Is the syslog daemon running and has it restarted recently?
systemctl status rsyslog --no-pager 2>/dev/null | head -5 || systemctl status syslog-ng --no-pager | head -5

# 6. Kernel UDP receive buffer defaults (per-socket limits the syslog listener inherits)
sysctl net.core.rmem_default net.core.rmem_max

# 7. Where are the gaps? Show per-second line counts, lowest first
awk '{print substr($0,1,15)}' /var/log/haproxy.log | uniq -c | sort -n | head

Notes on interpretation:

• Uptime matters. DroppedLogs is a process counter. A large value with a long Uptime_sec may be ancient history; the counter belongs to the current worker process (it is zeroed at process start and does not carry across reloads), and the 60-second delta is what tells you whether you are dropping now.
• Check the syslog daemon’s restart history. A syslog restart removes and recreates the local socket or briefly stops receiving; lines sent in that window are lost. If drop bursts line up with package updates or logrotate-triggered restarts, that is your cause.
• Check 7 finds thin seconds, not empty ones. Seconds with zero lines never appear in the output because they produce no input lines. During a busy period, look for counts of 1 or 2 amid thousands; that is your gap.

How to diagnose it

1. Establish the drop rate. Two samples of DroppedLogs 60 seconds apart. Zero means the incident is historical; positive means the pipeline is failing now. Express it as drops per second and compare to your log line rate, which tracks SessRate (one line per session in HTTP mode).

2. Identify the configured log path. Look at every log line in the global and defaults sections. The three shapes you will find: a Unix socket (/dev/log or an rsyslog-created socket), UDP to a local or remote host:port, or a file descriptor / ring target. The fix is different for each.

3. Check the syslog daemon end of the pipe. If you log to rsyslog, look at its internal queue statistics (impstats) if enabled: a growing main queue or a discarded-message counter on the imuxsock/imudp input is the smoking gun. If rsyslog forwards to a remote destination, check whether that downstream is slow, because rsyslog backpressure propagates back to the socket HAProxy writes to.

4. Time-correlate with restarts and config changes. Compare drop onset with syslog restarts, HAProxy reloads, logrotate runs, and any change that increased per-line size or lines per session. A custom log-format with extra sample fetches can make lines substantially longer; longer lines fill buffers faster.

5. Check buffer sizing against peak line rate. For a UDP or Unix socket target, the kernel receive buffer is the shock absorber between HAProxy’s burst and the syslog daemon’s read loop. If the burst exceeds buffer bytes divided by average line size, you drop. Defaults are often far too small for thousands of lines per second.

6. Rule out the network path (for remote syslog). Remember the counter’s blind spot: datagrams HAProxy sent successfully can still be lost in the network or dropped by the receiving host’s own socket buffer. If DroppedLogs is zero but lines are missing at the destination, the loss is downstream. Compare line counts at the receiver against HAProxy’s session counters for the same window.

Metrics and signals to monitor

Fixes

Grouped by cause. None of these require restarting HAProxy.

Syslog daemon cannot keep up

Make the receiver faster before touching HAProxy. Typical levers: move rsyslog’s main queue to a disk-assisted or purely in-memory queue sized for your peak line rate, disable expensive per-message processing (deep parsing, heavy templates) on the HAProxy input, and write logs with asynchronous flushing if filesystem latency is the bottleneck. If rsyslog forwards to a remote collector, decouple the forward with a local action queue so downstream slowness cannot backpressure the input socket.

Tradeoff: in-memory queues lose buffered messages if the daemon crashes. That is still better than losing them continuously at the socket.

Socket buffer too small

Increase the receive buffer on the syslog input socket and raise the kernel ceilings (net.core.rmem_max) so the daemon is allowed to request a big buffer. Note that not every input module exposes buffer sizing: rsyslog’s imudp exposes a per-listener rcvbufSize input parameter (for example input(type="imudp" port="514" rcvbufSize="1m")), imtcp has no receive-buffer parameter (its relevant input knob is SocketBacklog, the TCP listen backlog), and imuxsock exposes no buffer-size knob at all — check your rsyslog version’s documentation rather than assuming every module supports the knob. Size the buffer to absorb your worst realistic burst: peak lines per second multiplied by average line size multiplied by the seconds of stall you want to survive.

Tradeoff: bigger buffers hide brief stalls but cannot fix a receiver that is chronically slower than the line rate. If the daemon reads slower than HAProxy writes on average, no buffer size saves you.

Syslog restarts dropping logs

If drops align with syslog restarts, reduce restart frequency (config management churn is the usual cause) and make restarts fast. On the HAProxy side, a more durable fix is to change the transport.

Move to a buffered or reliable transport

Two structural options:

• Ring buffers. HAProxy can log to an internal ring buffer (ring@<name>) and have a log server drain it. This decouples emission from delivery: short downstream stalls are absorbed by the ring instead of dropping messages at emit time. The tradeoff is that a full ring overwrites old messages rather than counting drops, so you trade silent loss at the socket for silent loss in the ring unless you also watch the ring. Rings are drained through the runtime socket with show events <sink>, which can tail the buffer, but the ring itself exposes no operational overflow counter, so visibility into ring loss is limited: with a TCP ring forwarder attached, a stalled forwarder blocks the ring (docs note all servers attached to one ring receive the same copy and the ring advances at the speed of the slowest), and a ring nobody reads simply overwrites.
• TCP instead of UDP for the hop to the log collector, either from HAProxy’s log forwarding or from the local syslog daemon onward. TCP gives you backpressure and retransmission instead of fire-and-forget. TCP gives the transport the backpressure and retransmission semantics UDP lacks; the exact loss numbers under load depend on your own pipeline, so measure them with your ring/sink configuration rather than assuming a figure. The tradeoff is that backpressure now has somewhere to propagate; make sure the receiving end can genuinely keep up.

Reduce log volume

Sometimes the honest fix is emitting fewer or smaller lines: drop health-check and monitoring-probe traffic from logs with dontlognull or ACL-based no log rules, trim sample fetches from a bloated custom log-format, and stop logging at debug/info globally. Every byte you do not emit is buffer you do not need.

Prevention

• Alert on the delta. DroppedLogs should be zero forever. A delta greater than zero over any 5-minute window opens a ticket. This is cheap and catches the failure before the incident where you needed the logs.
• Watch the receiver, not just the sender. HAProxy’s counter cannot see network loss or receiver-side socket overflows. Monitor the syslog daemon’s discard counters and the kernel’s UDP receive buffer errors on the receiving host, or your zero-drop signal is incomplete.
• Load-test the log path, not just the proxy. When you capacity-test HAProxy, verify the log pipeline at the same line rate. A log sink that handles average traffic but not 3x peak is a time bomb.
• Treat log verbosity changes as capacity changes. A config diff that adds fields to log-format or removes a dontlog rule changes bytes per second through the pipeline. Review it like a traffic change.
• Survive syslog restarts. Prefer a log path that buffers across a syslog restart (local ring, persistent queue) so routine maintenance does not punch holes in your records.

How Netdata helps

• Netdata’s HAProxy collector reads show info continuously, so DroppedLogs becomes a per-second time series instead of a number you remember to check during an incident. A nonzero delta is visible immediately and alertable.
• Request rate, session rate, and connection counts are collected at the same frequency, so you can confirm the classic signature on one dashboard: drop rate tracking SessRate at peak, which separates “pipeline has no headroom” from “pipeline is broken.”
• Counter resets after reloads show up as discontinuities; correlating with Uptime_sec keeps you from misreading a fresh process as a cured problem.
• Netdata also monitors the syslog side of the pipe when the daemon runs on the same host: process liveness, CPU, disk write latency on the log volume, and UDP error counters, which is where most root causes live.
• During an incident, having the drop timeline next to the 5xx and latency timeline tells you quickly whether your log gaps were caused by the same event you are investigating.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy bandwidth (bin/bout): bytes in, bytes out, and capacity planning
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn