IPFYI

Bufferbloat Causing Latency Under Load

Intermediate Performance

Latency increases dramatically when the network link is saturated with bulk transfers, making interactive applications (VoIP, gaming, web browsing, SSH) nearly unusable during downloads or uploads. When the bulk transfer stops, latency immediately returns to normal. This is the classic bufferbloat problem caused by oversized buffers in routers, modems, or switches that fill with data before the network device applies back-pressure.

Symptoms

  • ⚠ Ping latency jumps from 10ms to 500ms or more when a large download or upload is in progress
  • ⚠ VoIP calls or video conferences degrade severely when any other device on the network is transferring data
  • ⚠ SSH sessions become unresponsive and keystrokes are delayed during bulk transfers
  • ⚠ Latency returns immediately to baseline as soon as the bulk transfer is paused or stopped
  • ⚠ DSLReports or Waveform bufferbloat test shows a grade of B or worse
  • ⚠ Traceroute or ping to the gateway shows elevated RTT even before traffic leaves the local network

Possible Root Causes

  • • Large FIFO queues in the router or modem allowing many seconds of data to accumulate before applying back-pressure, introducing latency proportional to buffer depth
  • • pfifo_fast or FIFO queue disciplines on network interfaces that have no mechanism to shed excess packets or reduce latency under load
  • • Large NIC ring buffer (ethtool -g) holding many frames before the kernel sees them, adding buffering at the driver level
  • • DSL or cable modem with a fixed deep hardware queue that cannot be disabled or tuned via software
  • • Upload saturation causing gateway ARP or TCP ACK packets to queue behind bulk data on the uplink, delaying acknowledgements and increasing download latency as well

Diagnosis Steps

Step 1 — Confirm bufferbloat with a baseline measurement

# Measure idle latency to your gateway
ping -c 10 192.168.1.1

# In another terminal, start a large download to saturate the link
curl -o /dev/null https://speed.hetzner.de/10GB.bin &

# Now measure latency again while the download runs
ping -c 30 192.168.1.1
# Compare: if RTT jumps from <5ms to >200ms, bufferbloat is confirmed

Step 2 — Measure bufferbloat systematically

# Install flent (Flexible Network Tester) for standardised bufferbloat tests
pip install flent

# Run RRUL (Realtime Response Under Load) test — the standard bufferbloat benchmark
flent rrul -p all_scaled -l 60 -H remote-host.com

# Or use netperf-based test
flent tcp_upload -p all_scaled -l 60 -H remote-host.com

Step 3 — Identify where the buffer is

# Check the local interface queue length and policy
tc qdisc show dev eth0        # Local interface (usually pfifo_fast or fq_codel)
tc qdisc show dev wlan0       # WiFi interface

# For PPPoE connections (typical DSL), check the PPP interface
tc qdisc show dev ppp0
ip link show ppp0 | grep qlen

Step 4 — Check driver and NIC ring buffer sizes

# Large NIC ring buffers contribute to bufferbloat at the driver level
ethtool -g eth0
# Look for: RX and TX current values — reduce these if they are very large (>256)

Step 5 — Test with active queue management (AQM)

# Temporarily apply fq_codel to the interface to test if it resolves the issue
sudo tc qdisc replace dev eth0 root fq_codel

# Repeat the ping-under-load test
curl -o /dev/null https://speed.hetzner.de/10GB.bin &
ping -c 30 192.168.1.1

Step 6 — Check if the router/modem firmware supports CAKE or fq_codel

# On OpenWrt routers, check installed packages
opkg list-installed | grep -i "cake\|sqm\|fq_codel"

# Or check via LuCI GUI: Network > SQM QoS

Solution

Fix 1 — Apply fq_codel (Fair Queuing + Controlled Delay) AQM

fq_codel is the modern Linux solution for bufferbloat. It applies delay-based AQM and fair queuing:

# Replace pfifo_fast with fq_codel on the egress interface
sudo tc qdisc replace dev eth0 root fq_codel

# Verify it is active
tc qdisc show dev eth0
# Expected output: qdisc fq_codel ...

# Make it persistent across reboots (NetworkManager)
sudo nmcli connection modify "Wired connection 1" \
  802-3-ethernet.qdisc "fq_codel"

Fix 2 — Apply CAKE (Common Applications Kept Enhanced)

CAKE is a more advanced successor to fq_codel, available in Linux 4.19+:

# Install kernel module if needed
modprobe sch_cake

# Apply CAKE with the link bandwidth set to your ISP speed (e.g., 100Mbit download)
# Setting bandwidth slightly below the link speed is critical for CAKE to work
sudo tc qdisc replace dev eth0 root cake bandwidth 95mbit diffserv4

# For OpenWrt routers, install SQM-Scripts which configures CAKE automatically
opkg update && opkg install luci-app-sqm

Fix 3 — Reduce NIC ring buffer size

# Check current ring buffer sizes
ethtool -g eth0
# RX: 4096, TX: 4096 is too large for desktop/edge use

# Reduce to lower hardware buffering
sudo ethtool -G eth0 rx 256 tx 256

# Make persistent in /etc/network/interfaces or udev
echo 'SUBSYSTEM=="net", ACTION=="add", KERNEL=="eth0", RUN+="/usr/sbin/ethtool -G eth0 rx 256 tx 256"' \
  | sudo tee /etc/udev/rules.d/99-ring-buffer.rules

Fix 4 — Configure SQM on OpenWrt (for home routers)

If using OpenWrt, SQM-Scripts provides an easy interface for CAKE configuration:

LuCI: Network > SQM QoS
  - Interface: pppoe-wan (or the WAN interface)
  - Download speed: 95% of ISP provisioned download
  - Upload speed: 95% of ISP provisioned upload
  - Queueing discipline: cake
  - Queue setup script: piece_of_cake.qos

Verification

# Re-run the under-load test
curl -o /dev/null https://speed.hetzner.de/10GB.bin &
ping -c 30 192.168.1.1
# Expected: latency should remain close to idle baseline (< 20ms)

Prevention

  • Deploy fq_codel or CAKE on all network interfaces at the edge (router/modem) as a baseline configuration, not as a reactive fix
  • Set the SQM bandwidth limit to 90-95% of the provisioned link speed so the AQM operates on the correct queue, not the modem's hardware buffer
  • Reduce NIC ring buffer sizes on servers and desktops to the minimum that maintains line rate (typically 256-512 frames)
  • Select ISP equipment (cable modem, DSL modem) that supports SQM or AQM in firmware, or use a separate router that can run OpenWrt with CAKE
  • Monitor idle vs. loaded latency with flent or similar tools as part of regular network health checks

Related Protocols

TCP UDP ICMP HTTP HTTP2 RTP SIP

Related Terms

latency packet-loss bandwidth throughput qos tcp isp

More in Performance

High CDN Cache Miss Rate Advanced High Latency to Specific Geographic Region Intermediate Packet Loss During Peak Hours Intermediate Suboptimal File Transfer Speeds Beginner TCP Window Scaling Bottleneck Advanced