Suboptimal File Transfer Speeds

Step 1 — Measure raw network throughput (baseline)

# Start an iperf3 server on the remote host
iperf3 -s -p 5201

# Run a TCP throughput test from the client
iperf3 -c remote-host -t 30 -P 4

# If the network baseline is close to link speed, the bottleneck is not the network

Step 2 — Compare protocol speeds

# Test raw TCP speed (no encryption overhead)
iperf3 -c remote-host -t 30

# Test SFTP speed
time sftp user@remote-host:/path/to/large-file /dev/null

# Test SCP speed
time scp user@remote-host:/path/to/large-file /dev/null

# If SFTP/SCP are much slower than iperf3, encryption CPU is likely the bottleneck

Step 3 — Check CPU usage during transfer

# On the server during an active transfer
top -b -n 5 | grep sshd

# Check if AES-NI hardware acceleration is available
grep aes /proc/cpuinfo

# If available, check the cipher being negotiated
ssh -vvv user@remote-host 2>&1 | grep -i "cipher"

Step 4 — Measure disk I/O (eliminate storage bottleneck)

# Test local read speed on the source
dd if=/path/to/file of=/dev/null bs=1M count=1024

# Test local write speed on the destination
dd if=/dev/zero of=/tmp/testfile bs=1M count=1024 conv=fdatasync

# Check for disk I/O wait during transfer
iostat -x 1 10

Step 5 — Check TCP window size and buffer settings

# Check current TCP buffer settings
sysctl net.core.rmem_max net.core.wmem_max net.ipv4.tcp_rmem net.ipv4.tcp_wmem

# Check the window size on an active connection
ss -tn | grep remote-host

Step 6 — Test with rsync for comparison

# rsync with compression (good for text files, bad for binary)
rsync -avz --progress user@remote-host:/path/to/ /local/dest/

# rsync without compression (better for already-compressed files)
rsync -av --progress user@remote-host:/path/to/ /local/dest/

Fix 1 — Use a faster SSH cipher

Switch to AES-128-GCM or AES-256-GCM which uses hardware acceleration on modern CPUs:

# Specify a faster cipher explicitly
sftp -o [email protected] user@remote-host

# Or update ~/.ssh/config to always use it
echo "Ciphers [email protected],[email protected],[email protected]" \
  >> ~/.ssh/config

Fix 2 — Tune TCP buffer sizes for high-throughput links

Edit /etc/sysctl.conf on both client and server:

# Increase TCP socket buffer limits (example for a 10 Gbps link with 10ms RTT)
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem='4096 87380 134217728'
sudo sysctl -w net.ipv4.tcp_wmem='4096 65536 134217728'
sudo sysctl -w net.ipv4.tcp_window_scaling=1

# Make persistent
echo "net.core.rmem_max = 134217728" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.tcp_rmem = 4096 87380 134217728" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Fix 3 — Use multiple parallel streams

For large transfers, use parallel connections to overcome single-stream TCP limits:

# Transfer multiple files in parallel with rsync
rsync -av --progress --compress-level=0 \
  user@remote-host:/source/ /dest/ &
# Or use parallel with sftp
echo "get large_file" | sftp -b - user@remote-host &

# Use bbcp for high-performance parallel transfers
bbcp -P 4 -s 8 user@remote-host:/path/to/file /local/dest/

Fix 4 — Use rsync instead of SFTP for bulk transfers

# rsync over SSH with no compression (for binary/compressed files)
rsync -av --no-compress --progress \
  -e "ssh -c [email protected]" \
  user@remote-host:/source/ /dest/

Fix 5 — Use FTP or HTTP for non-sensitive transfers

If security is not a requirement, HTTP (via nginx) or FTP removes encryption overhead entirely:

# Serve files via nginx for fast download (no encryption CPU cost)
# nginx.conf snippet:
# location /files/ {
#     root /var/www/;
#     autoindex on;
# }
wget -O /dev/null http://remote-host/files/largefile.bin

• Verify that AES-NI hardware acceleration is available and enabled before deploying SSH-based file transfer solutions at scale
• Set appropriate TCP buffer sizes during initial server provisioning; add them to a sysctl baseline template
• For high-volume file transfers, evaluate dedicated transfer tools (rsync, bbcp, GridFTP) rather than ad-hoc SFTP
• Monitor transfer speeds in CI/CD pipelines that involve artifact downloads and alert when they drop below a baseline
• When provisioning storage, use SSDs or NVMe to ensure disk I/O is not the bottleneck for network-speed transfers