Asymmetric Routing with Stateful Firewall Dropping Packets

Step 1: Confirm Asymmetric Paths with Traceroute

# From client to server
traceroute 203.0.113.50

# From server to client (using mtr for continuous view)
mtr --report --report-cycles 10 192.0.2.100

# Compare the paths — they should be symmetric; different paths = asymmetric routing

Step 2: Capture Traffic on Both Interfaces

# On the multi-homed server, capture on all interfaces simultaneously
sudo tcpdump -n -i eth0 host 192.0.2.100 &
sudo tcpdump -n -i eth1 host 192.0.2.100 &

# Look for:
# - Inbound traffic on eth0 (ISP A)
# - Outbound replies on eth1 (ISP B)
# This is asymmetric — the firewall tracks state per-interface and drops the reply

Step 3: Check Stateful Firewall Logs

# iptables: check for INVALID state drops
sudo iptables -L FORWARD -n -v | grep DROP
sudo iptables -L INPUT -n -v | grep DROP

# Enable conntrack logging to see why packets are dropped:
sudo iptables -A INPUT -m conntrack --ctstate INVALID -j LOG --log-prefix "INVALID: "
sudo tail -f /var/log/kern.log | grep INVALID

# nftables
sudo nft list ruleset | grep drop

Step 4: Check Routing Policy (Linux)

# Linux policy routing — show all routing tables
ip rule show
# Expected for dual-NIC: separate rules routing per-interface traffic back out the same interface
# Absence of rules = default table used for all traffic = asymmetric

ip route show table 100  # Custom routing table for eth1
ip route show table 200  # Custom routing table for eth2

Step 5: Verify Connection Tracking State

# Show active conntrack entries
sudo conntrack -L | grep 192.0.2.100

# Packets marked INVALID have no matching conntrack entry — they are not part of a known flow
sudo conntrack -L --output extended | grep INVALID

Fix: Implement Linux Policy Routing (Source-Based Routing)

Create per-interface routing tables that force replies out the same interface the request arrived on:

# /etc/iproute2/rt_tables: add table IDs
echo "100 isp_a" | sudo tee -a /etc/iproute2/rt_tables
echo "200 isp_b" | sudo tee -a /etc/iproute2/rt_tables

# eth0 = ISP-A (203.0.113.50/24, gateway 203.0.113.1)
# eth1 = ISP-B (198.51.100.50/24, gateway 198.51.100.1)

# Table isp_a: all traffic from 203.0.113.50 exits via eth0
sudo ip route add default via 203.0.113.1 dev eth0 table isp_a
sudo ip rule add from 203.0.113.50 lookup isp_a priority 100

# Table isp_b: all traffic from 198.51.100.50 exits via eth1
sudo ip route add default via 198.51.100.1 dev eth1 table isp_b
sudo ip rule add from 198.51.100.50 lookup isp_b priority 200

# Verify:
ip rule show
ip route show table isp_a

Make permanent using /etc/network/interfaces post-up directives or Netplan.

Fix: Disable conntrack INVALID Drop (Temporary Workaround)

# If policy routing cannot be implemented immediately, allow INVALID packets through:
sudo iptables -D INPUT -m conntrack --ctstate INVALID -j DROP

# WARNING: This reduces security — only use as a short-term measure while implementing source routing

Fix: Configure ECMP to Use Per-Flow Hashing

If the asymmetry is caused by ECMP, ensure the hash function is flow-based (5-tuple: src IP, dst IP, src port, dst port, protocol) so forward and return paths use the same hash:

# Linux kernel: ECMP hash policy
sudo sysctl net.ipv4.fib_multipath_hash_policy=1  # 0=L3, 1=L4 (5-tuple)

• Design multi-homed servers with source-based policy routing from day one; document the routing tables and rules in the infrastructure repo
• Test asymmetric routing explicitly during network changes by running mtr in both directions and comparing hop counts
• Use --ctstate ESTABLISHED,RELATED in firewall accept rules and log all INVALID drops during initial deployment
• In cloud environments, rely on the cloud provider's routing guarantees rather than bare-metal multi-NIC policy routing when possible
• Implement network monitoring that alerts when inbound and outbound path MTU or hop counts diverge significantly