IPFYI
🔧 Network Troubleshooting 10 dk okuma

Firewall Blocking Legitimate Traffic: Debug Guide

Debug firewall rules that block legitimate traffic using iptables logs, AWS Security Group audits, tcpdump captures, and testing tools like telnet and nc.

The Challenge of Firewall Debugging

Firewalls are silent by default. A packet dropped by a firewall rule produces no error on the client — the connection simply times out. This makes firewall issues hard to distinguish from server crashes, routing failures, or application problems. Systematic testing eliminates the other causes and narrows you down to a firewall issue.

Identify Blocked Traffic with Logs

The first step is enabling logging on the firewall to see what it is dropping.

iptables Logging

# Add a logging rule BEFORE your DROP rules
sudo iptables -I INPUT 1 -j LOG --log-prefix "IPTABLES-DROP: " --log-level 4

# Or log only drops (add at the end, after all ACCEPT rules)
sudo iptables -A INPUT -j LOG --log-prefix "IPTABLES-DROP: " --log-level 4
sudo iptables -A OUTPUT -j LOG --log-prefix "IPTABLES-DROP: " --log-level 4
sudo iptables -A FORWARD -j LOG --log-prefix "IPTABLES-DROP: " --log-level 4

# View the logs
sudo dmesg | grep "IPTABLES-DROP"
sudo journalctl -k | grep "IPTABLES-DROP"
sudo tail -f /var/log/kern.log | grep "IPTABLES-DROP"

Example log entry:

IPTABLES-DROP: IN=eth0 OUT= SRC=203.0.113.50 DST=10.0.0.1
PROTO=TCP SPT=52412 DPT=8080 FLAGS=SYN WINDOW=65535

This tells you: incoming TCP SYN from 203.0.113.50 to port 8080 was dropped.

nftables Logging

# In nftables: add log statement to a chain
sudo nft add rule inet filter input tcp dport 8080 log prefix "NFTABLES-DROP: " drop

# View nftables rules
sudo nft list ruleset

# Monitor logs
sudo journalctl -k -f | grep "NFTABLES"

UFW (Ubuntu's iptables frontend)

# UFW logging
sudo ufw logging on
sudo ufw logging medium   # or 'high' for more detail

# View UFW logs
sudo tail -f /var/log/ufw.log

# Example UFW log entry:
# UFW BLOCK IN=eth0 SRC=203.0.113.50 DST=10.0.0.1 PROTO=TCP DPT=8080

iptables/nftables Rule Analysis

Read existing rules to understand what is being blocked.

# List iptables rules with line numbers and packet counts
sudo iptables -L -n -v --line-numbers
sudo iptables -L INPUT -n -v --line-numbers
sudo iptables -L FORWARD -n -v --line-numbers

# List NAT table
sudo iptables -t nat -L -n -v

# Save current rules
sudo iptables-save > /tmp/iptables-backup.txt

# nftables: full ruleset
sudo nft list ruleset

# nftables: specific table
sudo nft list table inet filter

Reading the output:

Chain INPUT (policy DROP)                    <-- Default DROP policy
num   pkts bytes target     prot opt in     out     source               destination
1       45  3240 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0    tcp dpt:22
2        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0    tcp dpt:80

The pkts counter shows how many packets matched each rule. A rule with zero packets either has never matched or is unreachable because a preceding rule catches all relevant traffic.

# Reset packet counters to start fresh monitoring
sudo iptables -Z

# Then trigger the traffic and check which rule increments
sudo iptables -L INPUT -n -v

Security Group Audit (AWS)

AWS Security Groups are stateful firewalls attached to EC2 instances. Unlike iptables, Security Groups only have ALLOW rules (no explicit DENY); anything not matched is implicitly denied.

# List Security Groups for an instance
INSTANCE_ID="i-0123456789abcdef0"
aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[].Instances[].SecurityGroups'

# Get inbound rules for a Security Group
SG_ID="sg-0123456789abcdef0"
aws ec2 describe-security-groups --group-ids $SG_ID \
  --query 'SecurityGroups[].IpPermissions'

# Check VPC Network ACLs (stateless, applied before Security Groups)
aws ec2 describe-network-acls --filters "Name=vpc-id,Values=vpc-xxx"

Common Security Group mistakes:

Mistake Effect Fix
Port range wrong (e.g., 4430 instead of 443) HTTPS blocked Correct the port number
Source IP not updated after IP change New IP blocked Update source CIDR
Rule on wrong interface Traffic allowed on eth0, blocked on eth1 Check which SG is on which network interface
NACLs blocking what SG allows Traffic dropped at subnet level NACLs are stateless — add both inbound AND outbound rules 
# Use AWS VPC Reachability Analyzer for automatic path analysis
aws ec2-instance-connect send-ssh-public-key --region us-east-1 ...

# Or use the console: VPC → Reachability Analyzer → Create analysis
# Specify source ENI and destination, port, and protocol
# The analysis identifies exactly which rule in which security group blocks traffic

Stateful vs Stateless Rules

Understanding the difference prevents a common misconfiguration:

Stateful firewalls (iptables conntrack, AWS Security Groups) automatically allow return traffic for established connections. You only need to allow the inbound direction.

Stateless firewalls (AWS Network ACLs, traditional ACLs on routers) require explicit rules for both directions: the initial request AND the return packets.

# iptables: ensure ESTABLISHED traffic is allowed (stateful)
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Without this rule, return packets for outbound connections get dropped

# AWS NACL stateless example:
# Inbound rule: allow TCP 80 from 0.0.0.0/0
# Outbound rule: allow TCP 1024-65535 to 0.0.0.0/0  <-- required for return traffic
# The ephemeral port range (1024-65535) is where the client's source port resides

Debugging with tcpdump

tcpdump shows packets at the network interface level, BEFORE firewall rules are applied (inbound) or AFTER (outbound). This lets you confirm a packet is arriving and whether it leaves the interface.

# Capture all traffic on port 8080
sudo tcpdump -i eth0 port 8080

# More verbose output
sudo tcpdump -i eth0 -nn port 8080 -v

# Capture SYN packets only (new connection attempts)
sudo tcpdump -i eth0 "tcp[tcpflags] & tcp-syn != 0" and port 8080

# Capture and save to file for analysis
sudo tcpdump -i eth0 port 8080 -w /tmp/capture.pcap

# If SYNs arrive but no SYN-ACK goes out, a firewall is dropping the packet
# If SYNs never arrive, the problem is upstream (routing or remote firewall)

Common Misconfigurations

Misconfiguration Symptom Fix
DROP default policy with no ESTABLISHED rule Existing connections drop after rule reload Add ESTABLISHED,RELATED accept rule
Rules in wrong order Expected ACCEPT overridden by earlier DROP Move ACCEPT rules before DROP
Masquerade/SNAT missing for routed traffic Traffic arrives, no reply reaches client Add MASQUERADE or SNAT in POSTROUTING
Interface name changed (eth0 → ens3) Rules written for eth0 no longer apply Update rules with correct interface name
IPv6 not configured IPv6-native clients can't connect Configure ip6tables separately

Testing with telnet and nc

After adjusting firewall rules, always verify with a direct connection test before assuming the fix works.

# Test TCP port connectivity
telnet 10.0.0.1 8080
# "Connected to 10.0.0.1" = firewall allows it
# "Connection refused" = firewall allows, but nothing listening
# Timeout = firewall blocking

# nc (netcat) — preferred, works for both TCP and UDP
nc -zv 10.0.0.1 8080          # TCP test
nc -zvu 10.0.0.1 53            # UDP test (DNS)

# Test from the server side: listen on a port
nc -l -p 8080                  # server
nc -v server_ip 8080           # client

# Scan multiple ports quickly
nc -zv 10.0.0.1 80 443 8080 8443

# curl for HTTP-specific testing
curl -v --max-time 5 http://10.0.0.1:8080/

Always test from the same network context as the failing client — a rule that allows traffic from within the VPC may not allow external traffic.