VPN Not Connecting: Troubleshooting All Protocols

Understanding VPN Connection Failures

VPN failures can be deceptively hard to diagnose because the error message is often the same ("connection timed out" or "authentication failed") regardless of whether the root cause is a firewall rule, a configuration typo, an expired certificate, or a kernel module that failed to load. A systematic approach by protocol saves hours of guesswork.

WireGuard Debugging

WireGuard is intentionally minimalist — it produces almost no diagnostic output when things go wrong. Use the kernel's built-in debug interface.

# Enable WireGuard debug logging (Linux)
sudo modprobe wireguard
echo module wireguard +p | sudo tee /sys/kernel/debug/dynamic_debug/control

# View live debug output
sudo dmesg -wH | grep -i wireguard

# Check interface status
sudo wg show

The wg show output is the most important diagnostic tool:

interface: wg0
  public key: abcdef1234...
  listening port: 51820

peer: xyz789...
  endpoint: 198.51.100.1:51820
  allowed ips: 10.8.0.0/24
  latest handshake: 2 minutes, 14 seconds ago   <-- good
  transfer: 14.52 KiB received, 6.73 KiB sent

If latest handshake is missing or stale (more than 3 minutes), the tunnel is not passing traffic. Common causes:

# Test if UDP port is reachable from client
nc -u -v 198.51.100.1 51820

# Check if IP forwarding is enabled on server
cat /proc/sys/net/ipv4/ip_forward
# Must be 1; enable with:
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo sysctl -w net.ipv4.ip_forward=1

OpenVPN Log Analysis

OpenVPN is verbose by default. Increase verbosity for troubleshooting:

# Add to OpenVPN config or pass via command line
verb 4

# Tail the log
sudo journalctl -u openvpn@client -f
sudo tail -f /var/log/openvpn/client.log

Key log messages and their meaning:

TLS Error: TLS handshake failed

→ Certificate mismatch, expired CA, or time skew. Check date on both ends.

AUTH_FAILED

→ Wrong credentials or revoked certificate. Check CRL and user account.

RESOLVE: Cannot resolve host address

→ DNS failure for VPN server hostname. Try connecting with IP directly.

NOTE: --mute triggered...
Connection reset, restarting

→ Persistent reconnections usually mean a routing loop. Check redirect-gateway.

# Verify certificate validity
openssl verify -CAfile ca.crt client.crt
openssl x509 -in client.crt -noout -dates

# Check if TUN/TAP module is loaded
lsmod | grep tun
sudo modprobe tun

IPsec/IKEv2 Issues

IPsec is the most complex VPN protocol to debug. On Linux, use ip xfrm and strongswan or libreswan logs.

# StrongSwan status
sudo swanctl --list-conns
sudo swanctl --list-sas

# StrongSwan logs (systemd)
sudo journalctl -u strongswan -n 100

# Show IPsec SAs (Security Associations)
sudo ip xfrm state list
sudo ip xfrm policy list

# Libreswan
sudo ipsec status
sudo ipsec whack --trafficstatus

Common IKEv2 errors:

Firewall and NAT Traversal

Many connection failures are simply firewall rules blocking VPN traffic.

# Required ports by protocol
# WireGuard: UDP 51820 (configurable)
# OpenVPN:   UDP 1194 or TCP 443 (configurable)
# IPsec:     UDP 500 (IKE) + UDP 4500 (NAT-T) + ESP (protocol 50)

# Test from client
nmap -sU -p 51820 198.51.100.1    # WireGuard
nmap -sU -p 1194 198.51.100.1     # OpenVPN UDP
nmap -p 443 198.51.100.1          # OpenVPN TCP

# Check iptables on server
sudo iptables -L -n -v
sudo iptables -t nat -L -n -v

# Add missing MASQUERADE rule (required for traffic routing)
sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

For double-NAT environments (client behind NAT, server behind NAT), WireGuard and OpenVPN handle this well, but IPsec requires NAT Traversal (UDP encapsulation over port 4500).

DNS Leak After Connect

A DNS leak means your DNS queries bypass the VPN tunnel, revealing your real location and browsing activity.

# Test for DNS leaks
# Visit https://dnsleaktest.com or use command line:
dig +short myip.opendns.com @208.67.222.222

# On Linux: check resolv.conf after VPN connects
cat /etc/resolv.conf
# Should show VPN's DNS server, not your ISP's

# Check systemd-resolved
resolvectl status

# Force DNS through VPN (Linux, NetworkManager)
# Set DNS server in VPN connection profile
nmcli connection show "VPN-name" | grep dns

For OpenVPN, add to server config:

push "dhcp-option DNS 10.8.0.1"
push "redirect-gateway def1 bypass-dhcp"

For WireGuard, add to [Interface]:

DNS = 10.8.0.1

Split Tunneling Configuration

Split tunneling routes only specific traffic through the VPN, keeping other traffic on the local network. This improves performance but requires careful AllowedIPs configuration.

# WireGuard: route only specific subnets through VPN
[Peer]
AllowedIPs = 10.0.0.0/8, 172.16.0.0/12, 192.168.100.0/24
# Do NOT include 0.0.0.0/0 for split tunnel

# Route ALL traffic through VPN
AllowedIPs = 0.0.0.0/0, ::/0

# Exclude specific subnets from full-tunnel VPN
# Use the "Allowed IPs Calculator" to compute inverse routes:
# https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/

For OpenVPN split tunnel:

# Remove redirect-gateway from server push
# Client config: add specific routes
route 10.0.0.0 255.0.0.0
route 172.16.0.0 255.240.0.0

MTU and Fragmentation

VPN tunnels add overhead (WireGuard adds ~60 bytes, OpenVPN adds ~30-80 bytes depending on cipher). If the resulting packets exceed the path MTU, they get fragmented or silently dropped.

# Find the effective MTU for VPN traffic
# Send ping with progressively larger sizes until it fails
ping -M do -s 1400 198.51.100.1
ping -M do -s 1420 198.51.100.1
ping -M do -s 1440 198.51.100.1
# "Frag needed" error indicates the size is too large

# Recommended MTU values
# WireGuard interface: 1420 (accounts for WireGuard overhead)
# OpenVPN tun: 1400-1420

# Set MTU on WireGuard interface
[Interface]
MTU = 1420

# Set MTU on Linux interface directly
sudo ip link set wg0 mtu 1420

# Fix TCP MSS clamping to prevent large TCP segments
sudo iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

Symptoms of MTU issues: large file transfers stall, HTTPS pages load partially, SSH connections drop after the initial banner. These are hallmarks of path MTU black holes.