IPsec Phase Negotiation Failure (IKEv1/IKEv2)

Step 1: Check IKE Status

# Strongswan (IKEv2)
sudo ipsec statusall
sudo swanctl --list-sas        # swanctl-based configs
sudo swanctl --list-conns

# Libreswan
sudo ipsec whack --status
sudo ipsec auto --status

# Check for active SAs:
sudo ip xfrm state list         # Shows active IPsec SAs (kernel-level)
sudo ip xfrm policy list        # Shows traffic selectors (policies)

Step 2: Enable Verbose Logging

# Strongswan: increase log level in /etc/strongswan.conf or /etc/strongswan.d/charon-logging.conf
# charon {
#   filelog {
#     /var/log/charon.log {
#       time_format = %b %e %T
#       ike = 2
#       knl = 2
#       cfg = 2
#     }
#   }
# }

sudo systemctl restart strongswan-starter
sudo tail -f /var/log/charon.log

# Libreswan
sudo ipsec pluto --stderrlog --debug-all 2>&1 | head -100

Step 3: Decode the Error

Common errors and their meaning:

Step 4: Capture and Analyse IKE Traffic

# Capture IKE packets (UDP 500 and 4500 for NAT-T)
sudo tcpdump -n -i eth0 -w /tmp/ike.pcap '(udp port 500 or udp port 4500)'

# Open in Wireshark with: Analyze > Decode As > IKE
# Look for:
# - Exchange Type (IKE_SA_INIT, IKE_AUTH for IKEv2; Main Mode, Aggressive Mode for IKEv1)
# - NOTIFY payloads containing error codes

Step 5: Compare Proposals on Both Peers

# Strongswan — show configured proposals
grep -r "proposals\|ike\|esp" /etc/ipsec.conf /etc/ipsec.d/ /etc/swanctl/conf.d/

# Verify both sides use identical:
# - IKE version (1 or 2)
# - Encryption: aes128/aes256/3des
# - Integrity/PRF: sha1/sha256/sha384
# - DH group: modp1024/modp2048/ecp256
# - Lifetime values (optional but useful to match)

Fix: Align Phase 1 and Phase 2 Proposals

Edit /etc/ipsec.conf (Libreswan) or /etc/swanctl/conf.d/tunnel.conf (Strongswan swanctl) to match the remote peer's capabilities exactly:

# Libreswan /etc/ipsec.conf example
conn site-to-site
    left=203.0.113.1
    leftsubnet=10.10.0.0/24
    right=198.51.100.1
    rightsubnet=10.20.0.0/24
    authby=secret
    ike=aes256-sha256;modp2048   # Must match remote peer
    esp=aes256-sha256            # Must match remote peer
    ikelifetime=8h
    keylife=1h
    auto=start

# Restart and re-initiate
sudo ipsec restart
sudo ipsec auto --up site-to-site

Fix: Correct Traffic Selectors

Ensure both peers specify identical leftsubnet/rightsubnet values (or use narrowing if supported):

# Both peers must agree:
leftsubnet=10.10.0.0/24
rightsubnet=10.20.0.0/24

# IKEv2 with strongswan: enable TS narrowing if using dynamic addresses
connections {
  my-tunnel {
    children {
      net {
        local_ts  = 10.10.0.0/24
        remote_ts = 10.20.0.0/24
      }
    }
  }
}

Fix: Verify Pre-Shared Key

# Check the PSK on both ends
sudo cat /etc/ipsec.secrets
# Format: 203.0.113.1 198.51.100.1 : PSK "your-secret-key"
# The key must be byte-for-byte identical on both peers
# Use quotes and escape special characters: $ \ " must be preceded by backslash

Fix: Allow NAT-T Through the Firewall

sudo iptables -A INPUT -p udp --dport 500 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 4500 -j ACCEPT
sudo iptables -A INPUT -p esp -j ACCEPT
sudo iptables -A INPUT -p ah -j ACCEPT

• Maintain a shared proposal matrix document agreed upon by both network teams before tunnel deployment
• Use IKEv2 with strong modern ciphers (AES-256-GCM, SHA-384, ECC P-384) and refuse 3DES, MD5, and DH group 1/2
• Script the configuration with version control so both ends are always symmetrical and diff-auditable
• Enable persistent logging at level 1 for IKE and ESP; review logs after every maintenance window
• Test tunnel re-establishment after planned rekeying to confirm the proposal still matches after key rollover