Secure network con­nec­tions with IPsec

IP packets, the fun­da­ment­al unit of internet data transfer, consist of two parts: the actual payload (voice, text, images) and header data con­tain­ing sender and recipient addresses. The core problem: IP itself provides no con­fid­en­ti­al­ity or au­then­tic­a­tion at the network layer – re­gard­less of any ap­plic­a­tion-level en­cryp­tion the payload may already carry. Without ad­di­tion­al mech­an­isms, packets can be read or ma­nip­u­lated in transit, leaving three security prop­er­ties un­pro­tec­ted: con­fid­en­ti­al­ity, integrity, and au­then­tic­a­tion (the security services defined in the IPsec ar­chi­tec­ture, RFC 4301).

IPsec (Internet Protocol Security) was developed to close this gap. The protocol suite operates at the network layer and secures IP com­mu­nic­a­tions in­de­pend­ently of the ap­plic­a­tion being used – which is why it remains the backbone of most site-to-site VPN con­nec­tions today.

What is IPsec?

IPsec is a protocol suite stand­ard­ized by the Internet En­gin­eer­ing Task Force (IETF). Early IPsec spe­cific­a­tions (from RFC 1825, 1995, onward) addressed both IPv4 and IPv6. The key dis­tinc­tion today is that IPsec support is mandatory for IPv6 (RFC 6434) and optional for IPv4, where it is deployed as an add-on. In practice, IPsec is widely used in both protocol versions. The suite divides into three func­tion­al groups:

• Transfer protocols: Au­then­tic­a­tion Header (AH, RFC 4302), En­cap­su­lat­ing Security Payload (ESP, RFC 4303)
• Key man­age­ment: Internet Key Exchange v2 (IKEv2, RFC 7296); ISAKMP (RFC 2408, now Historic) is a legacy pre­de­cessor used with IKEv1
• Databases: Security As­so­ci­ation Database (SAD), Security Policy Database (SPD) – defined in the IPsec ar­chi­tec­ture RFC 4301

Com­pon­ents of the protocol suite

The two main transfer protocols differ sig­ni­fic­antly in what they protect and how:

ESP (En­cap­su­lat­ing Security Payload) is the standard for all modern IPsec de­ploy­ments. It provides en­cryp­tion, au­then­tic­a­tion, and integrity pro­tec­tion in one header. When used with an AEAD cipher like AES-GCM, ESP handles en­cryp­tion and au­then­tic­a­tion in a single pass – no separate HMAC cal­cu­la­tion is needed.

AH (Au­then­tic­a­tion Header) provides integrity and origin au­then­tic­a­tion only – but no en­cryp­tion. More crit­ic­ally, AH au­then­tic­ates the entire packet including the outer IP header, making it in­com­pat­ible with NAT – a problem in virtually every modern network. ESP with au­then­tic­a­tion achieves equi­val­ent pro­tec­tion and works through NAT. Current BSI (TR-02102-3) and NIST guidelines recommend ESP ex­clus­ively; AH is con­sidered ef­fect­ively obsolete.

ESP protects data through four mech­an­isms:

• Con­fid­en­ti­al­ity – payload content is encrypted
• Integrity – any tampering in transit is detected
• Origin au­then­tic­a­tion – confirms the packet came from the expected peer
• Replay pro­tec­tion – each packet carries a mono­ton­ic­ally in­creas­ing sequence number; the receiver maintains an anti-replay window and rejects any packet whose number falls outside the window or has already been processed

SAD and SPD

The Security Policy Database (SPD) defines which traffic should be protected and with which protocol. The Security As­so­ci­ation Database (SAD) holds the active session para­met­ers ne­go­ti­ated by IKEv2: for each SA, the SAD stores the Security Parameter Index (SPI), the chosen al­gorithms, and the as­so­ci­ated key material. Each SA is uni­direc­tion­al – a single protected con­nec­tion therefore requires two SAs, one per direction, each with its own SPI and keys.

IKEv1 vs. IKEv2

IKEv2, stand­ard­ized in RFC 7296 (2014), is the current key man­age­ment standard for IPsec. IKEv1 (RFC 2409, 1998) is de­prec­ated and should not be used in new de­ploy­ments – it has known cryp­to­graph­ic weak­nesses, a complex handshake, and lacks features standard in modern im­ple­ment­a­tions. If your VPN gateway still defaults to IKEv1, update the con­fig­ur­a­tion.

IKEv2 re­designed the handshake from the ground up and adds several critical cap­ab­il­it­ies over IKEv1:

• Simpler handshake: always 4 messages (IKEv1 Main Mode required 6–9)
• Built-in NAT traversal: automatic ESP-in-UDP en­cap­su­la­tion on port 4500 when NAT is detected
• MOBIKE (RFC 4555): allows a tunnel to survive an IP address change without rene­go­ti­at­ing – essential for devices switching between Wi-Fi and cellular networks
• EAP au­then­tic­a­tion: native support for cer­ti­fic­ates, pre-shared keys, and EAP
• Dead Peer Detection: built in via IN­FORM­A­TION­AL exchanges (IKEv1 required a separate extension)
• Perfect Forward Secrecy: con­fig­ur­able per child SA – ensures past sessions cannot be decrypted even if a long-term key is later com­prom­ised

The two types of IPsec: Tunnel vs. Transport

There are two existing transfer modes for secure con­nec­tions with IPsec: transport mode, in which two end hosts are connected directly, and tunnel mode, which creates a con­nec­tion between two IP networks.

Transport mode

In transport mode, IPsec inserts its headers between the existing IP header and the payload. The original IP header – including source and des­tin­a­tion addresses – remains visible. Only the payload is protected. This mode is efficient and fast, but does not conceal the com­mu­nic­at­ing endpoints. The cryp­to­graph­ic and com­mu­nic­at­ive terminal points are identical.

Typical use: host-to-host con­nec­tions, network man­age­ment traffic, and as the inner tunnel in L2TP/IPsec con­fig­ur­a­tions for remote access clients.

Tunnel mode

In tunnel mode, the entire original IP packet is en­cap­su­lated inside a new packet with a new outer IP header. Both the payload and the original header (with the real source and des­tin­a­tion addresses) are hidden inside the encrypted ESP payload – only the gateway addresses are visible in transit.

Typical use: site-to-site VPNs (gateway to gateway) and road warrior VPN clients where endpoint identity must be concealed. When IKEv2 detects NAT in the path during its handshake, it auto­mat­ic­ally en­cap­su­lates ESP inside UDP on port 4500, allowing IPsec tunnels to pass through routers and firewalls that would otherwise drop raw ESP packets (IP protocol 50).

Cipher suites and security re­com­mend­a­tions

Choosing the right al­gorithms is as important as deploying IPsec correctly. The following is aligned with BSI Technical Guideline TR-02102-3 (2024) and current NIST re­com­mend­a­tions.

En­cryp­tion (ESP):

• AES-256-GCM – Re­com­men­ded. Au­then­tic­ated en­cryp­tion (AEAD) – no separate HMAC needed. Hardware-ac­cel­er­ated via AES-NI on modern CPUs.
• AES-128-GCM – Ac­cept­able for most de­ploy­ments.
• AES-256-CBC – Ac­cept­able if combined with HMAC-SHA-256 or SHA-384. Avoid without separate au­then­tic­a­tion.
• 3DES / DES – Do not use. De­prec­ated by NIST (2023). Vul­ner­able to the Sweet32 attack (3DES) or trivially broken (DES).

Integrity / Au­then­tic­a­tion (for non-AEAD ciphers):

• HMAC-SHA-256 – Minimum re­com­mend­a­tion.
• HMAC-SHA-384 / SHA-512 – Preferred for high-security en­vir­on­ments.
• HMAC-MD5 / HMAC-SHA-1 – Do not use.

Key exchange – IKEv2 Diffie-Hellman groups:

• Group 19 (P-256 / ECDH) – Re­com­men­ded default.
• Group 20 / 21 (P-384 / P-521) – Higher security margin for sensitive de­ploy­ments.
• Group 14 (2048-bit MODP) – Minimum ac­cept­able; only for legacy com­pat­ib­il­ity.
• Groups 1, 2, 5 (768–1536-bit) – Do not use. Con­sidered broken.

Looking ahead: Post-quantum cryp­to­graphy

Quantum computers capable of breaking current asym­met­ric key exchange are not yet op­er­a­tion­al, but the IETF is already preparing the trans­ition. RFC 9370 (2023) defines multiple key exchanges in IKEv2, enabling hybrid classical/post-quantum key agreement. NIST finalized its first post-quantum algorithm standards in 2024 (FIPS 203, 204, 205). High-security and long-lifecycle de­ploy­ments should monitor BSI and NIST guidance on migration timelines.

IPsec: strengths and weak­nesses

When using VPNs – the most common ap­plic­a­tion area of the protocol suite – IPsec operates in­de­pend­ently from any ap­plic­a­tion at network level. Once the con­nec­tion is made, different forms of data (e-mail, file transfer, IP telephony) can be sent without ap­plic­a­tion-specific tools. It scales to thousands of sim­ul­tan­eous tunnels, is supported natively by major operating systems, and benefits from wide­spread hardware ac­cel­er­a­tion (AES-NI).

The trade-offs: IPsec is complex to configure correctly, and mis­con­fig­ur­a­tion is a genuine risk. Its large codebase is difficult to audit fully. Devices that still ship with IKEv1 or weak ciphers enabled by default require manual hardening.

Com­par­is­on with modern al­tern­at­ives:

• IPsec / IKEv2 (network layer): best for site-to-site VPNs, en­ter­prise en­vir­on­ments, and mobile clients – mature, scales well, complex con­fig­ur­a­tion.
• SSL / TLS VPN (ap­plic­a­tion layer): best for remote access and cli­ent­less browser-based scenarios – easier firewall traversal, per-app tunneling.
• WireGuard (network layer): a modern al­tern­at­ive in­teg­rated into the Linux kernel since version 5.6 (2020). Its minimal codebase (~4,000 lines) makes it sig­ni­fic­antly easier to audit and it con­sist­ently out­per­forms IPsec in bench­marks on con­strained hardware. WireGuard uses a fixed, modern cipher suite (ChaCha20-Poly1305, Curve25519) and is worth eval­u­at­ing for green­field Linux de­ploy­ments. It lacks the en­ter­prise feature set and vendor in­ter­op­er­ab­il­ity that IPsec provides.

Use cases

Site-to-site VPN: The most common IPsec de­ploy­ment. Two network gateways establish a permanent tunnel in tunnel mode, en­crypt­ing all traffic between two office or data­cen­ter networks. IKEv2 handles key exchange and re-keying; ESP with AES-256-GCM encrypts the payload. Dead Peer Detection auto­mat­ic­ally re-es­tab­lishes the tunnel if a peer becomes un­reach­able.

Remote access (road warrior): IKEv2/IPsec combined with EAP au­then­tic­a­tion allows in­di­vidu­al users to connect securely from laptops and mobile devices. MOBIKE ensures the tunnel persists when the device’s IP address changes – for example, when switching from a hotel Wi-Fi to a cellular con­nec­tion. Windows, macOS, iOS, and Android all include a native IKEv2 client, requiring no third-party software in­stall­a­tion.

Host-to-host (transport mode): Used for securing traffic between specific servers – database rep­lic­a­tion, inter-data­cen­ter syn­chron­iz­a­tion, or man­age­ment plane com­mu­nic­a­tion – without routing through a gateway. Transport mode adds minimal overhead but requires IPsec policy con­fig­ur­a­tion on both endpoints.