Internet Protocol Security or IPSec is a network security protocol for authenticating and encrypting the data packets sent over an IPv4 network. IPSec protocol works at layer-3 or OSI model and protects data packets transmitted over a network between two entities such as network to network, host to host, and host to the network.


What is an IPSec Protocol?

An IPSec protocol is a protocol which cryptographically secures your internet traffic through authentication and encryption of IP data packets. The IPSec protocol provides an additional layer of security to the Internet Protocol (IP) data packets which carry your internet traffic.

Different protocols involved in IPSec

IPSec uses different protocols for carrying out various functions such as Authentication headers (AH), Encapsulating Security Payloads (ESP), Security Associations (SA), and Internet Security Association and Key Management Protocol (ISAKMP).

Authentication Header (AH)

The Authentication Header provides authentication to the data packets and maintains their integrity. It achieves this by inserting a keyed one-way hash function to the data packet. The sender enters this hash function and sends it. The details of the hash function are also transmitted to the receiver.

The receiver upon receiving the data packets verifies the details and finds out if the data packets have been tampered or not. This hash function also involves a secret value, which is known only to the sender and the receiver.

This way authentication is full proof and verifiable. The Hash function is created either through an MD5 method or a Secure Hash Algorithm method.

The MD5 or the Message Digest 5 algorithm produces a 128-bit hash. This hash is created from a message of random length and a 16-byte key.

The Secure Hash Algorithm creates a 160-bit hash and a 20-byte key. Usually, the latter method is preferred because it makes use of a larger hash.

Encapsulating Security Payload

The AH protocol will protect the data packets from intervention, but it won’t prevent other people from seeing them. That is where the Encapsulating Security Protocol comes in. It allows you to encrypt the packet, provide authentication and maintain content integrity.

You also have a freedom to decide as to what to do with the data packet. You have the option to encrypt and authenticate the packet. Similarly, you also have an opportunity to either encrypt the data packet or authenticate the data packet.

ESP uses following algorithms –

Data Encryption Standard (DES)

DES uses a 56-bit key for encryption. The standard text in a data packet is turned into ciphertext by the DSP through an encryption algorithm. The decryption algorithm on the receiver’s side, in turn, decrypts the ciphertext into standard text.

The secret keys exchanged between the two parties enable the encryption and the decryption.

Triple DES Algorithm (3DES)

The data packets for 3DES are broken into 64-bit blocks. The algorithm then processes each of the blocks three times using a 56-bit key. So, in a way, it doubles the encryption strength compared to the DES protocol.

Security Associations (SA)

Before exchanging data between two hosts on either AH or ESP, an agreement is done on necessary parameters, which is provided by security associations. These parameters are the algorithm of encryption, such as DES or IDEA, and a hash function, such as MD5 or SHA.

Internet Security Association and Key Management Protocol (ISAKMP)

ISAKMP provides the necessary framework for establishing security associations and cryptographic keys in an internet environment.

How does an IPSec Protocol work?

The IPSec protocol involves the exchange of a security key through which they can communicate securely between two hosts. This exchange of the key between your computer and the VPN server would determine the encryption algorithm for verification and authentication.

The distribution and management of this key are crucial for creating the VPN tunnel. The key can be generated manually, automatically or through a Diffie-Hellman exchange.


The network administrator at both the ends personally configure all the security parameters. The mechanism is suitable for a small network where operating and tracking the key is not a herculean task.

It is at a disadvantage however when exchanging the keys over a long distance. This is because you are never sure if the information you send is being received at the other end without intervention.


When the exchange involves creating numerous VPN tunnels, it is not possible to configure each security element manually.

The keys are generated automatically and exchanged along with the security associations through the Internet Key Exchange (IKE) protocol. The exchange can take place either through pre-shared keys or certificates.

In the case that the exchange takes place through a pre-shared key, each side should have configured and shared the key in advance. An Autokey, after an exchange, can change its keys automatically at decided intervals using the IKE protocol.

Since the change is frequent, the security aspect is taken care of. Moreover, it also eliminates the responsibility of manually looking after the keys.

Each side generates a public-private key and obtains a certificate in the second scenario. The certificate issuing authority has to be trusted by both the parties.

This will allow each of them to view the other one’s public key and authenticate its signature. In this scenario, there is no need to track the keys. The IKE takes care of that.

Diffie-Hellman Exchange

The DH exchange allows both the parties to share a secret value. The parties create this value along an unsecured path without having to share the value of the path.

This is one of the critical advantages of this method. The modulus of each DH group is of a different size. Thus, both the parties must agree on using the same group for the exchange.


IPSec protocol modes

An IPSec protocol primarily consists of tunnel mode and transport mode. In the Transport mode, only the payload of an IP data packet is encrypted. The payload is the segment of the overall IP data.

The Tunnel Mode, however, protects the entire IP packet. The whole packet is covered by IP Headers which make it completely safe. Transport mode is always preferred over transport mode.

The Tunnel mode is typically used for securing a connection between two whole gateways.

Final words

IPSec protocol offers a great deal of authentication and encryption to the data flowing between two hosts over the internet. It has proved to be a hugely successful VPN protocol for a long time. Even as on date, it continues to be the most used VPN protocol used across the globe.

Internet Protocol Security or IPSec is a network security protocol for authenticating and encrypting the data packets sent over an IPv4 network. IPSec protocol works at layer-3 or OSI model and protects data packets transmitted over a network between two entities