IPsec Modes of Operation and Security Options

As highlighted in Chapter 4, IPsec has many protocol options. This section provides a brief look at the main elements. One concept that's important to understand for IPsec is the concept of security associations, or SAs. An SA is simply a connection that provides security services to the traffic carried by it. When you use IPsec, you build SAs between the communicating devices. The SA contains information about which data traffic is going to be encrypted, how it is encrypted, whether and how it is authenticated, and the encryption keys and how often they are refreshed.

The Three Elements of IPsec

Reduced to its most critical elements, IPsec comprises three main technologies:

• Internet Key Exchange (IKE)
• Authentication Header (AH)
• Encapsulating Security Payload (ESP)

IKE

IKE, defined in RFC 2409, can be thought of as the maintenance protocol for IPsec. It takes care of negotiating the various IPsec options, authenticating each side of the communication (including public key exchange when applicable), and managing session keys for the IPsec tunnels. IKE is a "meta" protocol comprising elements of Oakley (RFC 2412), Internet Security Association and Key Management Protocol (ISAKMP) (RFC 2407), and Secure Key Exchange Mechanism (SKEME). SKEME is used to authenticate both sides of the IKE SA, Oakley is used to derive the encryption key for the session, and ISAKMP defines the message format for the exchanges. IKE uses User Datagram Protocol (UDP) port 500 (generally for both source and destination) for communication.

AH

AH, detailed in RFC 2402, is primarily an authentication option for IPsec. As the RFC states, AH "is used to provide connectionless integrity and data origin authentication for IP datagrams . . . and to provide protection against replays."

Figure 10-3 shows an original IP datagram and how it is modified with AH in transport mode and tunnel mode. (More on tunnel and transport modes is in the next section.) AH uses IP protocol 51 for communication.

Figure 10-3. AH

NOTE

In all of the packet details in this chapter, TCP is listed as the Layer 4 (L4) protocol. However, any L4 protocol can be used with IPsec because IPsec is an L3 function.

ESP

ESP, detailed in RFC 2406, affords most of the protections of AH plus the addition of confidentiality. As the RFC states, ESP "is used to provide confidentiality, data origin authentication, connectionless integrity, an anti-replay service . . . and limited traffic flow confidentiality."

These features can be enabled or disabled as desired by the administrator. Figure 10-4 shows an IP datagram before and after ESP in transport mode and tunnel mode. As you can see, the IP header is the only item AH can authenticate that ESP cannot.

Figure 10-4. ESP

As a designer, the first thing to realize is that almost no one uses AH. All the rest of the designs discussed in this book employ ESP. This is primarily because you get almost every benefit of AH by using ESP. And since ESP offers confidentiality, it is superior from a security standpoint. Even IP header authentication, which ESP doesn't offer in transport mode, is possible in tunnel mode (for the original IP header). ESP uses IP protocol 50 for communication.

Transport Mode and Tunnel Mode

After understanding the three main components of IPsec, it is useful to compare the two main modes in which IPsec can operate: transport mode and tunnel mode. Transport mode is used when an SA is built between two hosts, as shown in Figure 10-5.

Figure 10-5. Transport Mode

Someday in the future, transport mode IPsec may be used ubiquitously for all IP communications. If the Public Key Infrastructure (PKI) issues can be addressed (and I'm not sure they can), any host on the Internet will be able to look up any other host and validate its identity and authenticate its data (in the case of AH) or encrypt its entire communications (in the case of ESP).

Tunnel mode is used when an SA is built between two IPsec gateways, as shown in Figure 10-6. It is also used when an SA is built between an IPsec gateway and a host, as shown in Figure 10-7.

Figure 10-6. Tunnel Mode Gateway to Gateway

Figure 10-7. Tunnel Mode Host to GW

Today, tunnel mode is preferred almost everywhere in networking because it allows private addressing and the broadest range of connectivity options. In both site-to-site VPNs and remote user VPNs, it is required per the RFCs. Transport mode is used in one-off connectivity situations, remote management, and when combined with an alternate encapsulation protocol such as GRE.

IPsec SA Establishment

Oftentimes when discussing IPsec, you'll hear mention of terms such as main mode, aggressive mode, and quick mode. Sometimes these terms are replaced with phase 1 and phase 2. This nomenclature problem is born out of the ISAKMP protocol's use of the term phase and the Oakley use of mode. An IKE SA is bidirectional and is a secure, authenticated channel for the establishment of IPsec SAs. The IKE SA creation is called "phase 1" and comprises either a "main mode" exchange or an "aggressive mode." Phase 2 comprises the "quick mode" exchange and negotiates the parameters necessary for the establishment of the IPsec unidirectional SAs. Figure 10-8 shows the three SAs necessary for basic IPsec connectivity, and Figure 10-9 shows the different phases and modes and the key characteristics of each.

Figure 10-8. SA Establishment for Basic IPsec

Figure 10-9. IKE Phases and Modes

Phase 1

IKE phase 1 is started by the IPsec initiator that forwards interesting traffic out the link. Interesting traffic is defined in the security policy database (SPD) as defined in RFC 2401. The SPD takes its form on many devices as an access control list (ACL). Traffic matching the ACL is sent to the IPsec process; traffic not matching is sent in the clear. For IKE phase 1 to begin, traffic that the initiator wishes to encrypt must be seen. This makes IPsec a just-in-time mechanism. When you do your pilot deployment of IPsec in your own organization, you will likely see some of the initial ping packets used to start the IKE process fail because the IPsec SAs are not set up quickly enough.

Main Mode

Main mode is the standard method of IKE negotiation. It takes six messages to negotiate and supports the widest range of options. When using preshared keys instead of digital certificates, main mode requires a known static IP address for both peers. This is because the IPsec identity information is protected in the negotiation. This means the key used for communications must be determined before the identity information is available. This is done by using the source IP address of the communications.

Aggressive Mode

Aggressive mode differs from main mode in the number of options available. Certain items such as the Diffie-Hellman (DH) group (more on DH later in this chapter) must be exact matches as opposed to negotiated parameters. In addition, the identity information is sent in the clear. This allows clients in remote user VPNs to use preshared keys with dynamic IP addressing. Authentication options are discussed in the following section. In general, however, you should use main mode whenever possible and aggressive mode when necessary (such as for remote clients).

Phase 2: Quick Mode

IKE quick mode provides the information necessary to set up the IPsec SAs. Three messages are sent, and all the parameters for the IPsec SA are negotiated. When using perfect forward secrecy (described later in this chapter), a new DH exchange is used to produce the keys.

Other Security Options

Besides making decisions such as AH versus ESP, tunnel versus transport, and main mode versus aggressive mode, you can consider a number of more specific options within IPsec that can affect the design. These include the following:

• Authentication methods
• DH group
• Perfect forward secrecy (PFS)
• Encryption protocol selection
• Authentication/integrity protocol selection

Authentication Methods

Within IKE, there are three main methods of authenticating the remote party:

• Preshared keys
• Digital signatures
• Encrypted nonces

Feel free to read about the third option if you are interested, but the first two make up 99 percent of IPsec deployments. As such, this chapter focuses on them.

In addition to these methods, there are two prestandard techniques that aid in remote user VPN authentication:

• Mode Config
• Extended Authentication (Xauth)

Preshared Keys

Preshared keys are the simplest form of IPsec peer authentication. In this method, the same key is statically configured on each peer. Although quick to set up and test (the examples in this book all use preshared keys), this method is not scalable. Consider 100 or 1000 peers each needing a unique key pair to ensure security. If you build a fully meshed network, you enter the classic "n squared problem."

Even assuming hub-and-spoke, which makes up the majority of IPsec deployments today, that is still too many keys. Many organizations get around this by using something called a group preshared key (sometimes called a wildcard preshared key). A group preshared key is a key that is used on a device for multiple peers rather than being unique to a specific peer. This method eases the initial configuration requirements but is much less secure. Consider if one router is compromised and the preshared key is determinedall the peer routers must be immediately reconfigured with a new key. Not fun.

Digital Signatures

As an alternative, digital signatures allow each device to assert its identity in a verifiable way to other parties. The most common method is using a CA as described in the "Basic PKI" section of Chapter 4.

From a design standpoint, digital certificates should be used whenever possible for large site-to-site deployments. Although the definition of "large varies" from organization to organization, anything over about 25 peers is a good justification for PKI, even fewer if you are deploying a full mesh (discussed later). Because of the difficulty getting and managing digital certificates on your users' workstations, most organizations choose to use wildcard preshared keys with Xauth and OTP (discussion to follow for remote user VPNs).

This PKI should be a "closed" PKI and need not interact with any other identity systems in your network. Whether you choose to source the CA yourself or work through a service, the end result is the same.

NOTE

One essential element of your PKI not already discussed is the concept of certificate revocation lists (CRLs). From time to time, an IPsec peer may compromise its trust with the rest of the network. This could be caused by a device compromise or theft. It is essential that the rest of your network be told of this event; otherwise, users will continue to trust the device. CRLs are the means to distribute this information. Housed at the CA, a CRL is like a list at a store identifying people who have tried to pass bad checks. Before accepting an IPsec peer as trusted, the CRL is checked in advance by the IPsec device. Although many devices list CRLs as an optional feature, they should always be implemented when using digital certificates.

TIP

When using CRLs and a PKI, it is important for each device to have the accurate time because devices need to know when to retrieve a new CRL list as well as when to check the validity dates on a certificate.

Mode Config and Extended Authentication (Xauth)

Mode Config is a method for provisioning an "internal" IP address for the IPsec ESP tunnel. It is implemented in some IPsec gear now, though the standard for this will not be released until IKEv2. When a remote user VPN is established from a client, it is useful to provision that client with an internal (perhaps RFC 1918) address by which policy decisions can be made for the client. Without Mode Config, the inner address of the tunnel would be the dynamic IP address assigned to the client, which is not generally suitable for access control decisions within an organization's network.

Xauth defines a method of extended authentication that occurs between IKE phase 1 and phase 2. It allows for a username and password to be required before IKE establishes the IPsec SAs. In this way, remote user VPNs can take advantage of group preshared keys for IKE phase 1 but can require more extensive, user-specific authentication before phase 2 (often an OTP). This addresses some of the limitations of preshared keys, but it is still not as secure as digital certificates. But because an organization can take advantage of its existing AAA infrastructure, Xauth is considerably easier to manage.

Xauth and Mode Config as currently implemented, will need to be updated after IKEv2 is released to be standards compliant. IKEv2, for example, mandates EAP as the Xauth method.

Diffie-Hellman Group

As mentioned previously, Oakley (a part of IKE) is used to derive a session key that enables encryption for the IKE information. Diffie-Hellman (DH) is the algorithm used for this negotiation. The coolest part about DH (if indeed crypto is ever cool) is that this shared session key is derived with no secrets shared between the parties in advance and without the secret key being transmitted between the parties. This was quite an achievement when it was introduced in 1976. DH is vulnerable to man-in-the-middle (MITM) attacks, which is why SKEME is used to authenticate each peer in the DH exchange. DH as part of Oakley can operate in several different groups. For example, group 1 is 768-bit, group 2 is 1024-bit, and group 5 is 1536-bit. As with most algorithms in cryptography, if all other things are equal, more bits means more secure. It also means more computation. As you go through the pilot phase of your VPN, you might wish to try different DH group values and determine whether there is a noticeable difference in IPsec gateway CPU utilization. If not, higher groups are more secure.

Perfect Forward Secrecy

As discussed earlier, the DH algorithm is used by IPsec to derive the session key used in an SA. This occurs for both IKE and IPsec SAs. Perfect forward secrecy (PFS) is the idea that the compromise of any one of these session keys should not allow the attacker any useful additional data to aid in compromising the others. Without PFS, for example, if an attacker were able to compromise the IKE SA session key, the attacker could perhaps use it to increase the chances of compromising the IPsec SAs session keys. PFS ensures that this can't happen by requiring that unique DH exchanges occur for each of the needed session keys. From a design standpoint, it is very unlikely that someone will be able to compromise any of these keys. However, it doesn't take much work to enable PFS on your devices. The only design impact (besides the security difference) is the potential performance impact the additional DH exchanges take. Ask your hardware vendor if enabling PFS will have a significant impact on the crypto chips the vendor uses. Like DH group selection, you should experiment with this feature in the pilot stage to determine impact.

Encryption Protocol Selection

After examining many of the supporting design choices you can make, one of the biggest decisions you have to make is which encryption mechanism you use for the transport of the data. This section very briefly describes the three main protocols you can select to encrypt your SAs.

• Data Encryption Standard (DES)
• Triple DES (3DES)
• Advanced Encryption Standard (AES)

DES

First standardized in 1977, DES should no longer be used. In a 1999 decryption challenge, 56-bit DES ciphertext was cracked in just over 22 hours by using an exhaustive key search. See the following URL for more details: http://www.rsasecurity.com/rsalabs/challenges/des3/.

Triple DES

Triple DES (3DES) essentially encrypts the data three times using DES and three different 56-bit keys. This is considered secure by most and comprises the majority of IPsec VPN deployments today. Although it depends on the type of VPN hardware you are using, 3DES has additional performance requirements for your VPN hardware compared to DES.

AES

AES was ratified by the U.S. National Institute of Standards and Technology after a 4-year study to determine a suitable replacement to DES. It supports key sizes of 128, 192, and 256 bits as opposed to 56 bits for DES. It is defined in Federal Information Processing Standards (FIPS) publication 197. If your vendor is still using 3DES, you should make sure it has a short-term road map to AES support. AES is the new standard for data encryption and should be used instead of 3DES whenever possible.

Authentication/Integrity Protocol Selection

When using IKE/IPsec, you have a choice of two different hash mechanisms for authentication and integrity checks: the Secure Hash Algorithm (SHA) and message digest 5 (MD5). This is a simple choice: SHA-1 is 160-bit and MD5 is 128-bit. But because they operate in different ways, you can't just examine the number of bits to determine the security level. That being said, SHA-1 is considered more secure but more computationally intensive. Just like any other crypto function on an IPsec gateway, consider the performance penalty, if any, before opting for the more secure choice.