MCSA/MCSE Self-Paced Training Kit (Exam 70-214): Implementing and Administering Security in a Microsoft Windows 2000 Network (Pro-Certification)

Lesson 4: Troubleshooting IPSec Configuration

IPSec is an extremely complex protocol, and when using certificates to perform IKE Authentication, you will find IPSec configuration probably more difficult and error prone than any other configuration task in Windows 2000. There are literally hundreds of things that can go wrong, but if you consider your scenarios carefully, understand the IPSec mechanisms, start with simpler manual keying to ensure IPSec works before you deploy certificates, and use troubleshooting tools to narrow the scope of the problem, you will be able to fix IPSec problems quickly.

To complete this lesson, you will need

• Domain controller dc01

• Member server ms01

• Domain controller gdi-dc-01

After this lesson, you will be able to

• Determine why IPSec might fail

• Fix problems with IKE keying

• Fix IPSec problems due to certificates that are not valid

Estimated lesson time: 15 minutes

Why IPSec Might Fail

In Windows 2000, IPSec can fail in two ways:

• Communication does not occur.

• Communication is not encrypted.

Of these two, the first is problematic, but the second is disastrous. It's easy for an administrator to think that IPSec is configured and working, while communication is flowing without being secured at all. This provides a false sense of security. When communication is not flowing, you know that you at least have security. Be careful to test that encryption is actually occurring by examining communications using a sniffer between the two end systems.

If the IPSec Policy Agent service is not running, all IP communications will automatically be accepted. You won't have any warning that IPSec is not in place when this service isn't running.

Despite the complexity of IPSec communications, there are only a few reasons why IPSec fails:

• Communication between the end systems doesn't function correctly.

• Filters are not set up correctly.

• IKE does not have a matched set of secret keys.

• IPSec cannot negotiate a compatible set of encryption or authentication algorithms.

• An intermediate system is modifying packets between the end systems.

Don't be fooled by this short list, however, because a myriad of problems might cause any one of these failures. The following sections detail troubleshooting techniques and will help you determine in which of the problem domains you should be troubleshooting.

Communication Between the End Systems Doesn't Function Correctly

All the IPSec troubleshooting in the world won't fix a routing problem or a down circuit. You absolutely must ensure that you have proper connectivity between systems before you attempt to secure communications using IPSec. Test all protocols that you will be using over IPSec when your configurations are complete, including ICMP (ping) and DNS.

Filters Are Not Configured Correctly

IPSec configuration is performed by setting up IP filters. If the filters are not created correctly and do not exist reciprocally on both systems, IPSec communications will not occur.

Common problems include mistaking IP addresses; mismatching AH, ESP, and tunnel mode on each end; and selecting different filter actions on each end. Ensure that your IPSec filters are exactly the same on both ends, and don't include mutually exclusive filters, such as one filter that specifies all protocols or addresses, and another that specifies a certain action for only certain protocols or addresses.

IKE Does Not Have a Matched Set of Secret Keys

This problem can be difficult to differentiate from all the other types of problems. If you suspect you're having problems with secret key exchange, you can enable IKE logging to get detailed information about IPSec negotiations and determine exactly what's going wrong.

It is possible to "over specify" IKE methods. Only one authentication method is valid between a pair of hosts, and the hosts will both choose the first method found that appears to be compatible. If more than one IKE method is specified by policy between the two hosts, a mismatch can occur. Be certain that you haven't specified more than one method that could be misapplied on the hosts involved.

Better yet, as a test, simplify IKE negotiations by manually keying. After both end systems are manually keyed using IKE, and communication is working correctly between systems, IKE will succeed.

Manual keying can be used to determine whether your problem really is IKE.

Once you know you have a problem with IKE, it's time to look at the keying method. If you're using Kerberos, the only requirement is that the servers are in the same domain or that a bi-directional trust relationship exists between the domains.

If you are using certificates for IKE, you must ensure that the certificates are rooted in the same trusted CA and contain private keys. IKE uses a certificate's private key as its secret key to authenticate clients. The certificate need not be specifically created for IPSec. Any valid certificate in the machine local store containing a private key should work. You can verify whether a specific certificate has a private key by using the Certificates management console.

You might find it necessary to restart the IPSec policy agent if you've changed the authentication method and you can't determine why IKE has not converged.

If you stop the IPSec policy agent, communications will flow unsecured between the hosts without warning. It is safest to restart the computers involved if you need to restart the IPSec policy agent to ensure that traffic does not flow in an unencrypted state when security is critical.

Remember to be patient when applying changes through Group Policy, because it takes time for Group Policy to propagate and to be refreshed on servers. Consider using Local IPSec policy for testing purposes, because it takes effect immediately.

IPSec Cannot Negotiate a Compatible Set of Encryption or Authentication Algorithms

This problem occurs when servers don't have a compatible set of authentication or encryption protocols. This problem typically does not occur between Windows 2000 hosts, because the same set of protocols is available to them all, but it can occur if you've removed lower-quality Data Encryption Standard (DES) encryption from your policy but have not enabled high encryption on hosts. This is not a problem with Windows 2000 Service Pack 2 and later, which includes high encryption by default.

You might find that security protocols cannot be authenticated correctly because multiple policies exist that apply to both servers. Be certain that only a single policy applies between any two hosts in an IPSec security association.

An Intermediate System Is Modifying Packets Between the End Systems

Tampering with AH packets will cause them to fail and could potentially be used as a denial of service attack in a rare and sophisticated type of "man-in-the-middle" attack. A far more common scenario is that Network Address Translation (NAT) is being used somewhere between the end systems, and the NAT gateway is modifying header addresses, which causes AH to fail due to checksum errors.

AH is not compatible with NAT, and depending on the version of NAT, ESP transport mode might not be compatible either. ESP tunnel mode tends to be the most compatible form of IPSec for transport through NAT, because it uses typical IP headers.

The only good way to get IPSec working when NAT is being used is to disable NAT, which might not be an option for the interior of most networks. If you need to get IPSec traffic through a NAT device, you may have no option except to use an IPSec tunnel that ends on the NAT router. Typically, border routers perform NAT for the interior network and can be used as end points for IPSec tunnels. This is how the majority of existing IPSec connections between private networks work today.

In this practice, you explore various troubleshooting methods that can be used to determine what might be wrong with an IPSec link that you are trying to establish.

Exercise 1: Refreshing Domain IPSec Security Policy

In this exercise, you refresh domain IPSec policy. Use this procedure whenever you make a change to a domain GPO and you want to test its effect on a specific computer that the GPO applies to.

To refresh domain IPSec policy

1. Click Start, and then click Run to open the Run dialog box.

2. Type cmd, and press Enter. A command prompt appears.

3. Type secedit /refreshpolicy MACHINE_POLICY, and press Enter.

4. Type exit, and press Enter.

Exercise 2: Determining Whether a Certificate Has a Private Key

In this exercise, you determine whether a machine certificate contains a private key, which is required for IKE to negotiate a security association.

To create a Certificates console

1. Click Start, and then click Run to open the Run dialog box.

2. Type mmc, and click OK.

3. From the Console menu, choose Add/Remove Snap-in.

4. In the Add/Remove Snap-in dialog box, click Add.

5. In the Add Standalone Snap-in dialog box, double-click Certificates. The Certificates Snap-in Wizard appears.

6. Select Computer Account, and click Next.

7. Select Local Computer.

8. Click Finish.

9. Click Close to close the Add Standalone Snap-in dialog box.

10. Click OK to close the Add/Remove Snap-in dialog box.

11. From the Console menu, choose Save.

12. Type Certificates (local machine) as the name of the management console, and then click Save.

To determine whether a certificate includes a private key

1. Open the Certificates (Local Machine) management console.

2. Expand Certificates and Personal, and then select Certificates.

3. Double-click the certificates shown in the certificates list. The Certificate Information dialog box will appear, as shown in Figure 8.28.

   Figure 8-28. A certificate's information shows whether it contains a private key

   If the certificate contains a private key, it will be noted at the bottom of the certificate.

4. Click OK and close the Certificates (Local Machine) management console.

Exercise 3: Restarting the IPSec Policy Agent

In this exercise, you restart the IPSec policy agent to solve any problems that might have occurred when an authentication method was changed.

Communications are not secured when the IPSec policy agent is not running.

To restart the IPSec policy agent

1. Open a command prompt.

2. Type Net stop policyagent, and press Enter.

3. Type Net start policyagent, and press Enter.

Exercise 4: Enabling IKE Logging

In this exercise, you enable IKE logging on a computer to determine exactly why an IKE negotiation has failed. Use this procedure whenever you suspect that you have a problem with IKE and you cannot determine what is failing.

To enable IKE logging

1. Click Start, and then click Run to open the Run dialog box.

2. Type regedt32, and then click OK. The Registry Editor appears.

3. Select the HKEY_LOCAL_MACHINE window.

4. Expand System, CurrentControlSet, Services, and PolicyAgent.

5. If the Oakley key doesn't exist, from the Edit menu, choose Add Key. The Add Key dialog box appears.

6. Type Oakley as the name of the key, and click OK.

7. Right-click Oakley, and click Add Value.

8. Type EnableLogging as the name of the value.

9. Select Data Type: REG_DWORD, and click OK.

10. Enter value 1 and click Hex as the Radix.

11. Click OK, and close the Registry Editor.

12. Restart the policy agent. The log file will be written to windir\debug\oakley.log.

The following questions are intended to reinforce key information in this lesson. If you are unable to answer a question, review the lesson and try the question again. Answers to the questions can be found in the appendix.

1. What IPSec problem could have the most serious consequences?

2. What utility is used to refresh Group Policy on a computer that receives its IPSec configuration from a domain GPO?

3. What utility is used to enable IKE logging?

4. What's the easiest way to eliminate IKE as a potential problem source?

5. What problems might cause certificate-based IKE negotiations to fail?

Lesson Summary

• IP Security fails for the following common reasons: communication doesn't work correctly between the end systems, filters are not set up correctly, IKE secrets don't match, IKE could not negotiate a matched set of protocols, or an intermediate system is modifying packets between the end systems.

• The first step in troubleshooting IPSec is to troubleshoot everything else ensure that the network is functioning correctly and that higher-layer protocols like DNS, Kerberos, and ICMP are all functioning correctly before attempting to get IPSec functioning.

• Creating custom filters can be an error-prone process. Be sure that you understand clearly what you're trying to accomplish, and design your policies in advance before attempting to implement them.

• IKE problems are the problem that most commonly affects IPSec communications in Windows 2000. Typical problems include mismatched or incompatible certificates, incorrectly specified manual keys, and multiple IKE policies that could be applied. Only one IKE policy is selected, so ensure that you know which one it is by having only one available.

• Test IKE problems with certificates by using manual keying. Once you have IPSec working correctly with manual keying, attempt to use certificates and troubleshoot any problems as certificate problems.

• There must be compatible IPSec algorithms on both end systems. Normally, this isn't an issue, but if you've removed some encryption methods to force high encryption or you are attempting to get IPSec working with a third party system, you might need to compare both systems closely to make sure that they have a set of encryption protocols in common.

• IPSec is incompatible with Network Address Translation (NAT). NAT is widely deployed at the boundary between private networks and the public Internet, so it's likely that you won't be able to create IPSec connections that go from the private to a public network. Clever planning and security design will be required to solve problems like these.