Method for diagnosing IPsec problems in Win 2003

You have implemented IPsec to protect communications within your corporate LAN. Although you have carefully followed all technical documentation, you can't be sure that network communications are actually protected from eavesdropping. How can I re-establish the confidence that IPsec does encrypt your computer's network communications?

Let's take a look at how to make sure everything goes smoothly, then ——if you find the problem during the verification process—— take a look at the overall situation of troubleshooting IPsec-related verification issues . During the verification process, I will show you some Windows Server 2003 built-in tools. If you really find that network communication is not encrypted, you can use these tools to diagnose IPsec problems (but please note: I am in this article The techniques discussed may not be applicable to a particular IPsec application, such as an IPsec card and virtual private network based IPsec.

1. Check if IPsec is running

Even if you have successfully deployed IPsec, there are still many problems. When you are upset about the loss of a typical network connection on a Windows Server 2003 or Windows XP machine, the operating system guarantees that you will be aware of the problem. Unfortunately, there is no similar way to keep you informed of the state of IPsec. Relying on IPsec deployment, when you have problems, either release all network connections, or —— is more likely, and more secretive is —— network communication can continue, but no longer encrypted. Imagine how shocked you are when you believe you are protected and find that your network communication is not safe at all.

The quickest way to check if IPsec is running is to use Network Monitor to capture packets coming in and out of your computer and check if they are encrypted. Network Monitor is included with Windows Server 2003. You can install it by opening the “Control Panel"*“Add or Remove Programs>, Add/Remove Windows Components. After installation, you can start the package capture by selecting “Start” from the “Capture" menu. Then, create some network behavior, generate some data, and perform common actions such as browsing a shared folder on another computer. Finally, under the “Capture" menu, select “Stop and View”.

Figure 1 shows the results of two packet captures. On the left is the packet capture that is performed when the computer is not configured to use IPsec, and you will see multiple protocols listed in the Protocols column. On the right is the packet capture that is performed when the computer is configured to use IPsec. You will see that only one protocol is listed: Encapsulating Security Payload (ESP) protocol. When a Windows Server 2003 computer is configured to use the default IPsec policy, only ESP and Internet Control Message Protocol (ICMP) will appear in the packet capture. The Internet Control Message Protocol will appear because the default IPsec policy allows ICMP communication. So, if you see multiple protocols when you capture network traffic, you have reason to think that IPsec is not working properly.

Figure 1: Comparison of packet capture

Almost all IPsec problem occurs because Internet Key Exchange (Internet Key Exchange, IKE) verification difficult phase. When two computers attempt to establish a security association (SA), they go through a process of verifying each other's identities. IKE is an algorithm for negotiating a security association. Authentication is based on pre-shared keys, digital certificates, or Kerberos. The default Windows Server 2003 IPsec policy uses Kerberos. In most instances, troubleshooting IPsec means troubleshooting the verification process.

2. Restart IPsec Service

Once you have determined that IPsec is not running, you should first try to restart the IPsec service. Restarting the IPsec service will completely clear the IKE negotiation status. This approach usually makes it work again when IPsec becomes ineffective after a major change in policy. The two advantages of this solution are that it does not require a server restart and is less time consuming to implement. You can restart the IPsec service by running the following Net command from a command prompt on a Windows Server 2003 computer:

The following is a code snippet: net stop policyagent net start policyagent

Now, perform the network packet capture test again, or run the Netsh command I will talk about later.

3. Diagnosing IKE Problems in the Event Log

If restarting IPsec does not solve the problem, you can check the security event log instead. The act of creating and deleting security associations is audited as a network login event. These events are logged if you enable auditing for success and failure in the audit login event policy. When everything is working, the success codes that appear are 541, 542, and 543. However, this article discusses what you should do when IPsec is not running, so in our case, you need to pay special attention to the failure code shown in Table 1. Figure 2 shows the failure event 547.

Table 1: Failure Audit Code

Figure 2: Failure Event 547

a safety issue associated with the event log that the event will soon fill your log. If you specifically audit the login event but don't want your security event log to be filled with IKE entries, you can create a <;HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Lsa\\Audit\\DisableIKE Audits” subkey and set its key value to 1 to prohibit IKE audit.