IPsec troubleshooting in Microsoft Windows 2000 Server (257225)


The information in this article applies to:

  • Microsoft Windows 2000 Server
  • Microsoft Windows 2000 Professional
  • Microsoft Windows 2000 Advanced Server
  • Microsoft Windows 2000 Datacenter Server
This article was previously published under Q257225
Important This article contains information about modifying the registry. Before you modify the registry, make sure to back it up and make sure that you understand how to restore the registry if a problem occurs. For information about how to back up, restore, and edit the registry, click the following article number to view the article in the Microsoft Knowledge Base:

256986 Description of the Microsoft Windows Registry

SUMMARY

To troubleshoot IPsec connection problems in Microsoft Windows 2000, first verify the success of Internet Key Exchange (IKE) security negotiation. To do this, enable Audit policy and then examine the Security log. Next, use the Netdiag.exe command-line tool to display debugging information. Then, depending on whether the problem occurs in phase one or in phase two, examine your IPsec policy properties and IPsec rules.

Use IP Security Monitor to view more information about IPsec and security associations. You can also use IP Security Monitor to view IKE statistics. Use Network Monitor to analyze network traffic and the status of the various protocols used in your network. You can use the Netsh command to troubleshoot instances where IP offloading occurs on IPsec packets.

You can also use the information in this article to do the following:

  • Obtain an Oakley log
  • Understand the contents of event logs
  • Troubleshoot "bad SPI" messages
  • Restart the policy agent
  • Verify the integrity of your policies

Contents

INTRODUCTION

This article contains guidelines for troubleshooting Internet Protocol security (IPsec) connection problems in Microsoft Windows 2000. IPsec relies on the Internet Key Exchange (IKE) protocol to establish shared security parameters and authenticated keys between two computers. The IKE protocol uses two phases. In phase one, Windows 2000 uses the IKE Security Association and Key Management Protocol (ISAKMP) Main Mode exchange. (Windows 2000 does not support Aggressive Mode.) When the phase one exchange provides a secured channel, the computers obtain an authenticated key and an IKE security association. This secured channel is used in phase two to help protect the Quick Mode Exchange. The Quick Mode Exchange provides IPsec security associations.

MORE INFORMATION

Basic IPsec troubleshooting

To troubleshoot IPsec, first enable Audit policy, and then verify the results of phase one and phase two exchanges. When you enable Audit policy, security events are logged in the Security log. By examining the Security log, you can determine whether IKE security association negotiation is successful. To enable Audit policy, follow these steps:
  1. In Group Policy, expand Local Computer Policy.
  2. Locate and then click Computer Configuration/Windows Settings/Security Settings/Local Policies/Audit Policy.
  3. In the details pane, right-click Audit logon events, and then click Security.
  4. Click to select Success, click to select Failure, and then click OK.
  5. In the details pane, right-click Audit object access, and then click Security.
  6. Click to select Success, click to select Failure, and then click OK.
Note If you are using a domain policy for auditing, the domain policy overwrites your local policy.

Next, type the following command to use the Netdiag.exe command-line tool:

netdiag /test:ipsec /debug

This command displays debugging information about phase two.

Note To use Netdiag.exe, the Windows 2000 Support Tools package must be installed on your computer. To install the Windows 2000 Support Tools, follow these steps:
  1. Start Windows 2000.

    Note You must log on as a member of the administrator group to install these tools.
  2. Insert the Windows 2000 CD into your CD drive.
  3. Click Browse this CD, and then open the Support\Tools folder.
  4. Double-click Setup.exe, and then follow the instructions that appear on the screen.
You can also use Netdiag.exe to view the policy without an active connection. To do this, type the following command at a command prompt, and then press ENTER:

netdiag /test:ipsec /v

This command displays the current policy and IPsec statistics with regard to phase one.

If the logged events indicate that phase one Main Mode exchange fails, verify the IKE settings and the IKE authentication methods in your IPsec policy properties. To do this, follow these steps:
  1. Click Start, click Run, type secpol.msc, and then click OK.
  2. Click the IPsec rule that you want to click, right-click IPsec rules and then click Properties.
  3. Click the General tab, and then verify that the settings are correct.
  4. Click Advanced, examine the settings, click Methods, and then examine the settings.
  5. Click OK two times.
  6. Click Rules tab, click Edit, and then click the Authentication Methods tab.
  7. Examine the settings on this tab.
If the logged events indicate that phase two Quick Mode fails, verify the IPsec security methods in the IPsec rules and in your IPsec policy properties. To do this, follow these steps:
  1. Click the IPsec rule that you want to verify, click Edit, and then click the Filter Action tab.
  2. Click the filter action that is enabled, click Edit, and examine the settings.

Using IP Security Monitor

You can use IP Security Monitor to monitor your security associations, IPsec statistics, and IKE statistics. In particular, you can use IP Security Monitor to verify the success of authentication and security associations. To start IP Security Monitor, click Start, click Run, type ipsecmon, and then click OK.

Note By default, IP Security Monitor displays statistics for the local computer. To specify a remote computer, click Start, click Run, type ipsecmon computer_name, and then click OK.

The upper group box in the IP Security Monitor dialog box displays the active security associations and the configuration of the active policy. The lower left group box displays the following IPsec statistics:
  • Active Associations
    The number of active security associations.
  • Confidential Bytes Sent
    The number of bytes sent by using the Encapsulating Security Payload (ESP) security protocol (decimal ID 50).
  • Confidential Bytes Received
    The number of bytes received by using the ESP security protocol.
  • Authenticated Bytes Sent
    The number of bytes sent with the authentication property enabled.
  • Authenticated Bytes Received
    The number of bytes received with the authentication property enabled.
  • Bad SPI Packets
    The number of packets whose Security Parameters Index (SPI) is not valid. A positive number probably indicates that the security association has expired or is no longer valid.

    The SPI is a unique identifying value in the security association. This value lets the receiving computer determine the security association to use to process the packet.
  • Packets Not Decrypted
    The number of packets that the receiving IPsec driver cannot decrypt. A positive number may indicate one or more of the following problems:
    • The security association has expired
    • The security association is no longer valid
    • Authentication did not succeed
    • Integrity checking did not succeed
  • Packets Not Authenticated
    The number of packets that were not authenticated to the IPsec driver. A positive number may indicate that the security association has expired or is no longer valid. The IPsec driver must have the information in the security association in order to process the packets.

    A positive number may also indicate that the two computers have incompatible authentication settings. Verify that the authentication method is the same for each computer.
  • Key Additions
    The number of keys that the ISAKMP/Oakley mechanism sent to the IPsec driver. A positive number indicates that the ISAKMP phase two security associations were successfully negotiated.
ISAKMP/Oakley statistics are located in the lower-right window pane. The lower-right pane displays the following statistics for the ISAKMP/Oakley security mechanism:
  • Oakley main modes
    The number of successful security associations that were established during ISAKMP phase one. A positive number indicates that the key information exchange was successful. Identities were authenticated and common keying material was established.
  • Oakley quick mode
    The number of successful security associations that were established during ISAKMP phase two. A positive number indicates that the negotiation for protection services during the data transfer was successful.
  • Soft Associations
    The number of ISAKMP phase two negotiations that resulted in the computers agreeing only to a clear-text data transfer. A clear-text data transfer involves no encryption or signing of the packets.
  • Authentication failures
    The number of times that authentication of the computer identities did not succeed. If this number is positive, verify that the authentication method settings for each computer are compatible. A positive number may also indicate that the security association has expired.
  • IPsecmon configurations
    A configurable option that lets you adjust the update rate of the data.
IP Security Monitor also indicates whether IP Security is enabled. This information is in the lower-right group box of the IP Security Monitor dialog box. To reset the statistics in IP Security Monitor, restart the IP Security Policy Agent by using Computer Management (Compmgmt.msc).

Using Network Monitor

You can use Network Monitor to analyze the following:
  • Network traffic
  • The IKE exchange protocol
  • The IPsec protocol
  • The ESP protocol
  • Authentication Header (AH)

Obtaining an Oakley log

Warning If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use Registry Editor at your own risk.

Developers and network administrators who have advanced IKE knowledge can modify the registry to obtain an Oakley log. To do this, use Registry Editor to locate the following registry subkey. If the subkey does not exist, create it.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\Oakley

Add an entry of the REG_DWORD type value named "EnableLogging." Give this entry a value of 1. When this entry takes effect, an Oakley.log file is created in the %systemroot%\Debug folder.

Note To turn off logging, give the EnableLogging entry a value of 0.

Using the Netsh command

You can use the Netsh command to troubleshoot instances where IP offloading occurs on IPsec packets. IP offloading occurs when the network card instead of the CPU performs IP functions. For example, IP offloading occurs when the network card performs checksum calculations or performs packet encryption and decryption. IP offloading causes the IPsec driver to drop the packet. To determine whether an interface can perform IP offloading, follow these steps:
  1. Click Start, click Run, type cmd, and then click OK.
  2. At the command prompt, type netsh int ip show offload, and then press ENTER.
This command displays the offloading capabilities of the interface. However, the command does not display statistics. To view statistics, use IP Security Monitor to monitor confidential bytes received. Use these statistics to determine whether packets are lost or received.

To disable IP offloading, follow these steps to modify the registry:
  1. Click Start, click Run, type regedit, and then click OK.
  2. Locate and then click the following registry subkey:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\IPSEC

    If the EnableOffload entry is not present, follow these steps to create the entry:
    1. Right-click IPSEC, point to New, and then click DWORD value.
    2. Type EnableOffload to name the new value, and then press ENTER.
    3. Double-click EnableOffload.
    4. Type 0, and then press ENTER.
If the IP connection that you are troubleshooting succeeds, the problem is caused by IP offloading.

Event logs

The following events may be logged in the Security event log: 
  • Informational event 279
Source: PolicyAgent 
Category: None
    This event indicates the following:
    • Whether an IPsec policy is in effect
    • The source of the IPsec policy
    • The Active Directory polling interval, if the source of the policy is a domain
    Additionally, if an IPsec policy was changed, the event text includes "Updating IPsec Policy."
  • Error event 284
Source: PolicyAgent 
Category: None
    This event indicates that the IP Security Policy Agent cannot contact the Active Directory for the domain that the computer belongs to.
  • Audit event 541
Source: Security 
Category: Logon/Logoff
    This event indicates that an IPsec hard security association has been established. Soft security associations are not audited.
  • Audit event 542
Source: Security 
Category: Logon/Logoff
    This event indicates that an IPsec security association has ended. The ended security association may be hard or soft.
The following events may be logged in the Application event log. The Application event log includes messages from ISAKMP/Oakley. The following events indicate that a domestic version of Windows 2000 tried to negotiate greater security than an export client can support. 
  • Warning event 541
Source: Oakley 
Category: None
    This event indicates that the export client cannot generate domestic-strength key material. The resulting negotiation agrees only on export-strength key material.
  • Warning event 542
Source: Oakley 
Category: None
    This event indicates that the export client cannot perform encryption stronger than Data Encryption Standard (DES). The resulting negotiation agrees only on DES if the other computer can support DES.

General troubleshooting

Troubleshooting "bad SPI" messages in the Event Viewer

"Bad SPI" messages are logged in the following circumstances:
  • If a key lifetime value is set too low
  • If the sender continues to transmit data to the receiver after the security association has expired
When you receive a new security association, you must start transmitting data on it. However, if you communicate with a slower responder, the slower responder may receive IPSed-protected data that it does not recognize. The responder considers an SPI that it does not recognize a "bad SPI." To determine the problem and to correct it, use IP Security Monitor to examine the number of rekeys. Consider how long the connections have been active. If the number of rekeys is very large, configure longer key lifetimes in the policy. Acceptable values for high-traffic Ethernet connections are more than 50 megabytes and more than five minutes.

Configuring longer values may not prevent bad SPIs. However, configuring longer values can significantly reduce the number of bad SPIs. Typically, Windows 2000 Server logs event 4268 to indicate that packets were discarded because of a bad SPI.

If IP Security Monitor indicates that secured security associations are not established, nonsecure security associations may be preventing secured security associations from being established.

Note A secured security association is also known as a hard security association. An nonsecure security association is also known as a soft security association.

Run IP Security Monitor on one of the peer computers. If a security association exists, and the security setting is None, an nonsecure security association exists. An nonsecure security association remains on the computer as long as traffic is regularly sent. To prevent this condition, stop all traffic until the security association times out. Typically, the security association times out in five minutes. Use IP Security Monitor to make sure that the security association is no longer established, and then start traffic again. If policies are compatible, a secured security association is automatically established. Restart the policy agent to delete all nonsecure security associations.

If the files that are required for IPsec components have been removed or deleted, reinstall the IPsec components by removing and then reinstalling the TCP/IP network protocol. Files that IPsec components require include the following:
  • ISAKMP/Oakley
  • IPsec Policy Agent
  • The IPsec driver
IPsec components are reinstalled when you reinstall TCP/IP.

IPsec negotiations may fail because of incompatible IPsec policy settings. Examine the Security event log on each computer that participates in a negotiation. Recent events may record attempts to perform an Oakley negotiation. The events may include a description of the success or the failure.

Verify the integrity of the policy on each computer. To determine the cause of a policy mismatch, follow these steps:
  1. Make sure that the authentication methods are compatible.
  2. Make sure that at least one compatible security method is correctly configured.
  3. If you use IPsec tunneling, make sure that the tunnel endpoint settings are correct. Also make sure that the endpoint computers are functioning correctly.

    Note The tunnel endpoint settings include settings for ISAKMP/Oakley, the IPsec Policy Agent, and the IPsec driver.
If "Wrong IPsec policy location" appears in the event log, examine the last lines in the policy agent log. The log may indicate the location of the policy that was used. If the policy location is not logged, follow these steps:
  1. Click Start, click Run, type cmd, and then click OK.
  2. Type the following command, and then press ENTER:

    findstr %systemroot%\ipsecpa.log

    Note This command is case-sensitive.
The log indicates whether Group Policy settings or local computer policy settings are used.

Restarting the policy agent

When you restart the policy agent, you remove old or nonsecure security associations. Restart the policy agent if IP Security Monitor does not show any security negotiations. Also restart the policy agent if you want to download a policy from the domain or from the policy store.

Verifying policy integrity

Active Directory assumes that the most recent changes are current. However, if multiple administrators try to change a policy at the same time, the links between policy components may break. A policy integrity check resolves this problem by verifying the links in all IPsec policies. Run an integrity check after any modifications are made to a policy. To test IPsec policy integrity, follow these steps:
  1. Click Start, click Run, type secpol.msc and then click OK
  2. Right-click IP Security Policies on the Local Machine, point to All Tasks, and then click Check Policy Integrity.

Reviewing the IPsec driver and policy agent registry settings

The settings for the IPsec driver are located in the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec

You can modify the values of the following entries:
  • SAIdleTime

    This REG_DWORD entry configures the Security Association Idle Timer. The default value is 300 seconds. You can specify a value of 300 to 3600 seconds.
  • CacheSize

    This REG_DWORD entry configures the IP header-based cache size. The default value is 64 KB. You can specify a value of 64 to 1024 KB.
  • SAHashSize

    This REG_DWORD entry configures the size of the SPI. It also configures the destination table for inbound security associations. The default value is 64 KB. You can specify a value of 64 to 1024 KB.
The settings for the policy agent are located in the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent

You can modify the values of the following entries:
  • Debug

    The data type is REG_DWORD. The default value is 0. A value of 1 turns on logging. When logging is turned on, the Ipsecpa.log file is created in the %system root%\Debug folder.
  • Log

    The data type is REG_SZ. This entry specifies the name of the log file to open when the Debug entry is set to 1.
The global IPsec policy references are located in the following registry subkeys:
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\Policy\GPTIPSECPolicy
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\IPsec\GPTIPSECPolicy
  • HKEY_CURRENT_USER\SOFTWARE\Microsoft\GroupPolicyObjects\{GUID}Machine\Software\Policies\Microsoft\Windows\IPsec\GPTIPSECPolicy
  • HKEY_CURRENT_USER\SOFTWARE\Microsoft\GroupPolicyObjects\{GUID}Machine\System\CurrentControlSet\Services\PolicyAgent\Policy\GPTIPSECPolicy
  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\IPsec\GPTIPSECPolicy

REFERENCES

For more information about IPsec, visit the following Microsoft Web sites:

http://www.microsoft.com/technet/prodtechnol/windows2000serv/howto/ispstep.mspx

231585 Overview of secure IP communication with IPsec in Windows 2000

For more information about Layer 2 Tunneling Protocol (L2TP)/IPsec connections, click the following article number to view the article in the Microsoft Knowledge Base:

248750 Description of the IPSec policy created for L2TP/IPSec

Modification Type:MajorLast Reviewed:3/23/2006
Keywords:kbinfo kbIPSec kbnetwork kbtshoot KB257225 kbAudGeneralUser
©2004 Microsoft Corporation. All rights reserved.