How to Use Internet Protocol Security to Secure network traffic between two hosts in Windows 2000 (301284)


The information in this article applies to:

  • Microsoft Windows 2000 Server
  • Microsoft Windows 2000 Advanced Server
  • Microsoft Windows 2000 Professional
This article was previously published under Q301284

SUMMARY

This article is a step-by-step instruction guide to enable advanced users to configure Internet Protocol security (IPSec) so that they can secure the communications between two host computers.

IPSec Terminology

Ensure that you know what the following terms mean before you perform the following instructions:
  • Authentication: The process to determine if the identity of a computer is legitimate. Windows 2000 IPSec supports three kinds of authentication: Kerberos, certificates, and preshared keys. Kerberos authentication can work only if both endpoints (computers) are in the same Windows 2000 domain. This type of authentication is the preferred method. If the computers are in different domains, or at least one of them is not in a domain, you must use either certificates or preshared keys. Certificates can work only if each endpoint contains a certificate that is signed by an authority that the other endpoint trusts. Preshared keys have the same problems that passwords do: They do not remain secret for a very long period of time. If the endpoints are not in the same domain and you cannot obtain certificates, preshared keys are your only authentication option.
  • Encryption: The process of making data indistinct in preparation for transmission between two endpoints. By using well-tested algorithms, each endpoint constructs and exchanges cryptographic keys. The process ensures that only the endpoints know the keys; and if any key-exchange sequences are intercepted, the interceptor obtains nothing of value.
  • Filter: A description of the Internet Protocol (IP) addresses and protocols that can trigger the establishment of an IPSec security association.
  • Filter action: The security requirements that can be enabled when the traffic matches the filters in a filter list.
  • Filter list: A collection of filters.
  • Internet Protocol security policy: The collection of rules that describe how communications between computers are secured.
  • Rule: The link between a filter list and a filter action. When the traffic matches a filter list, the corresponding filter action can be triggered. An IPSec policy can contain multiple rules.
  • Security association: The collection of authentication and encryption methods that the endpoints negotiate to establish a secure session.

Find IPSec in Microsoft Management Console

You configure IPSec by using Microsoft Management Console (MMC). Windows 2000 creates an MMC with the IPSec snap-in during the installation process. To locate IPSec, click Start, point to Programs, click Administrative Tools, and then click Local Security Policy. In the MMC that opens, click IP security policies on local machine in the left pane. Then, MMC displays the existing default policies in the right pane.

Change the IP Address, Computer Names, and User Names

For the purposes of this example, Alice is a user that has a computer named "Alicepc" with IP address 172.16.98.231 and Bob has a computer named "Bobslap" with IP address 172.31.67.244. They connect their computers by using the Abczz program.

Alice and Bob must ensure that the traffic is encrypted when they directly connect to each other by using the Abczz program. When Abczz makes its connection, the initiator uses a random high port on itself and connects (for the purposes of this example) to the destination on port 6667/TCP or 6668/TCP (where TCP is the abbreviation for Transmission Control Protocol). Typically, these ports are used for Internet Relay Chat (IRC). Because either Alice or Bob can initiate connections, the policy must exist on both ends.

Create the Filter List

The menus for creating IPSec policies are accessible if you right-click IP Security Policies in the MMC console. The first menu item is "Create IP security policy." Even though this location may seem to be the place to begin, it is not the correct location. Before you can create a policy and its associated rules, you need to define filter lists and filter actions, which are necessary components of any IPSec policy. Begin your work by clicking Manage IP filter lists and filter actions.

The dialog box that is displayed has two tabs: One for filter lists and the other for filter actions. First, the Manage IP filter lists tab opens. There are already two predefined filter lists that you do not use. Instead, you can create a specific filter list that corresponds to the other computer that you want to connect to.

Assume that you create the policy on the computer that belongs to Alice:
  1. Click Add to create a new filter list. Name the list "Abczz to Bob's PC".
  2. Click Add to add a new filter. A wizard starts.
  3. Click My IP address as the source.
  4. Click a specific IP address as the destination, and then enter the IP address (172.31.67.244) of the computer that belongs to Bob. Alternatively, if the computer that belongs to Bob is registered in the Domain Name System (DNS) or the Windows Internet Name Service (WINS), you can select a specific DNS name, and then enter the name of the computer that belongs to Bob instead, which is "Bobslap".
  5. Abczz uses TCP for its communication, so click TCP as the protocol type.
  6. For the IP protocol ports, click From any port. Click To this port, type: 6667, and then click Finish to complete the wizard.
  7. Repeat the preceding steps, except this time type: 6668 as the port number, and then click Close.
Your filter list contains two filters: One for communications from Alice to Bob on port 6667 (which belongs to Bob) and one on port 6668 (which belongs to Bob). (Bob has both port 6667 and 6668 set up on his computer: One port is for outgoing communication and the other for incoming communication.) These filters are mirrored, which is generally necessary anytime you create an IPSec filter. For every filter that is mirrored, the list can contain (but not display) an exact opposite filter where the source and destination addresses are reversed. Without mirrored filters, IPSec communications is usually unsuccessful.

Create the Filter Action

You have defined the kind of traffic that must be secured. Now you must specify the security mechanism. Click the Manage filter actions tab. There are three defaults that are listed. Rather than using the Require security action, you must create a new action that is more stringent.

To create the new action:
  1. Click Add to create a new filter action. A wizard starts. Name the action "Encrypt Abczz".
  2. For the General option, click Negotiate security, and then click Do not communicate with computers that do not support IPSec.
  3. Click the High for the IP Traffic Security option, and then click Finish to close the wizard.
  4. Double-click the new filter action (which you previously named "Encrypt Abczz").
  5. Click to clear the Accept unsecured communication, but always respond using IPSec check box. This step ensures that the computers must negotiate IPSec before an Abczz packet is sent.
  6. Click Session key perfect forward secrecy to ensure that key material is not reused, click OK, and then click Close.

Create the IPSec Policy

You have obtained the policy elements. Now you can create the policy itself. Right-click the right pane of the MMC, and then click Create IP security policy. When the wizard starts:
  1. Name the policy "Alice's IPSec".
  2. Click to clear the Activate the default response rule check box.
  3. Click Edit properties if it is not selected, and then finish the wizard. The Properties dialog box of the policy opens.
For an IPSec policy to work, it must contain at least one rule that links a filter list to a filter action.

To specify rules in the Properties dialog box:
  1. Click Add to create a new rule. When the wizard starts, click This rule does not specify a tunnel.
  2. Click Local area network (LAN) for the network type.
  3. Click Windows 2000 default (Kerberos V5 protocol) for the authentication method if both the computers of Alice and Bob are in the same Windows 2000 domain. If not, click Use this string to protect the key exchange (preshared key), and then enter a string (use a long string that you can remember and type without making mistakes).
  4. Select the filter list that you created earlier. In this example, the filter list is "Abczz to Bob's PC". Then, select the filter action that you created earlier. In this example, the filter action is "Encrypt Abczz".
  5. Finish the wizard, and then click Close.

Configure the Other Endpoints

Repeat on the computer that belongs to Bob all of the preceding procedures that had been applied to the computer that belongs to Alice. The necessary changes are obvious, for example, "Abczz to Bob's PC" must be changed to "Abczz to Alice's PC".

Assign the Policies

You have defined the policies on both ends. Now you must assign them:
  1. In the Local Security Settings MMC, right-click the policy (Abczz in this example).
  2. Click Assign.
Only one IPSec policy can be assigned at one time, but a single policy can have as many rules as you need. For example, if Alice also needs secure communications with Eve by using a different protocol, you have to create the appropriate filter lists and actions, and then add a rule to the IPSec (which belongs to Alice) that links together that specific filter list and filter action. Click Use a different shared key for this rule. The policy for Alice now has two rules: One for Abczz communications with Bob and another for the communications with Eve. Because Bob and Eve do not need to communicate securely to each other, the policy for Bob does not have anything added to it, and the policy for Eve contains a single rule for communications with Alice.

Troubleshooting

Use IPSecMon to Test Your Policy

Windows 2000 includes a utility (IPSecMon.exe) that you can use to test whether an IPSec security association is successfully established. To start IPSecMon:
  1. Click Start, and then click Run.
  2. Type: ipsecmon, and then press ENTER.
  3. Click Options.
  4. Change the refresh interval to 1.
You must establish communications from one endpoint to the other. There can be a delay because it takes a few seconds for the endpoints to exchange cryptographic information and complete the security association. You can observe this behavior in IPSecMon. When the endpoints each build their security associations, you can observe an entry in IPSecMon that displays this behavior.

If you expect a security association to be built, but nothing happens, go back and review the filter lists on each endpoint. Ensure that you have received the correct definitions for the protocols that you use as you can easily reverse the source and destination addresses or reverse the ports. You may want to consider the creation of a new filter list that specifies all traffic. Also, you can add a new rule to the policy that uses this filter list, and then disable the existing rule. Perform these steps on both endpoints. Then, you can use the ping command to test connectivity: The ping command can display "Negotiating IP security" during the security association phase, and then display its normal results when the security association is established.

NAT and IPSec Are Incompatible

If there is any Network Address Translation (NAT) between the two endpoints, IPSec does not work. IPSec embeds endpoint addresses as part of the payload. IPSec also uses source addresses when it computes packet checksums before depositing the packets on the wire. NAT can change the source address of outbound packets, and the destination uses the address in the header when it computes its own checksums. The original source-computed checksums, carried in the packets, do not match the destination-computed checksums, and the destination can drop the packets. You cannot use IPSec with any type of NAT device.

REFERENCES

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

313390 How to use IPSec IP filter lists in Windows 2000

Modification Type:MajorLast Reviewed:2/26/2006
Keywords:kbHOWTOmaster KB301284 kbAudITPro
©2004 Microsoft Corporation. All rights reserved.