Remote Assistance connection to Windows Server 2003 with FIPS encryption does not work (811770)


The information in this article applies to:

  • Microsoft Windows Server 2003, Web Edition
  • Microsoft Windows Server 2003, 64-Bit Datacenter Edition
  • Microsoft Windows Server 2003, Datacenter Edition
  • Microsoft Windows Server 2003, 64-Bit Enterprise Edition
  • Microsoft Windows Server 2003, Enterprise Edition
  • Microsoft Windows Server 2003, Standard Edition
  • Microsoft Windows XP Professional

SYMPTOMS

Microsoft has added the FIPS Compliant setting to the options for Terminal Services encryption levels in Windows Server 2003. A Windows Server 2003-based server with the encryption level set to FIPS Compliant cannot permit Remote Assistance connections from a computer that is running Microsoft Windows XP or Windows XP Service Pack 1 (SP1).

When you try to connect from a Windows XP-based client to a Terminal Services server, the connection may not succeed, and you may receive the following error message:
Because of a security error, the client could not connect to the terminal server. After making sure that you are logged on to the network, try connecting to the server again.

CAUSE

Windows XP and Windows XP SP1 do not support the FIPS Compliant encryption level, and therefore cannot connect to the Windows Server 2003-based server for remote assistance.

RESOLUTION

Federal Information Processing Standard (FIPS) support on the client is handled in the Terminal Services client, not by the operating system. Upgrading to the RDP 5.2 (Windows Server 2003) client permits a client that is running Windows XP or Windows XP SP1 to connect to a Terminal Services session on a Windows Server 2003-based server that is configured for FIPS-compatible encryption, but does not permit the Remote Assistance client to connect. Clients that are running Windows XP or Windows XP SP1 cannot provide Remote Assistance connections to Windows Server 2003-based computers that are configured to require FIPS-compatible encryption.

WORKAROUND

To work around this problem, disable the FIPS encryption level. To disable the FIPS encryption level, you can change the Encryption level setting in the RDP-Tcp Properties dialog box, or you can use the Group Policy Object to disable FIPS data encryption system-wide. To disable the FIPS encryption level, use one of the following methods:

Note There are two ways to enable the FIPS encryption level. If you have to disable the FIPS encryption level for Terminal Services, you must do this by using the same method that you originally used to enable the FIPS encryption level.

Method 1

To disable the FIPS encryption level by changing the Encryption level setting in the RDP-Tcp Properties dialog box, follow these steps:
  1. Click Start, click Run, type tscc.msc in the Open box, and then click OK.
  2. Click Connections, and then double-click RDP-Tcp in the right pane.
  3. In the Encryption level box, click to select a level of encryption other than FIPS Compliant.

    Note If the Encryption level setting is disabled when you try to change it, the system-wide setting for System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing has been enabled, and you must disable this system-wide setting by using method 2.

Method 2

To use the Group Policy Object to disable FIPS data encryption system-wide, follow these steps:
  1. Click Start, click Run, type gpedit.msc in the Open box, and then click OK.
  2. Expand Computer Configuration, expand Windows Settings, expand Security Settings, expand Local Policies, and then click Security Options.
  3. In the right pane, double-click System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing, click Disable, and then click OK.

    Note Encryption level settings in Terminal Server are unavailable when FIPS is enabled.
For more information about scoping Group Policy Objects and troubleshooting the resultant policies on your server, click the following article number to view the article in the Microsoft Knowledge Base:

818735 White Paper: Administering Group Policy by Using the Group Policy Management Console

For more information about the GPO setting for System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing, click the following article number to view the article in the Microsoft Knowledge Base:

811833 The effects of enabling the "System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing" security setting in Windows XP and later versions

STATUS

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

MORE INFORMATION

The FIPS Compliant setting requires that all data between the client and the server be encrypted by using encryption methods that are validated by Federal Information Processing Standard 140-1. When a Windows XP-based client tries to connect to a Windows Server 2003-based server that requires FIPS-compliant encryption, the following errors occur:
  • On the client, you receive the following error message from Remote Assistance:
    A Remote Assistance connection could not be established. You may want to check for network issues or determine if the invitation expired or was cancelled by the person who sent it.
  • The following error is logged in the System event log on the server:
    Event ID: 50
    Source: TermDD
    Type: Error
    Description: The RDP protocol component "DATA ENCRYPTION" detected an error in the protocol stream and has disconnected the client.
Modification Type:MinorLast Reviewed:9/30/2005
Keywords:kbHotfixServer kbQFE kbPrb KB811770 kbAudITPRO
©2004 Microsoft Corporation. All rights reserved.