Error message in a Windows Server 2003-based domain or in a Windows 2000 Server-based domain: "The remote procedure call failed and did not run" (911799)

SYMPTOMS

When a remote procedure call (RPC) fails in a Microsoft Windows Server 2003-based domain or in a Microsoft Windows 2000 Server-based domain, you may receive one or more of the following error messages:

You receive the following error message when the RPC service can connect to port 135, but subsequent RPC calls fail:
The remote procedure call failed and did not run
The Active Directory directory service replication logs an event that similar to the following event in the Directory Service log:Event Type: Warning
Event Source: NTDS Replication
Event Category: DS RPC Client
Event ID: 1232
Date: MM/DD/YYYY
Time: HH:MM:SS AM|PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: Computer_Name
Description:
Active Directory attempted to perform a remote procedure call (RPC) to the following server. The call timed out and was cancelled.

Server: GUID._msdcs.example.com
Call Timeout (Mins): 5
Thread ID: 984

Additional Data: Internal ID: 5000a96

For more information, see Help and Support Center at http://support.microsoft.com.
When you run the Domain Controller Diagnostic Tool (Dcdiag.exe) or the repadmin /showreps command, the output may include the following error message:
The replication generated an error (1727): The remote procedure call failed and did not execute.

Additionally, the Active Directory Knowledge Consistency Checker (KCC) process may fail. When the KCC process fails, the following events are logged in the Directory Service log:

Event message 1

Event Type: Warning
Event Source: NTDS KCC
Event Category: Knowledge Consistency Checker
Event ID: 1925
Date: MM/DD/YYYY
Time: HH:MM:SS AM|PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: Computer_Name
Description:
The attempt to establish a replication link for the following writable directory partition failed.

Directory partition: DC=Computer_Name,DC=Domain_Name,DC=com
Source domain controller: CN=NTDS Settings,CN=DomainController_Name,CN=Server_Name,CN=site01,CN=Sites ,CN=Configuration,DC=cas,DC=net Source domain controller address: GUID._msdcs.example.com Intersite transport (if any): CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=Domain_Name,DC=com

This domain controller will be unable to replicate with the source domain controller until this problem is corrected.

User Action: Verify if the source domain controller is accessible or network connectivity is available.

Additional Data: Error value: 1727 The remote procedure call failed and did not execute.

Event message 2

Event Type: Warning
Event Source: NTDS KCC
Event Category: Knowledge Consistency Checker
Event ID: 1265
Date: MM/DD/YYYY
Time: HH:MM:SS AM|PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: Computer_Name
Description:
The attempt to establish a replication link with parameters Partition: DC=Computer_Name,DC=Domain_Name,DC=comSource DSA DN: CN=NTDS Settings,CN=DomainController_Name,CN=Server_Name,CN=site01,CN=Sites,CN=Configuration ,DC=domain,DC=com
Source DSA Address: GUID._msdcs.example.com Inter-site Transport (if any): CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=Domain_Name,DC=com failed with the following status:

The remote procedure call failed and did not execute.

The record data is the status code. This operation will be retried.
Additionally, an event that resembles the following may be logged in the System log:Event Type: Error
Event Source: NETLOGON
Event Category: None
Event ID: 5719
Date: MM/DD/YYYY
Time: HH:MM:SS AM|PM
User: N/A
Computer: Computer_Name
Description:
This computer was not able to set up a secure session with a domain controller in domain Domain_Name due to the following:
The remote procedure call failed and did not execute.

This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.

CAUSE

Cause 1

The RPC failure that is reported in error 1727 may occur because RPC needs a port that is blocked. Faulty packets, port filtering, and firewall rules may block a port when the firewall or the network router is configured incorrectly.

Error 1727 indicates that the local domain controller cannot reach the replication RPC process of a replication partner domain controller. At the same time, the domain controller can reach its end-point mapper. If the domain controller cannot reach its end-point mapper, you receive the following error message:

RPC Server is unavailable

Note The universally unique identifier (UUID) for the replication RPC process is E3514235-4B06-11D1-AB04-00C04FC2DCD2.

Cause 2

The RPC failure may occur because Windows Server 2003 Service Pack 1 adds a new RPC element. This new RPC element is known as "multiple transfer syntax negotiation." With multiple transfer syntax negotiation, the clients and the servers can discover and can negotiate their capabilities during the bind time instead of during the first request in the verification trailer.

However, some firewalls, some routers, and some virtual private networks (VPNs) may not recognize this change to the RPC protocol. If the frames are not recognized, the routers, the firewalls, and the VPN connections drop the new RPC Bind request frames. Therefore, any operation that requires an RPC Bind request may now fail if the RPC caller is a Windows Server 2003 Service Pack 1-based server.

The following products may experience this issue:

Microsoft Internet Security and Acceleration (ISA) Server 2004
ISA Server 2000
Products from Check Point Software Technologies
Products from WatchGuard Technologies

WORKAROUND

To work around the RPC failure, use one of the following methods.

Method 1: Troubleshoot cause 1

Review the event log for related error messages

If any events are logged for the RPC failure, review the event logs of the replication partner for related error messages. If the domain controllers in the domain are separated by a firewall, the firewall may be blocking the dynamic ports that are used for Active Directory replication. By default, these dynamic ports start with 1024.

Verify the RPC connectivity by capturing network traffic

To verify the RPC connectivity, follow these steps to capture network traffic:

Capture a network trace on both replication partner domain controllers at the same time.
Try to synchronize replication from one of the domain controllers.
Verify that the replication initiator sends the RPC Bind request on the replication RPC process.
Verify that the RPC Bind request arrives at the replication partner subnet.
Verify that the replication partner sends the RPC Bind Ack answer to the replication partner subnet.

If one of these frames is not sent to the replication partner subnet, check the configuration of the firewall.

Method 2: Troubleshoot cause 2

ISA Server

If the computer is running ISA Server 2004 Standard Edition or ISA Server 2000, the ISA Server RPC filter may block the RPC-based operations. For more information about how to work around this problem, click the following article number to view the article in the Microsoft Knowledge Base:

887222 The ISA Server RPC filter blocks RPC traffic after Windows Server 2003 Service Pack 1 is installed on a computer that is running ISA Server 2004 or ISA Server 2000

Firewall product or VPN product

If the RPC-based operations fail across a VPN or across a firewall immediately after you install Windows Server 2003 Service Pack 1, contact the firewall vendor or the VPN vendor to see whether an updated RPC filter is available. For information about how to contact the firewall vendor or the VPN vendor, click the appropriate article number in the following list to view the article in the Microsoft Knowledge Base:

65416 Hardware and software vendor contact information, A-K

60781 Hardware and software vendor contact information, L-P

60782 Hardware and software vendor contact information, Q-Z

For more information about how to work around this problem, click the following article number to view the article in the Microsoft Knowledge Base:

899148 Some firewalls may reject network traffic that originates from Windows Server 2003 Service Pack 1-based computers

If the RPC-based operations are blocked by filters on a Check Point Software Technologies product, see Check Point Software Secure Knowledge article SK30784, or visit the following Check Point Software Technologies Web site:

http://www.checkpoint.com

The information and the solution in this document represents the current view of Microsoft Corporation on these issues as of the date of publication. This solution is available through Microsoft or through a third-party provider. Microsoft does not specifically recommend any third-party provider or third-party solution that this article might describe. There might also be other third-party providers or third-party solutions that this article does not describe. Because Microsoft must respond to changing market conditions, this information should not be interpreted to be a commitment by Microsoft. Microsoft cannot guarantee or endorse the accuracy of any information or of any solution that is presented by Microsoft or by any mentioned third-party provider.

Microsoft makes no warranties and excludes all representations, warranties, and conditions whether express, implied, or statutory. These include but are not limited to representations, warranties, or conditions of title, non-infringement, satisfactory condition, merchantability, and fitness for a particular purpose, with regard to any service, solution, product, or any other materials or information. In no event will Microsoft be liable for any third-party solution that this article mentions.

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, regarding the performance or reliability of these products.