MSDTC fails to mutually authenticate when computers do not run in the same domain (827805)

Important This article contains information about how to modify the registry. Make sure to back up the registry before you modify it. Make sure that you know how to restore the registry if a problem occurs. For more information about how to back up, restore, and modify the registry, click the following article number to view the article in the Microsoft Knowledge Base:

256986 Description of the Microsoft Windows registry

SYMPTOMS

When two computers that are not running in the same domain use Microsoft Distributed Transaction Coordinator (MSDTC) for communication and for distributed transactions, the mutual authentication may fail.

When you run a distributed transaction against a linked server in Microsoft SQL Server 2000 on a computer that is running Microsoft Windows Server 2003 Enterprise Edition, you may receive the following error message:

Server: Msg 7391, Level 16, State 1, Line 2
The operation could not be performed because the OLE DB provider 'SQLOLEDB' was unable to begin a distributed transaction. OLE/DB provider returned message: New transaction cannot enlist in the specified transaction coordinator.

WORKAROUND

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

For information about how to edit the registry, view the "Change Keys and Values" Help topic in Registry Editor (Regedit.exe) or the "Add and Delete Information in the Registry" and "Edit Registry Data" Help topics in Regedt32.exe. Microsoft recommends that you back up the registry before you edit it.

To work around this problem, turn off the RPC security on your Windows Server 2003 computers. RPC security is a new DTC feature in Windows Server 2003. When you disable RPC security, the DTC authentication security level for RPC calls goes back to a level that is available in Microsoft Windows 2000 Server. To do this, follow these steps to set the DWORD value for the TurnOffRpcSecurity registry value to 1:

Start Registry Editor (Regedt32.exe).
Locate the following key in the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC
On the Edit menu, click Add Value, and then add the following registry value:
Value name Data type Value
TurnOffRpcSecurity REG_DWORD 1
Quit Registry Editor.

Note After you install Windows Server 2003 Service Pack 1, the registry entry TurnOffRpcSecurity is set to 0. You have to restore the value to 1 by using Registry Editor before you restart the computer.

MORE INFORMATION

Steps to reproduce the behavior

Make sure that both computers are running Windows Server 2003.
Install SQL Server 2000 with Service Pack 3 (SP3) on both the computers.
Make sure that MSDTC is started on both the computers.
On the first computer, start the SQL Query Analyzer (Isqlw.exe) utility, and then connect to the local SQL Server.
Add the second computer as the linked server. To do so, run the following Transact-SQL statement in SQL Query Analyzer:
```
EXEC sp_addlinkedserver  'remote_server',  N'SQL SERVER'
GO
```
Note Replace remote_server with the name of the second computer.
Run the following Transact-SQL statement in SQL Query Analyzer:
```
SET xact_abort ON 
GO
USE  pubs
GO
BEGIN DISTRIBUTED TRANSACTION
SELECT  *  FROM remote_server.pubs.dbo.authors
COMMIT TRAN
GO
```
You may receive the error message that is mentioned in the "Symptoms" section of this article.

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

329332 You receive error 7391 when you run a distributed transaction against a linked server

For more information about Microsoft COM+ 1.5, visit the following Microsoft Developer Network (MSDN) Web site:

http://msdn.microsoft.com/library/en-us/cossdk/htm/whatsnewcomplus_350z.asp