How to troubleshoot a memory leak or an out-of-memory exception in the BizTalk Server process (918643)

MORE INFORMATION

The BizTalk Server process may be experiencing a memory leak when memory usage in Microsoft Windows Task Manager consumes more than 50 percent of the physical RAM. A memory leak may cause an out-of-memory exception when memory usage increases until the process runs out of system memory or until the process stops functioning.

Note In Microsoft BizTalk Server 2006 and in BizTalk Server 2004, the BizTalk Server process is the Btsntsvc.exe process. In BizTalk Server 2002 and in BizTalk Server 2000, the BizTalk Server process is the Mscis.exe process.

When this problem occurs, a warning message that resembles the following message is logged in the event log: Event Type: Warning
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 5410
Date: 3/6/2006
Time: 5:00:11 PM
User: N/A
Computer: ComputerName
Description: An error has occurred that requires the BizTalk service to terminate. The most common causes are an unexpected out of memory error and an inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.
Error message: Exception of type System.OutOfMemoryException was thrown.
Error source:
BizTalk host name: BizTalkServerApplication
Windows service name: BTSSvc{DCC899FE-C62F-41BE-851A-8720B2EB9C14}

Important considerations

Physical RAM and memory usage

Because it may be expected behavior for a process to use about half the physical RAM, use the memory usage as a guideline. For example, if the computer that is running BizTalk Server has 2 gigabytes (GB) of RAM, and the BizTalk Server process uses about 500 megabytes (MB) of RAM, there may not be leak. The memory consumption may be caused by a long-running stored procedure or orchestration. Make sure that you know what behavior is expected in your environment to determine whether a memory leak is occurring.

Large messages

When BizTalk Server processes large messages, the system may appear to have a memory leak. However, the messages are just using lots of memory. For more information, visit the following Microsoft Developer Network (MSDN) Web site:Also, consider that high memory usage may be expected if BizTalk Server is processing large messages. You may want to upgrade your hardware to meet the performance requirements of BizTalk Server in your environment.

How long it takes to reproduce the memory leak

Memory leaks can occur immediately or they may accumulate over time. Both scenarios are common.

Use of the /3GB switch

When the /3GB switch is enabled, a process can access 3 GB of virtual address space that is reserved for user mode operation. (Typically, a process can access 2 GB of virtual address space.) The /3GB switch is an option for systems that require more addressable memory. This option may improve memory usage for processing messages. However, the /3GB switch allows for only 1 GB of addressable memory for kernel mode operations. Additionally, this switch may increase the risk of running out of pool memory.

The /3GB switch is supported in BizTalk Server 2006. If you must enable the /3GB switch to meet the performance requirements of a computer that is running BizTalk Server 2006, you may want to consider adding servers to your environment. The /3GB switch is not supported in BizTalk Server 2004 or in BizTalk Server 2002. If you are concerned about user mode memory on a computer that is running BizTalk Server 2002 or BizTalk Server 2004, you may want to consider using multiple hosts.

BizTalk components that run inside a Microsoft Internet Information Services (IIS) process may also benefit when the /3GB switch is enabled. These components include IIS 6.0 in Windows Server 2003 , IIS 5.1 in Windows XP, IIS 5.0 Isolation Mode in Windows Server 2003, and IIS 5.0 in Microsoft Windows 2000.

Note Additionally, the PAE switch is not supported in BizTalk Server 2006 or in BizTalk Server 2004.

Use of custom components

If you are using custom components, such as pipelines, Application Integration Components (AIC) for BizTalk Server 2002, or service components, together with BizTalk Server, you must know what these components are doing. You must also know the potential effect of these components on memory usage. A common memory problem occurs when a component is transforming a document. The transform operation is a memory-intensive operation. When a document is transformed, BizTalk Server passes the message stream to the Microsoft .NET Framework XslTransform class in BizTalk Server 2006 and in BizTalk Server 2004. BizTalk Server passes the message stream to the XML Document Object Model (DOM) in BizTalk Server 2002 and in BizTalk Server 2000.

Another common issue occurs when there is intensive string manipulation. Intensive string manipulation can consume lots of memory. For more information about ways to improve performance, visit the following MSDN Web site:

Version of the .NET Framework

The Microsoft .NET Framework 2.0 and the .NET Framework 1.1 have different memory behavior. Therefore, you may see varying results between them. If you are using the .NET Framework 1.1, confirm that the .NET Framework 1.1 Service Pack 1 (SP1) is installed. SP1 addresses many memory issues. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

Number of processors

The common language runtime (CLR) has the following garbage collectors (GCs):

• Workstation (Mscorwks.dll)
• Server (Mscorsvr.dll)

If the computer that is running BizTalk Server is a multiprocessor system, the .NET Framework uses the Server version of the execution engine. This is the default behavior. The Server garbage collector is designed for maximum throughput. Additionally, the Server garbage collector scales to provide very high performance. This garbage collector allocates memory and then later frees memory to provide high performance on the system. Therefore, a computer that is running BizTalk Server together with some .NET Framework components may appear to have a memory leak. However, in this scenario, high memory usage is the expected behavior. If the computer runs out of system memory, or if the process stops working because of insufficient addressable memory, a memory leak condition may exist.

If the computer that is running BizTalk Server is a single processor system, the .NET Framework uses the Workstation version of the execution engine. This is the default behavior. The Workstation garbage collector allocation algorithm is not designed for scaling or for maximum throughput. This garbage collector uses concurrent garbage collector methods. These methods are designed for applications that have complex user interfaces. Such applications may require more aggressive garbage collection.

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

Sometimes, it may be appropriate to run the Workstation version of the execution engine on a multiprocessor system. You can use the following registry key to switch to the Workstation version of the execution engine:For more information, visit the following MSDN Web sites:

Known issues

BizTalk Server 2004

If you are using BizTalk Server 2004, you may experience some memory-related issues. To resolve these issues, apply the hotfixes that are described in the following Microsoft Knowledge Base (KB) articles:

BizTalk Server 2002 Service Pack 1

If you use Microsoft BizTalk Server 2002 Service Pack 1 (SP1), you may experience some memory-related issues. To resolve these issues, apply the hotfix that is described in the following KB article:Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

If the KB885223 hotfix does not resolve the issue, try changing the value of the HeapDeCommitFreeBlockThreshold registry key. By default, the value of this key is 0. A value of 0 means that the heap manager decommits each 4-kilobyte (KB) page that becomes available. Decommit operations can cause virtual memory fragmentation. The size of the HeapDeCommitFreeBlockThreshold setting in the heap manager will depend on the kind of work that the system is doing. A size of 0x00040000 is a recommended starting value.

Consider the following information before you change the value of the HeapDeCommitFreeBlockThreshold registry key:

• This change only applies to memory fragmentation problems.
• This change is system-wide. Therefore, most processes will use more memory on startup.
• Only consider this change for systems that have BizTalk Server as their primary mission.

To help reduce virtual memory fragmentation, you can increase the size of the HeapDeCommitFreeBlockThreshold setting in the heap manager by changing the value of the following registry key:For more information about the HeapDeCommitFreeBlockThreshold registry key, click the following article number to view the article in the KB:

Common causes and resolutions in BizTalk Server 2006 and in BizTalk Server 2004

Transform operations

When BizTalk Server 2006 and BizTalk Server 2004 perform XML transform operations on relatively large messages in a receive port, in a send port, or in XLANG, XSL transforms load the whole message in memory.

To resolve this issue, use one of the following methods:

• Decrease the number of messages that BizTalk Server processes at the same time.
• Decrease the size of the XML message that is being transformed.

BizTalk Server 2006 significantly improves memory management for large documents. To do this, BizTalk Server implements a configurable message size threshold for loading documents into memory during transform operations. The default message size threshold is 1 MB. For more information, visit the following MSDN Web site:

Large attribute values and large element values

When BizTalk Server 2006 and BizTalk Server 2004 execute a receive pipeline or a send pipeline on an XML document, the payload is processed in memory if the document contains one or more of the following entities:

• Large attribute values
• Large element values
• Large attribute or element tags

To resolve this issue, limit the size of these entities. If this method is not possible, make sure that your process does not process multiple documents such as these at the same time.

Custom pipeline components

You are using a custom pipeline component that loads the whole stream into memory. All the components that are included with BizTalk Server, except transforms, support streaming. These components do not use as much memory during streaming. However, custom pipeline components may not support streaming.

Streaming under heavy stress

Send hosts run out of memory when they operate under heavy stress. BizTalk Server send pipelines and send adapters support streaming. In streaming, each component loads a small fragment of the stream into memory. Because each message includes other data structures, together with a message context that can be big or small, this behavior affects the behavior of BizTalk Server under heavy stress.

The behavior of BizTalk Server is affected because the engine loads a pre-configured number of messages. The number of messages that the engine loads is based on the values that appear in the LowWaterMark field and the HighWaterMark field of the Adm_serviceClass table. The Adm_serviceClass table is in the BizTalk Management Database. These values control the number of messages that BizTalk Server processes or sends at the same time.

The HighWaterMark value is the total number of messages that the engine processes at the same time. The default value is 200 messages per CPU. Therefore, on an 8-processor server, the send host will try to process 1,600 messages (200*8) at the same time. If you assume that each message is 50 KB, the messages equal 80 MB (1,600*50=80,000KB).

To resolve this issue, you can change the HighWaterMark value and the LowWaterMark value in the database. The values that you use depend on the size of the messages.

For more information about common causes of an out-of-memory condition, see the "Memory Growth in BizTalk Messaging" section at the following Microsoft Web site:For BizTalk Server 2006, you can change the default host throttling settings. For more information about how to change the default host throttling settings, visit the following MSDN Web site:

Try to simplify the issue

If you have identified a memory leak, try to determine the cause by removing custom components or by simplifying a map. Also, try to reproduce the issue by using a simple orchestration or a simple solution. Typically, you should create separate receive hosts for receive adapters. You should also create separate send hosts for send adapters. When you use this method, each adapter can run in a separate process. Therefore, if your BizTalk Server host or your BizTalk Server process experiences an out-of-memory condition, you will know which components are involved.

Troubleshooting steps

To troubleshoot an out-of-memory condition, use the Debug Diagnostics tool to monitor memory allocations over time. The Debug Diagnostics tool can create and analyze a memory leak dump file (.dmp). When you troubleshoot memory leaks, the goal is to attach Leaktrack.dll before the high memory condition reproduces to capture memory growth over time. Leaktrack.dll is included with the Debug Diagnostics tool.

1. Install the Debug Diagnostics tool. To obtain the Debug Diagnostics tool, visit the following Microsoft Web site:

   http://www.microsoft.com/windowsserver2003/iis/diagnostictools/default.mspx

   The Debug Diagnostics tool is part of the Microsoft Internet Information Services (IIS) Diagnostics Toolkit. To install the Debug Diagnostics tool, perform a custom installation. Select only the Debug Diagnostics tool. By default, the installation path of the Debug Diagnostics tool is as follows:

   C:\Program Files\IIS Resources\DebugDiag

   However, you can change the installation path during the installation process.
2. Use Performance Monitor to collect data about system performance. This data may provide important indicators about the efficiency of your BizTalk Server environment. The goal is to capture process performance over time. Therefore, enable Performance Monitor logging before the memory leak occurs.

How to use Performance Monitor logging

Select the data to logTo select the data to log, use the method that is appropriate for your operating system:

• For Windows Server 2003 or for Windows XP
  1. Expand Performance Logs and Alerts.
  2. Right-click Counter Logs, and then click New Log Settings. The New Log Settings dialog box appears.
  3. In the Name box, type a descriptive name, and then click OK.
  4. Note the log file location. (You can also click the Log Files tab, and then click Configure to change the log file location.)
  5. Click Add Counters.
  6. Select All counters and All instances.
  7. In the Performance object list, select the following objects. Click Add after you select each object.
     • .Net CLR Exceptions
     • .Net CLR Memory
     • BizTalk:Messaging
     • BizTalk:TDDS
     • Memory
     • Process
     If SQL Server is local, also add the following objects:
     • SQLServer:Databases
     • SQLServer:General Statistics
     • SQLServer:Memory Manager
  8. Click Close.
  9. Change the value in Data Sampling Interval to 5 seconds.
     
     Note The Data Sampling Interval value and the time to start monitoring are subjective. These values depend on when the memory leak is reproduced. Because the log file can be large, specify an interval in which you can obtain the information that you require without overwhelming the server.
  10. Click OK.
• For Windows 2000 Server
  1. Expand Performance Logs and Alerts.
  2. Right-click Counter Logs, and then click New Log Settings. The New Log Settings dialog box appears.
  3. In the Name box, type a descriptive name, and then click OK.
  4. Note the log file location. (You can also click the Log Files tab to change the log file location.)
  5. Click Add.
  6. Select All counters and All instances.
  7. In the Performance object list, select the following objects. Click Add after you select each object.
     • .Net CLR Exceptions
     • .Net CLR Memory
     • BizTalk:Messaging
     • BizTalk:TDDS
     • Memory
     • Process
     If SQL Server is local, also add the following objects:
     • SQLServer:Databases
     • SQLServer:General Statistics
     • SQLServer:Memory Manager
  8. Click Close.
  9. Change the value in Data Sampling Interval to 5 seconds.
     
     Note The Data Sampling Interval value and the time to start monitoring are subjective. These values depend on when the memory leak is reproduced. Because the log file can be large, specify an interval in which you can obtain the information that you require without overwhelming the server.
  10. Click OK.

Obtain the dump fileTo obtain the dump file, use one of the following methods:

• Method 1: Automatic
  If you know when the memory leak will occur, you can create a Memory and Handle Leak Rule. The Memory and Handle Leak rule automatically attaches Leaktrack.dll. When the process consumes the amount of memory that you have specified, a dump file is created.
  1. Click Start, point to Programs, point to IIS Diagnostics (32 bit), point to Debug Diagnostics Tool, and then click Debug Diagnostics Tool 1.0.
  2. Select Memory and Handle Leak, and then click Next.
  3. Select Btsntsvc.exe or Mscis.exe, and then click Next.
  4. On the Configure Tracking Duration page, you can specify a warm-up time. However, if the issue reproduces immediately, select the Start memory tracking immediately when rule is activated check box. For Tracking time, specify the time that will be required to reproduce the issue. At the end of the specified time, a dump file will be created. Make sure that the Auto-create a crash rule to get userdump on unexpected process exit check box is selected.
  5. Click Next.
  6. On the Select Dump Location And Rule Name page, click Finish. (You can also change the path of the dump file in the Userdump Location box on this page.)
     
     Note The status is now active. Every time that a dump file is created, the value will increase in the Userdump Count column on the Rules tab in the Debug Diagnostics tool. The default dump file location is C:\Program Files\IIS Resources\DebugDiag\Logs.
• Method 2: Manual
  If you do not know when the memory leak will occur, you can manually attach Leaktrack.dll and manually obtain the dump file manually. To do this, follow these steps:
  1. Click Start, point to Programs, point to IIS Diagnostics (32 bit), point to Debug Diagnostics Tool, and then click Debug Diagnostics Tool 1.0.
  2. Click the Processes tab.
  3. On the Processes tab, right-click Btsntsvc.exe or Mscis.exe, and then click Monitor For Leaks. The Debug Diagnostics Tool dialog box appears.
  4. In the Debug Diagnostics Tool dialog box, click Yes, and then click OK.
  5. When the process reaches 60 percent to 80 percent of RAM, right-click Btsntsvc.exe or Mscis.exe, and then click Create Full Userdump.

Stop Performance Monitor loggingStop Performance Monitor logging about two minutes after the dump file of the process has been created. To do this, follow these steps:

1. Expand Performance Logs and Alerts.
2. For each counter log that you created in the "Select the data to log" section, right-click the name of the counter log, and then click Stop.

Analyze the dump fileTo determine the cause of a memory leak, you can use the Debug Diagnostics tool to analyze the dump file. To do this, follow these steps:

1. Click the Advanced Analysis tab.
2. Click Add Data Files, and then locate the .dmp file.
3. Select the Memory Pressure Analysis script, and then click Start Analysis.

When the analysis is finished, an analysis report file (.mht) will be created in the C:\Program Files\IIS Resources\DebugDiag\Reports folder. The report file will also be displayed in your browser. The report file contains the results of the analysis. Additionally, the report file contains recommendations for how to resolve the memory leak.

If you use custom DLLs, you can add the symbol path of the custom .pdb files for analysis. To do this, follow these steps:

1. Open the Debug Diagnostics tool.
2. On the Tools menu, click Options and Settings.
3. In the Symbol Search Path For Debugging box, type the symbol path.

If you want help analyzing the dump file, contact Microsoft Product Support Services. For a complete list of Product Support Services telephone numbers and information about support costs, visit the following Microsoft Web site: Before you contact Product Support Services, compress the dump file, the Performance Monitor log, the analysis report file, and the updated event logs (.evt files). You may have to send these files to a BizTalk Server support engineer.