Troubleshooting ASP.NET using WinDbg and the SOS extension (892277)


The information in this article applies to:

  • Microsoft ASP.NET (included with the .NET Framework 1.1)
  • Microsoft ASP.NET (included with the .NET Framework 1.0)

ASP .NET Support Voice Column

Troubleshooting ASP.NET using WinDbg and the SOS extension

To customize this column to your needs, we want to invite you to submit your ideas about topics that interest you and issues that you want to see addressed in future Knowledge Base articles and Support Voice columns. You can submit your ideas and feedback using the Ask For It form. There's also a link to the form at the bottom of this column.
Hello again, and welcome to the January 2005 edition of the Support Voice Column. Once again, I would like to thank Jim Cheshire, a support engineer here at Microsoft supporting ASP.NET, for his contributions. Jim has great ideas for the Support Voice column and wanted to continue to share them. Look for Jim's contributions last month and over the next month or so, and as always, send us your suggestions for future columns. Thank you Jim!

Jim has been working with Microsoft for six years on the FrontPage, VB, and ASP.NET teams. During that time, he has written for the Office Developer Center on MSDN, and he is the author of a book on FrontPage, Special Edition Using Microsoft Office FrontPage 2003. Jim also has a Web site where he provides free add-ins for FrontPage to enable Web developers to make the most out of Microsoft products. Here is that Web site address:

http://www.jimcoaddins.com

So please, pull up a chair, kick off your shoes, and read through our column all about troubleshooting ASP.NET ,and remember, you can submit your ideas to us using the "ASK FOR IT" link included in every column we publish.
Jeremy

Troubleshooting ASP.NET using WinDbg and the SOS extension


A good developer will plan for any contingency. Naturally, part of that planning is developing some robust exception handling routines. Many ASP.NET developers will handle exceptions by presenting the user with a nicely formatted Web page, and then logging the exception information in a file for analysis later.

Scenario:

You have developed and tested a new ASP.NET application that logs exceptions into a log file. You deploy the ASPX pages and the assemblies to your production Web server. When people use your application, they are seeing an error page, but no exception logging is written to your log files. You also notice that there are many ASP.NET events in the Application Event Viewer log saying that Aspnet_wp.exe stopped unexpectedly.

This is a common type of problem that we see in Developer Support, and the majority of our customers don't really know how to approach troubleshooting it. Even worse is the dreaded "Server Application Unavailable" error message. This month, I will show you how you can troubleshoot these types of problems with the Debugging Tools for Windows.

Disclaimer: If you call Microsoft Developer Support, we may often ask you to send us a crash dump file or a hang dump file. The analysis of the dump file in this article will be quite easy and fast. In many cases, the cause of a problem is not apparent and it can take many hours of analysis to get to the root cause. Don't let the ease and straight-forwardness of this dump file analysis fool you into thinking that they are all this easy. Most, in fact, are not!

Getting prepared

Before we get started, you'll need to install the Debugging Tools for Windows. Even if you already have a copy installed, make sure that you have the latest version because we will be using an extension for managed debugging that is only included with the most recent version of the Debugging Tools for Windows.

You can download the Debugging Tools for Windows from the following Microsoft Web site:

http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

When you install the Debugging Tools for Windows, you may want to perform a custom installation to the C:\Debuggers folder. We will be running tools from the command line, and if the path is a short one, it will make doing so easier for you. After installation is complete, you will want to download the sample ASP.NET page that will crash the worker process.

http://download.microsoft.com/download/c/a/1/ca1c6329-3b3f-4d9e-a76d-54c78d3ddfbc/crash.exe

Note that in order for you to be able to generate the dump file using this page, you need to be running ASP.NET 1.1 with Service Pack 1 (SP1) installed. Without SP1, this page will run without generating a crash.

You will also want to set up the symbol path for WinDbg. Symbols are not required for function name resolution on managed assemblies, but you will need symbols for native function resolution. To set the symbol path in WinDbg:
  1. Open WinDbg. Click Start, point to All Programs, and then click Debugging Tools for Windows.
  2. On the File menu, click Symbol File Path.
  3. In the Symbol Path dialog box, type the following symbol path:

    srv*c:\symbols*http://msdl.microsoft.com/download/symbols

  4. Close WinDbg, and then type Yes at the prompt to save base workspace information.
Now you're ready to create the crash dump file.

Creating the crash dump file

In this walkthrough, clicking the button on the sample page will cause the ASP.NET worker process to crash. In many cases, you won't know precise steps to generate a crash. In fact, your production application may crash at some random time. As you'll soon see, it doesn't matter whether a crash occurs predictably or randomly. Creating a dump file is quite easy using Adplus, the VBScript file that automates the entire process.

To generate the dump file, you need to first browse to the sample page. (Don't click the button!) After the page has been successfully served, you know that the Aspnet_wp.exe process (or the W3wp.exe process on Windows Server 2003) has been launched. Now you're ready to attach the debugger to the worker process so that you can capture a dump file when it crashes.

To create a dump file:
  1. Open a new command prompt on the web server, and then change to the directory where the Debugging Tools for Windows are installed.
  2. Run the following command:

    cscript adplus.vbs -crash -pn aspnet_wp.exe -o c:\crashdump

    If you are running Microsoft Windows Server 2003, run the following command:

    cscript adplus.vbs -crash -pn w3wp.exe -o c:\crashdump

I won't go into great detail on Adplus.vbs here, but if you'd like more information, you can review the following Microsoft Knowledge Base article:

286350 How to use ADPlus to troubleshoot "hangs" and "crashes"

After you run this command, you will be presented with a couple of dialog boxes. Click OK in each dialog box. You should now see a button on your task bar for the debugger, CDB, the application name for debugger. However, it may be different in your window.

Note You must be at the console on the web server when you run Adplus in crash mode. You cannot run Adplus in crash mode remotely.

Now you're ready to generate a crash! Click the button on the page to crash the worker process. If you have JIT debugging enabled in Visual Studio .NET, a dialog will appear letting you know that an exception occurred but the attached debugger couldn't handle it. If you don't have JIT debugging enabled, the dump file will be created without any dialogs being displayed. In either case, after you see the CDB button disappear from the task bar, the dump file has finished, and you're ready to move on to the next step.

Checking out the dump file

  1. Start WinDbg.
  2. On the File menu, click Open Crash Dump to open the dump file. (Don't click the Open button on the toolbar.)
  3. Browse to the C:\Crashdump folder. In that folder, you will see another folder with a long name that starts with Crash_Mode_Date. Adplus saved the dump files inside of this folder. (You should have more than one dump file in the folder.) The dump file that you want to open in WinDbg is the 2nd chance exception dump file. It should be obvious by the name of the file which dump file is the 2nd chance exception dump file.
After you've opened the dump file, WinDbg will start downloading symbols. Depending on the speed of your Internet connection, this process may take several minutes. When it's complete, you will see the command prompt appear in WinDbg, as shown in the following figure.

WinDBg window

Figure 1 : WinDbg - The "0:000>"at the command prompt indicates the current thread, thread 0.

You can see the faulting stack by typing the kb command at the command prompt. Here is the output from the dump file:
0:000> kb

ChildEBP RetAddr  Args to Child              

00771018 791b6173 00000643 00000000 00b59bc0 mscorwks!_EH_prolog+0x2

00771928 791b60c0 00771a04 00000000 00000000 
mscorwks!EEClass::DoRunClassInit+0xbe

0077193c 791d75a7 00771a04 00771a50 00b59bc0 
mscorwks!MethodTable::CheckRunClassInit+0x1d

00771a14 791d7746 00000000 0086fb88 00771a50 
mscorwks!MethodDesc::DoPrestub+0x28e

00771a2c 00a92f76 00771a50 0180f154 00ee5cd4 mscorwks!PreStubWorker+0x42

WARNING: Frame IP not in any known module. Following frames may be wrong.

00773a38 0337f2bc 00f8463c 00eb593c 00ee5cc8 0xa92f76

00773a64 031cd7e4 00eb593c 0180ea7c 00f84560 0x337f2bc

00773aac 033a3299 00000000 00eb593c 0180ea7c 0x31cd7e4

00773b50 033a3031 00000001 00000000 0180e238 0x33a3299

00773bec 0332ee47 00edb274 0180df00 00f8463c 0x33a3031

00773c28 0332e71b 0180df00 033a35c5 0180d870 0x332ee47

00773cc4 033a3031 00000001 00000000 0180d2b4 0x332e71b

00773d60 0332ee47 00edb274 0180cf7c 00f8463c 0x33a3031

00773d9c 0332e71b 0180cf7c 033a35c5 0180c8ec 0x332ee47

00773e38 033a3031 00000001 00000000 0180c330 0x332e71b

00773ed4 0332ee47 00edb274 0180bff8 00f8463c 0x33a3031

00773f10 0332e71b 0180bff8 033a35c5 0180b968 0x332ee47

00773fac 033a3031 00000001 00000000 0180b3ac 0x332e71b

00774048 0332ee47 00edb274 0180b074 00f8463c 0x33a3031

00774084 0332e71b 0180b074 033a35c5 0180a9e4 0x332ee47


Note If you are running on a multi-processor system (or a hyper-threaded system), you will see function calls in mscorsvr instead of mscorwks.

You can see the function names in Mscorwks.dll because WinDbg downloaded symbols for mscorwks. However, following the calls into mscorwks, you see no module names and no function names. That's what managed code (code that runs under the Common Language Runtime) looks like in WinDbg by default, and it's obviously not helpful output. To see what happened in that managed code, you need to use the SOS extension for WinDbg.

The SOS extension

SOS is an extension for WinDbg that allows you to debug managed code. SOS adds many commands to WinDbg, all of which are focused on debugging managed applications and many are ASP.NET-specific. The extension is a DLL file called Sos.dll, and it's located in the Clr10 folder in the folder where the Debugging Tools for Windows are installed.

To load the SOS extension, type the following command at the command prompt in WinDbg:

.load clr10\sos

Don't skip the period at the beginning of the command! If all goes well, you'll be taken back to the command prompt. If an error occurs, make sure that you have the latest version of the Debugging Tools for Windows installed.

We'll cover a few of the commands from SOS to debug the crash dump file, but feel free to explore the other commands in SOS on your own time. You can use the !help command to get a list of the commands and a brief description of each.

Digging into root cause

Now that SOS has been loaded, let's have a look at some of the commands that are useful in troubleshooting a crash dump file.

Let's look at the managed stack for the current thread, thread 0. We'll do that using the !clrstack command. Type !clrstack at the command prompt. WinDbg starts dumping the stack. A normal stack should dump out completely in just a few seconds. In this case, the output seems to go on forever. Go ahead and press CTRL-BREAK (or click Break on the Debugmenu in WinDbg) and let's look at it. Here's a portion of the stack: 
Void System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)
Void System.Web.HttpServerUtility.Transfer(String,Boolean)
Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)
Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Web.UI.IPostBackEventHandler,String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)
Void System.Web.UI.Page.ProcessRequestMain()
Void System.Web.UI.Page.ProcessRequest()
Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
Void System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)
Void System.Web.HttpServerUtility.Transfer(String,Boolean)
Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)
Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Web.UI.IPostBackEventHandler,String)
Void System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)
Void System.Web.UI.Page.ProcessRequestMain()
Void System.Web.UI.Page.ProcessRequest()
Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext
If you look closely, you'll see a pattern here. We start off processing the request and raising a PostBack. The OnClick event is then fired for a button. That results in a Server.Transfer. The Server.Transfer then seems to start the entire process all over again. In this case, each time this loop executes, objects are pushed onto the stack. Eventually, the system runs out of stack space, and a StackOverflowException occurs.

To get a better idea of what happened here, you can get more detailed output from the stack by running the !clrstack -s command. This gives you the full stack. Here is some of that output. For the full output from this command, create a dump file using the sample page, and then run the command. I have edited it here because the output is quite verbose. 
0:000> !clrstack -s
Thread 0
System.Web.HttpServerUtility.ExecuteInternal(String,Class System.IO.TextWriter,Boolean)
    ESP/REG  Object   Name
    00773ac4 0180e7f4 System.EventArgs
    00773ac8 00f86c14 System.Web.HttpServerUtility
    00773acc 0180e238 System.Web.UI.WebControls.Button
    00773ad0 0180eac0 System.Object[]
    00773ad4 0180ea7c System.String    /crash.aspx
    00773ad8 00eb593c System.String    c:\inetpub\wwwroot\crash.aspx
    00773ae4 00f84754 System.Web.HttpResponse
    00773ae8 00f846bc System.Web.HttpRequest
    00773b00 00f86bec System.String    crash.aspx
    00773b04 00f86c14 System.Web.HttpServerUtility
    00773b2c 00eef57c System.String    btnCrash
    00773b34 00eef57c System.String    btnCrash
 00773b38 0180e2e4 System.Collections.Specialized.ListDictionary/DictionaryNode
    00773b44 00f797d8 System.String    CausesValidation
    00773b4c 0180e2e4 System.Collections.Specialized.ListDictionary/DictionaryNode
00773b60  033a3031 [DEFAULT] [hasThis] Void System.Web.HttpServerUtility.Transfer(String,Boolean)
    ESP/REG  Object   Name
    00773b60 0180e238 System.Web.UI.WebControls.Button
00773b6c  03360398 [DEFAULT] [hasThis] Void ASP.crash_aspx.btnCrash_Click(Object,Class System.EventArgs)
    ESP/REG  Object   Name
    00773b70 0180e7f4 System.EventArgs
00773b74  033a2fc5 [DEFAULT] [hasThis] Void System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
    ESP/REG  Object   Name
    00773b74 0180df00 ASP.crash_aspx
    00773b78 0180e238 System.Web.UI.WebControls.Button
    00773b7c 00f8601c System.Web.HttpValueCollection
    00773b80 0180e314 System.ComponentModel.EventHandlerList
00773b88  033a2da2 [DEFAULT] [hasThis] Void System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String)
You can tell from this output that the call to Server.Transfer is passing crash.aspx as a parameter. How can you tell that? One indication is that the page that executes after the call is crash.aspx. However, you can also tell by looking at the HttpRequest object.

You can see the HttpRequest object in this stack, so let's have a look at that to see what was passed in that request. We'll do that using the dump object command, !do. This command takes one parameter: the address of the object to dump. In the preceding output, the address is the second hex number in the output.

Here is the output of the HttpRequest object. I have edited this output for the sake of brevity.
0:000> !do 00f846bc 
Name: System.Web.HttpRequest
MethodTable 0x03089170
EEClass 0x02e9a1cc
Size 152(0x98) bytes
mdToken: 02000072  (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 030888b0
      MT    Field   Offset                 Type       Attr    Value Name
03089170  4000626       14                CLASS   instance 00f8457c _path
03089170  4000627       8c       System.Boolean   instance 0 _computePathInfo
03089170  4000628       18                CLASS   instance 00f8457c _filePath
03089170  400062e       2c                CLASS   instance 00f845a4 _pathTranslated
03089170  400062f       30                CLASS   instance 00e6703c _baseVirtualDir
03089170  4000630       34                CLASS   instance 00edce64 _contentType
03089170  4000631       88         System.Int32   instance 85 _contentLength
03089170  400063c       60                CLASS   instance 00f8601c _form
In this output, you can see the _filePath object. If you dump that using the !do command, you will see that it is a string that contains "/crash.aspx". This confirms that the recursion is caused by crash.aspx loading recursively. Why is that happening though? Let's dig deeper.

Look back at the output from !clrstack. If you look carefully, you'll see that the Click event for the button is firing each time the page loads. This tells you that you are in a PostBack state each time the page loads. You can also tell from the stack that Server.Transfer is called in that Click event. Therefore, to stop ASP.NET from crashing when this page is loaded, you will need to stop this from being a PostBack each time the page is loaded due to the Server.Transfer.

Note If you do a Server.Transfer in the ASP.NET Framework 1.1 with SP1 installed and you retain form data, it will always cause this problem. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

839521 FIX: The Server.Transfer method causes a stack overflow and causes the ASP.NET worker process to stop responding

The Server.Transfer method has two parameters; the first is the URL to which execution should transfer, and the second is a Boolean that specifies whether or not the form data should be retained. What happens if we retain the form data? For one thing, ASP.NET will consider that a PostBack. You can, therefore, stop this crash by setting the second parameter to the Server.Transfer call to false.

Wrapping it up

This article presented a brief walkthrough of how you can use WinDbg and SOS to debug your ASP.NET applications. The purpose of this article has been to introduce you to some of the concepts and terminology involved in user-mode debugging. It is a different skill set than source code debugging, and getting good at it requires a significant time investment. It is a skill that is learned by experience.
For more experience, I recommend that you load up your ASP.NET application in a development or staging environment, and capture some hang dump files at different times. You can do that by running Adplus in hang mode by using the following command:

cscript adplus.vbs -hang -pn aspnet_wp.exe -o c:\aspnet_dump


This command will dump the aspnet_wp.exe process immediately and create a dump file in the C:\Aspnet_dump folder. A hang dump file such as this can be run from a remote connection.

Microsoft's Patterns and Practices group (PAG) has also written some excellent and in-depth articles on debugging .NET applications using WinDbg. Visit the following Microsoft Web site to view these articles:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/DBGrm.asp

Indeed, one of my goals in writing this article was to provide you with the foundation necessary to take advantage of the excellent articles that PAG has provided. I strongly encourage you to take advantage of them.
I'll see you back here next month when I'll teach you how to use WinDbg and the SOS extension to analyze crash dump files. Until next time,

Jim Cheshire
Support Engineer
Microsoft Developer Support

As always, feel free to submit ideas on topics you want addressed in future columns or in the Knowledge Base using the Ask For It form.
Modification Type:MajorLast Reviewed:7/29/2005
Keywords:kbhowto kbASP KB892277 kbAudITPRO kbAudDeveloper
©2004 Microsoft Corporation. All rights reserved.