Scenario
From time to time, asp.net application response is very slow on Windows Server 2003

Some rants and the resolution

After turning on recycle events, logged message in application event log was Event message: Application is shutting down. Reason: Unknown. Slow response is always timed with this message in the application event log so that confirmed that Application Pool is terminating so no wonder asp.net response is slow from time to time.

However, the only missing piece was why? Since, the Reason is unknown :-). This application pool is configured for web garden with 6 app pools in it so we decided to attach debugger in production box to 2 worker processes.

If you are just starting out with debugging or have not read John Robbins Book on debugging, I would like to stress the followings when using debugger in production environment

1. By Default, ADPlus  writes the call stack on first-chance exception. Walking call stack also results in Symbol loading, symbol loading along with the stack walking causes a performance hit when a debugger is attached. The last thing you want in production environment is to cause performance hit because of  debugger.

2. Don’t just use ADPlus script to attach a debugger to the worker process by name because it will attach the debugger to each worker process in your production server causing  further performance hit.

3. Don’t use DebugDiag in production environment unless you really have a good reason for it.

The reason I am stressing about these points is because I noticed someone following all the worst practices in production environment while debugging.

The best practice is to use adplus configuration file to attach to a worker process and preferably by worker process id.

Identifying the issue

First of all, I would like to mention that Tess has excellent blog on AppDomain recycle http://blogs.msdn.com/tess/archive/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles.aspx

We just needed to find out the reason for AppPool shutdown.

Our goal is to attach a debugger to one worker process to generate a memory dump on crash and another goal is, ctrl+c in debugger console should not kill the worker process. We used optimized ADPlus Config file for production environment as shown below

<ADPlus>
<Settings>
<RunMode>CRASH</RunMode>
<Option>Quiet</Option>
<OutputDir> c:\dumps </OutputDir>
<ProcessID> 2684 </ProcessID>
</Settings>
<Exceptions>
<Option>   NoDumpOnFirstChance    </Option>
<Config>
<Code>    AllExceptions    </Code>
<Actions1>     Log     </Actions1>
<Actions2>    FullDump;  </Actions2>
<ReturnAction1>    GN    </ReturnAction1>
<ReturnAction2>   Q   </ReturnAction2>
</Config>
</Exceptions>
</ADPlus>

once we have the crash dump, just dump the httpruntime object to find out the shutdown reason and the call stack as shown below

internal void StartListeningToLocalResourcesDirectory(VirtualPath virtualDir)

{
    if (!this.IsFCNDisabled && ((this._callbackRenameOrCriticaldirChange != null) && (this._dirMonSpecialDirs != null)))

    {

[DllImport("webengine.dll")]
internal static extern void GetDirMonConfiguration(out int FCNMode);

Since, HttpRuntime controls the file monitoring so get the DirectoryMonitor object and calls StopMonitoring on it. Another way to probably implement is getting the FileChangesMonitor object and setting the FCNMode to 1 or setting the _dirMonSpecialDirs to null.

protected void Application_Start(object sender, EventArgs e)
{
System.Reflection.PropertyInfo p = typeof(System.Web.HttpRuntime)

Twitter This

Username: Password:

Description

I created a simple win32 console application using visual studio 2008, below is the code

class Person
{
private:
char* m_szFirstName;
char* m_szMiddleInitial;
public:
Person(char* szFirstName, char* szMiddleInitial)

{

m_szFirstName = szFirstName;m_szMiddleInitial = szMiddleInitial

}
char* FirstName(){return m_szFirstName;}
char* MiddleInitial(){return m_szMiddleInitial;}
};

int _tmain(int argc, _TCHAR* argv[])
{
Person* pPerson = new Person(”David”,NULL);
char msg[100];
try
{
sprintf_s(msg,”First Name = %s, Middle Initial = %s\n”, pPerson->FirstName(), pPerson->MiddleInitial());
}
catch(…)
{
sprintf_s(msg,”Exception Occured\n”);
}
printf(msg);
delete pPerson;
printf(”press key to exit”);
char* key;
scanf_s(”%c”, key);
return 0;
}

When I run this app, it crashes with Unhandled Exception - Access Violation, although I do have a catch all exception block - catch(…)

Analysis

I disassembled main function and below is what I noticed

Section .text (0×00401000)
;= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
; SYM:wmain <Int>
0×401000: 83EC6C                 SUB         ESP,0×6C           ; <==0×00401207(*+0×207)
0×401003: A100304000         MOV         EAX,DWORD PTR [__security_cookie <UInt>]; (0×403000); .data:0×4E 0xE6 0×40 0xBB
0×401008: 33C4                     XOR         EAX,ESP
0×40100A: 89442468             MOV         DWORD PTR [ESP+0x68],EAX
0×40100E: 56                          PUSH        ESI
0×40100F: 57                          PUSH        EDI
0×401010: 6A08                      PUSH        0×8
0×401012: E88F000000           CALL        __imp_??2@YAPAXI@Z ; (0×4010A6) ; operator new

In case you don’t know the internals of Win32 Structured Exception handling(SEH), please read A Crash Course on the Depths of Win32™ Structured Exception Handling by Matt Pietrek. You will notice the missing instruction to move DWORD pointer from FS:[0x0] register. FS:[0x00] -  4 bytes points to the current Structured Exception Handling (SEH) frame for each thread. Win32 SEH works on a per thread basis with each thread having its own exception handler callback function. On Intel machine, FS register points to the current TEB(Thread Environment Block). That’s why in SEH code, you will have a instruction to move DWORD ptr from FS:[0] register because that’s where you have a pointer to an EXCEPTION_REGISTRATION structure.

Why is it MISSING???

It turns out, visual studio 2008 default settings compiles with “Enable C++ Exception /EHsc” switch which will catch only c++ exceptions also called synchronous exceptions. More information on Exception handling model here http://msdn.microsoft.com/en-us/library/1deeycx5(VS.80).aspx Once you change the compiler switch to /EHa, everything is good as shown below

0×401000: 55                       PUSH        EBP                ; <==0×00401267(*+0×267)
0×401001: 8BEC                   MOV         EBP,ESP
0×401003: 6AFF                   PUSH        0xFF
0×401005: 681B194000       PUSH        __CxxFrameHandler3 + 0×001D; (0×40191B); .text:0×8B 0×54 0×24 0×08
0×40100A: 64A100000000  MOV         EAX,DWORD PTR FS:[0x0]

Twitter This

Username: Password: