ASP.NET App Slow Response and Application Pool/AppDomain Recycle, Event message: Application is shutting down. Reason: Unknown – Windows Server 2003

Scenario
From time to time, asp.net application response is very slow on Windows Server 2003

Rants and the resolution

After turning on recycle events, logged message in application event log was Event message: Application is shutting down. Reason: Unknown. Slow response is always timed with this message in the application event log so that confirmed that Application Pool is terminating so no wonder asp.net response is slow from time to time.

However, the only missing piece was why? Since, the Reason is unknown :-) . This application pool is configured for web garden with 6 app pools in it so we decided to attach debugger in production box to 2 worker processes.

If you are just starting out with debugging or have not read John Robbins Book on debugging, I would like to stress the followings when using debugger in production environment

1. By Default, ADPlus  writes the call stack on first-chance exception. Walking call stack also results in Symbol loading, symbol loading along with the stack walking causes a performance hit when a debugger is attached. The last thing you want in production environment is to cause performance hit because of  debugger.

2. Don’t just use ADPlus script to attach a debugger to the worker process by name because it will attach the debugger to each worker process in your production server causing  further performance hit.

3. Don’t use DebugDiag in production environment unless you really have a good reason for it.

The reason I am stressing about these points is because I noticed someone following all the worst practices in production environment while debugging.

The best practice is to use adplus configuration file to attach to a worker process and preferably by worker process id.

Identifying the issue

First of all, I would like to mention that Tess has excellent blog on AppDomain recycle http://blogs.msdn.com/tess/archive/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles.aspx

We just needed to find out the reason for AppPool shutdown.

Our goal is to attach a debugger to one worker process to generate a memory dump on crash and another goal is, ctrl+c in debugger console should not kill the worker process. We used optimized ADPlus Config file for production environment as shown below

<ADPlus>
<Settings>
<RunMode>CRASH</RunMode>
<Option>Quiet</Option>
<OutputDir> c:\dumps </OutputDir>
<ProcessID> 2684 </ProcessID>
</Settings>
<Exceptions>
<Option>   NoDumpOnFirstChance    </Option>
<Config>
<Code>    AllExceptions    </Code>
<Actions1>     Log     </Actions1>
<Actions2>    FullDump;  </Actions2>
<ReturnAction1>    GN    </ReturnAction1>
<ReturnAction2>   Q   </ReturnAction2>
</Config>
</Exceptions>
</ADPlus>

once we have the crash dump, just dump the httpruntime object to find out the shutdown reason and the call stack as shown below

at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace)
at System.Web.HttpRuntime.OnCriticalDirectoryChange(Object sender, FileChangeEvent e)
at System.Web.FileChangesMonitor.OnSubdirChange(Object sender, FileChangeEvent e)

at System.Web.DirectoryMonitor.FireNotifications()
at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state)
at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
ShutDownReason in memory dumps
0:005> !do 06e6c82c
Name: System.String
MethodTable: 79330a00
EEClass: 790ed64c
Size: 67576(0x107f8) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Directory rename change notification for ‘C:\Inetpub\<>\<>’.
Web dir change or directory renameAppDomain is shutting down because of directory rename.There are thousands of folders under the root website and after talking to customer I learnt that this is indeed the case.

Resolution

After googling or binging, I found out that there is no resolution and this was implemented in asp.net 2.0. AppDomain will recycle on directory rename or delete. However, I did come across a
Microsoft KB http://support.microsoft.com/kb/911272 which talks about changing this behavior from registry setting. The bad news is changing the registry setting [HKEY_LOCAL_MACHINE\SOFTWARE\
ASP.NET\”FCNMode”=dword:00000001 doesn’t work. Although, KB article does suggest you to contact Microsoft support for hot fix but also suggests that it will be addressed in the next .NET SP.

The point is registry setting didn’t work and below is what I found after looking at the HttpRuntime implementation in reflector.
System.HttpRuntime implements the following events for shut down
OnFileChange(FileAction, String, DateTime)
OnCriticaldirChange(Object, FileChangeEvent)
OnSubdirChange(Object, FileChangeEvent)Another cause of AppDomain recycle is overwhelming amount of file change notification but microsoft apparently fixed in .net 2.0 hot fix so who knows.

HttpRuntime also implements another method to monitor the directories

internal void StartListeningToLocalResourcesDirectory(VirtualPath virtualDir)

{
    if (!this.IsFCNDisabled && ((this._callbackRenameOrCriticaldirChange != null) && (this._dirMonSpecialDirs != null)))

    {

You will notice, how it checks for FCNDisabled which should have been the registry key since it return (this._FCNMode == 1);

All these settings are set in
internal
FileChangesMonitor() { UnsafeNativeMethods.GetDirMonConfiguration(outthis._FCNMode);

FCNMode value comes from native dll (webengine.dll)

[DllImport("webengine.dll")]
internal static extern void GetDirMonConfiguration(out int FCNMode);
So the only reason, I can think of why registry setting is not working is the file version of “webengine.dll”. In Microsoft KB the file version of webengine.dll is 2.0.50727.62 which is different than what we have in production box.
After searching more on httpruntime on internet, I came across a post on http://forums.asp.net/p/1310976/2581558.aspx where someone has suggested the workaround using reflection as shown below.

Since, HttpRuntime controls the file monitoring so get the DirectoryMonitor object and calls StopMonitoring on it. Another way to probably implement is getting the FileChangesMonitor object and setting the FCNMode to 1 or setting the _dirMonSpecialDirs to null.

protected void Application_Start(object sender, EventArgs e)
{
System.Reflection.PropertyInfo p = typeof(System.Web.HttpRuntime)

.GetProperty(“FileChangesMonitor”,
System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Static);object o = p.GetValue(null, null);
System.Reflection.FieldInfo f = o.GetType().GetField(“_dirMonSubdirs”,
System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.IgnoreCase);

object monitor = f.GetValue(o);

System.Reflection.MethodInfo m = monitor.GetType().GetMethod(“StopMonitoring”,
System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
m.Invoke(monitor, null);
}

Application Shutdown Unknown Message
I found a piece of code in Scotgu blog to log shutdown events using reflection in Global.asax code behind, below is the code. You will notice that it uses the _shutDownMessage and _shutDownStack member of HttpRuntime object to log the message and the stack.
This implementation saves you from looking at the crash dump and dumping the HttpRuntime object to find the exact shutdown message.
protected void Application_End(object sender, EventArgs e)
{
HttpRuntime runtime = (HttpRuntime)typeof(System.
Web.HttpRuntime).InvokeMember(“_theRuntime”,
BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.GetField, null, null, null);
if (runtime == null)
return;
string shutDownMessage = (string)runtime.GetType().InvokeMember(“_shutDownMessage”,
BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, runtime, null);
string shutDownStack = (string)runtime.GetType().InvokeMember(“_shutDownStack”,
BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, runtime, null);if (!EventLog.SourceExists(“.NET Runtime”))
{
EventLog.CreateEventSource(“.NET Runtime”, “Application”);
}

EventLog log = new EventLog();
log.Source = “.NET Runtime”;
log.WriteEntry(String.Format(“\r\n\r\n_shutDownMessage={0}\r\n\r\n_shutDownStack={1}”,
shutDownMessage,
shutDownStack),
EventLogEntryType.Error);
}

Important Note

Microsoft KB http://support.microsoft.com/kb/911272 which talks about changing this behavior from registry setting doesn’t work in Windows Server 2003 but it does work on Windows Server 2008/R2 with IIS7 integrated/classic pipeline. The best way to address this issue is by modifying the registry key otherwise file handles don’t get released when dirMonSpecialDirs is set to null using reflection

Your email address will not be published. Required fields are marked *

*