ASP.NET App Slow Response and Application Pool/AppDomain Recycle, Event message: Application is shutting down. Reason: Unknown – Windows Server 2003

Scenario
From time to time, asp.net application response is very slow on Windows Server 2003

Rants and the resolution

After turning on recycle events, logged message in application event log was Event message: Application is shutting down. Reason: Unknown. Slow response is always timed with this message in the application event log so that confirmed that Application Pool is terminating so no wonder asp.net response is slow from time to time.

However, the only missing piece was why? Since, the Reason is unknown :-) . This application pool is configured for web garden with 6 app pools in it so we decided to attach debugger in production box to 2 worker processes.

If you are just starting out with debugging or have not read John Robbins Book on debugging, I would like to stress the followings when using debugger in production environment

1. By Default, ADPlus  writes the call stack on first-chance exception. Walking call stack also results in Symbol loading, symbol loading along with the stack walking causes a performance hit when a debugger is attached. The last thing you want in production environment is to cause performance hit because of  debugger.

2. Don’t just use ADPlus script to attach a debugger to the worker process by name because it will attach the debugger to each worker process in your production server causing  further performance hit.

3. Don’t use DebugDiag in production environment unless you really have a good reason for it.

Continue reading

Caution when using System.IO.FileStream – ask what you need – System.UnauthorizedAccessException: Access to the path is denied

I have come across this issue quite a few times and this issue will be seen more often during deployment in QA/Production machine.

Its not unusual to see the below piece of code
using (FileStream fs = new FileStream(filePath, FileMode.Open))
{
XmlReader reader = XmlReader.Create(fs);
XmlSerializer serializer = new XmlSerializer(typeof(T));
Object o = serializer.Deserialize(reader);
}

Many times this will go unnoticed but if you look closely FileStream is using a constructor with FilePath and FileMode alone. System.IO.FileStream implementation of this constructor is

public FileStream(string path, FileMode mode) : this(path, mode, (mode == FileMode.Append) ? FileAccess.Write : FileAccess.ReadWrite, FileShare.Read, 0×1000, FileOptions.None, Path.GetFileName(path), false)
{
}

Default constructor asks for ReadWrite access, so now you see why your application is hosed in production, most of the developers use their system as admin user so of course they have the write access to the requested file. This is not to blame developers who run as admin because I can totally understand the pain.

By default, FileStream needs ReadWrite access that’s why System.UnauthorizedAccessException is thrown because on a production machine, User account under which asp.net worker process runs or a windows service or for that matter any process will not have the write access to a file by default.

Make sure, you ask for what you need. If your application needs only Read access to a file, make sure to specify that in your FileStream constructor. Don’t be GREEDY

using (FileStream fs = new FileStream(filePath, FileMode.Open,FileAccess.Read))

May be I would rather see a Read access by default in System.IO.FileStream implementation rather than ReadWrite.

LoadLibrary failed, Win32 error 0n193 “%1 is not a valid Win32 application.” Please check your debugger configuration and/or network access.

0:000> .loadby sos coreclr
The call to LoadLibrary(c:Program Files (x86)Microsoft Silverlight3.0.40624.0sos) failed, Win32 error 0n193
“%1 is not a valid Win32 application.”
Please check your debugger configuration and/or network access.

Make sure you are not using WinDbg 64 bit version. Silverlight is not 64 bit yet so even if you have a browser running on 64 bit os, sos dll for silverlight coreclr will fail to load on WinDbg 64 bit. Analyze your dump with WinDbg x86 version. I have WinDbg 32 bit and 64 bit both installed on my vista 64 bit os, although I still prefer XP or may be windows 7 from now on.

ProcDump sysiternals tool – really really helpful to create a memory dump based on CPU Usage

As described in Sysinternals documentation http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use) and unhandled exception monitoring. It also can serve as a general process dump utility that you can embed in other scripts.

You don’t need to write your own utility to create a memory dump by monitoring performance counter. Don’t forget to use the switch “-ma” to dump full memory(especially for .net app) because by default it only dumps thread and handle.

This is really helpful to get a memory dump based on CPU usage and we could probably get the memory dump without using ADPlus in most of the cases.

syntax to dump full memory given process id is

procdump <process id> -ma

syntax to dump full memory given process id and cpu usage 80%(threshold)

procdump <process id> -ma -c 80

.NET Crash/OutofMemoryException/Memory Leak – .NET windows forms and infragistics datagrid and why is System.Drawing.Image object not getting finalized??

Issue Description
Windows forms application has crashed with OOM exception. Before application crashes, cpu is almost pegged at 100% for a few minutes

Root Cause Analysis using WinDbg

Collect full memory dump at set intervals

  • You could get a crash dump and analyze the managed heap to find out rooted objects. But, since we have access to the system I prefer to get a dump at set intervals and compare the managed heap statistics because that makes it a little easier to find the objects which are surviving GC over a period of time.
  • We will use ADPlus to automate this task
  • I will run the script to get a full memory dump 4 times every 2 minutes
  • Command to automate this task is “cscript.exe adplus.vbs -hang -pn <myapp.exe> -quiet -r 4 120″
  • First Dump file size is around 800MB which also indicates process’s memory usage at that time
  • Second Dump file size is around 1.2 GB
  • Third Dump file size is around 1.6 GB and a little later application has crashed.

This is a pure .net application, so we are going to jump ahead and look at the managed heap stats, gc handles and the objects in finalize queue. We will use sos2.dll copied under the same folder as windbg executable, we will dump only pinned and strong gchanldes to identify gc handles increasing over the time because these handles could cause memory leak. Please note that,!gcht (gchandles by type) command is only available in our windbg extension sos2.dll. You could use sos.dll!gchandles to dump gchandles but it won’t give you the objects and their stats by type and you will have to figure out yourself probably by looking at the root.
GCHandles Stats from First Dump
0:000> .load sos2
0:000> !gcht -t p

Pinned GC Handle Statistics:
Pinned Handles: 60
Statistics:
………………………………………………….
Total 60 objects
0:000> !gcht -t s
Strong GC Handle Statistics:
Strong Handles: 185
Statistics:
………………………………………………………
Total 185 objects
GCHandles Stats from Second Dump
0:000> !gcht -t p
Pinned GC Handle Statistics:
Pinned Handles: 60
Statistics:
………………………………………………………………
Total 60 objects
0:000> !gcht -t s
Strong GC Handle Statistics:
Strong Handles: 186
Statistics:
……………………………………………………………..
Total 186 objects

Lets move over since we don’t see anything interesting with gchandles, no. of pinned gchandles remain same and strong gc handles count has increased only by one.

  • We will compare finalize queue stats in dumps, I am only including the interesting objects and the interesting comments for the sake of brevity

Finalize Queue in first dump

0:000> !finalizequeue
generation 2 has 9433 finalizable objects (05501508->0550a86c)
Ready for finalization 0 objects (0550af4c->0550af4c)
Statistics:
MT    Count    TotalSize Class Name
7ae3c9f8     1907 45768 System.Drawing.Bitmap
…………………………………………….
Total 9873 objects

Finalize Queue in second dump

0:000> !finalizequeue
generation 2 has 10545 finalizable objects (05501508->0550b9cc)
Ready for finalization 0 objects (0550bdac->0550bdac)
Statistics:
MT    Count    TotalSize Class Name
7ae3c9f8     2951 70824 System.Drawing.Bitmap
…………………………………………….
Total 10793 objects

Aha, Do we see something interesting here???? Of course, numbers of finalizable objects in generation 2 have increased by almost 1000 and on top of that number of objects ready to be finalized is 0. So why are these objects not getting finalized?

  • We have to find out why System.Drawing.Bitmap is not getting finalized.

As shown in above step,  generation 2 has 9433 finalizable objects (05501508->0550a86c).
We have finalizable objects starting from memory address 05501508 and ending at 0550a86c. You don’t want to dumpheap by type(System.Drawing.Bitmap) to look at the roots to this object, you will have to dump too many objects unless you get lucky. The better way is probably to display the memory and get the address of an object. Size of the System.Drawing.Bitmap object is 24 Bytes so we may be able to get the object address by specifying the address range ending with finalize queue @ 0550a86c. We will subtract 24*4 = 96 bytes(60) from 0550a86c which is 550A80C.
First column is the finalize queue address and the rest are the memory addresses of the objects
0:000> dd 550A80C 0550a86c

0550a80c  17b6e074 17b6e11c 17b6e1c4 17b6e26c
…………………………………………………………………………………….
0550a86c  17b76734
0:000> !do 17b6e074

Name: System.Drawing.Bitmap —-> Make sure this is System.Drawing.Bitmap
MethodTable: 7ae3c9f8
EEClass: 7ade4014
Size: 24(0×18) bytes

0:000> !gcroot -nostacks 17b6e704
DOMAIN(001581B0):HANDLE(Strong):ff11f8:Root:01981b64(System.Threading.Thread)->
………………………………………………………………………………………
01d00f54(MyApp.MyForm)->
160875cc(MyApp.Controls.MyControl)->
1618b578(Infragistics.Win.UltraWinGrid.UltraGridRow)->
14f1a7c0(Infragistics.Win.UltraWinGrid.CellsCollection)->
…………………………………………………………………………………….
17b6e674(Infragistics.Win.UltraWinGrid.UltraGridCell)->
17b6e6f4(Infragistics.Win.UltraWinGrid.UnBoundData)->
17b6e704(System.Drawing.Bitmap)

This is rooted in some strong handles so this is not rooted in finalization queue what that means is object is not ready to be finalized yet as we saw in finalizeQ stats. I am hiding the customer data so basically, we have a windows forms containing user control with infragistics UltraGrid and the System.Drawing.Bitmap is being set in a cell.

Let’s look at the sample code
foreach (UltraGridRow row in rows)
{
row.Cells[someindex] =<bitmap object>
}
This is where we have the problem because if there are let’s say 5000 rows then we are creating 5000 bitmap objects and as long as form is alive these objects will never be disposed. System.Drawing.Bitmap uses unmanaged GDIPlus library and this is not a lightweight object that’s why it was crashing with outofmemory exception and only in a particular scenario but this may go un-noticed during test cycle by QA team unless the test case covers this very particular scenario.
Resolution
I am sure there are many ways to fix it but one easy way to fix is create the drawing objects for rows visible in the client area and handle scroll/resize events to set the image and dispose the objects not in use.

before you install .net 3.5 SP1 or .NET 2.0 SP2

There are quite a few breaking issues with the .NET Framework 2.0 SP2 after you upgrade to the .NET Framework 3.5 SP1.
I will list down a few or follow the link from microsoft knowledge base

http://support.microsoft.com/kb/958481 there is an update to 3.5 sp1 is available with fixes http://support.microsoft.com/kb/959209

  • Exchange Web services generate exceptions because of Windows Communication Foundation (WCF) issues. Each of the exceptions results in a failed request. Therefore, the Exchange service seems to be temporarily down or not working.
  • When you create certain types in the runtime by using reflection as a product of deserialization, the runtime enters an infinite loop in 32-bit processes. In 64-bit processes, an out-of-memory exception occurs. The type must be a generic type that is instantiated by using a reference type. Additionally, the type must implement the ISerializable interface and contain a static field.
  • After you install the .NET Framework 3.5 SP1, you receive the following exception error message when a Web site is hosted under IIS:System.Runtime.InteropServices.COMException
  • AutoCommit behavior in Oracle transactions is different in the .NET Framework 2.0 SP1 from the behavior in the .NET Framework 2.0 SP2. In the .NET Framework 2.0 SP2, if an application starts a transaction, completes the transaction, and then starts a new transaction on the same connection, all the commands that are executed in the second transaction execute in auto-commit mode. The changes that are made by those commands are committed to the database even if the transaction is rolled back