Tag Archives: Debugging

WinDbg exhibits a memory leak when you debug postmortem managed dumps

Howdy there!
My name is Olegas and I’ll be blogging here from time to time. Prashant has a very nice collection already and I’ll be adding my 2 cents every once in a while.

Recently I’ve come across an interesting behavior in WinDbg and I decided to look into it a bit further.

The scenario:
• You are debugging a managed dump using WinDbg 6.11.0001.404 on 32bit platform.
• You are trying to dump hundreds of managed objects to inspect their properties. You use a script similar, but not limited to
.foreach (MyObj {!dumpheap -MT 00687320 -short}) {!do MyObj}

The observed behavior:
• After dumping few dozen objects, WinDbg begins to report
<Note: this object has an invalid CLASS field>
Invalid object
• Your 32bt machine begins to respond very slowly and you notice excessive paging.
• Perfmon shows a behavior within WinDbg consistent with a memory leak
Perfmon_Graph

Continue reading →

ASP.NET App Slow Response and Application Pool/AppDomain Recycle, Event message: Application is shutting down. Reason: Unknown – Windows Server 2003

Scenario
From time to time, asp.net application response is very slow on Windows Server 2003

Rants and the resolution

After turning on recycle events, logged message in application event log was Event message: Application is shutting down. Reason: Unknown. Slow response is always timed with this message in the application event log so that confirmed that Application Pool is terminating so no wonder asp.net response is slow from time to time.

However, the only missing piece was why? Since, the Reason is unknown :-) . This application pool is configured for web garden with 6 app pools in it so we decided to attach debugger in production box to 2 worker processes.

If you are just starting out with debugging or have not read John Robbins Book on debugging, I would like to stress the followings when using debugger in production environment

1. By Default, ADPlus  writes the call stack on first-chance exception. Walking call stack also results in Symbol loading, symbol loading along with the stack walking causes a performance hit when a debugger is attached. The last thing you want in production environment is to cause performance hit because of  debugger.

2. Don’t just use ADPlus script to attach a debugger to the worker process by name because it will attach the debugger to each worker process in your production server causing  further performance hit.

3. Don’t use DebugDiag in production environment unless you really have a good reason for it.

Continue reading →

Caution when using System.IO.FileStream – ask what you need – System.UnauthorizedAccessException: Access to the path is denied

I have come across this issue quite a few times and this issue will be seen more often during deployment in QA/Production machine.

Its not unusual to see the below piece of code
using (FileStream fs = new FileStream(filePath, FileMode.Open))
{
XmlReader reader = XmlReader.Create(fs);
XmlSerializer serializer = new XmlSerializer(typeof(T));
Object o = serializer.Deserialize(reader);
}

Many times this will go unnoticed but if you look closely FileStream is using a constructor with FilePath and FileMode alone. System.IO.FileStream implementation of this constructor is

public FileStream(string path, FileMode mode) : this(path, mode, (mode == FileMode.Append) ? FileAccess.Write : FileAccess.ReadWrite, FileShare.Read, 0×1000, FileOptions.None, Path.GetFileName(path), false)
{
}

Default constructor asks for ReadWrite access, so now you see why your application is hosed in production, most of the developers use their system as admin user so of course they have the write access to the requested file. This is not to blame developers who run as admin because I can totally understand the pain.

By default, FileStream needs ReadWrite access that’s why System.UnauthorizedAccessException is thrown because on a production machine, User account under which asp.net worker process runs or a windows service or for that matter any process will not have the write access to a file by default.

Make sure, you ask for what you need. If your application needs only Read access to a file, make sure to specify that in your FileStream constructor. Don’t be GREEDY

using (FileStream fs = new FileStream(filePath, FileMode.Open,FileAccess.Read))

May be I would rather see a Read access by default in System.IO.FileStream implementation rather than ReadWrite.

LoadLibrary failed, Win32 error 0n193 “%1 is not a valid Win32 application.” Please check your debugger configuration and/or network access.

0:000> .loadby sos coreclr
The call to LoadLibrary(c:Program Files (x86)Microsoft Silverlight3.0.40624.0sos) failed, Win32 error 0n193
“%1 is not a valid Win32 application.”
Please check your debugger configuration and/or network access.

Make sure you are not using WinDbg 64 bit version. Silverlight is not 64 bit yet so even if you have a browser running on 64 bit os, sos dll for silverlight coreclr will fail to load on WinDbg 64 bit. Analyze your dump with WinDbg x86 version. I have WinDbg 32 bit and 64 bit both installed on my vista 64 bit os, although I still prefer XP or may be windows 7 from now on.

ProcDump sysiternals tool – really really helpful to create a memory dump based on CPU Usage

As described in Sysinternals documentation http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use) and unhandled exception monitoring. It also can serve as a general process dump utility that you can embed in other scripts.

You don’t need to write your own utility to create a memory dump by monitoring performance counter. Don’t forget to use the switch “-ma” to dump full memory(especially for .net app) because by default it only dumps thread and handle.

This is really helpful to get a memory dump based on CPU usage and we could probably get the memory dump without using ADPlus in most of the cases.

syntax to dump full memory given process id is

procdump <process id> -ma

syntax to dump full memory given process id and cpu usage 80%(threshold)

procdump <process id> -ma -c 80

When Application or Process Isolation is not really the solution

Description

There is an Application Server which apparently dies with OutOfMemory Exception. It has been determined that App Server is not scalable at all because it needs one thread per user so that means 1000 user sessions will consume 1000 threads, holding on to 1GB of virtual memory with 30% cpu spent in Interrupt so yeah you hear the story right.

I came across this issue and the adopted solution which I think is worth mentioning.

The next thing I heard App Server has been split into 2 Application Server and the 2nd process contains user’s session consuming all the threading resources. One of the idea behind this architecture was to give each process 2GB of virtual address space. And on top of that, both the app servers need to be up and running in order to serve a request. And on top of that, no client is consuming services from the 2nd app server. Another point to note is, application server in question is a very simple application server with no algorithmic complexities, but rather act as a middle man processing the request with some caching and data manipulation.

Really ?? We all know that process isolation provides stability and reliability with fault tolerant architecture but at the same time they don’t depend on each other.

With the above architecture they didn’t only introduce latency because of inter process communication but the real issue which is one thread per user with user mode default stack size consuming 1 MB of virtual memory never got resolved.

If you are implementing process isolation because of the above reason, just don’t do it. This is the time to rethink and redesign your application server to make sure that your application server is scalable to support 1000 users without consuming 1000+ threads

win32 console app catch block not catching the Access Violation exception – Why SEH(Structured Exception Handling) not working in a win32 C++ app

Description

I created a simple win32 console application using visual studio 2008, below is the code

class Person
{
private:
char* m_szFirstName;
char* m_szMiddleInitial;
public:
Person(char* szFirstName, char* szMiddleInitial)

{

m_szFirstName = szFirstName;m_szMiddleInitial = szMiddleInitial

}
char* FirstName(){return m_szFirstName;}
char* MiddleInitial(){return m_szMiddleInitial;}
};

int _tmain(int argc, _TCHAR* argv[])
{
Person* pPerson = new Person(“David”,NULL);
char msg[100];
try
{
sprintf_s(msg,”First Name = %s, Middle Initial = %s\n”, pPerson->FirstName(), pPerson->MiddleInitial());
}
catch(…)
{
sprintf_s(msg,”Exception Occured\n”);
}
printf(msg);
delete pPerson;
printf(“press key to exit”);
char* key;
scanf_s(“%c”, key);
return 0;
}

When I run this app, it crashes with Unhandled Exception – Access Violation, although I do have a catch all exception block – catch(…)

Analysis

I disassembled main function and below is what I noticed

Section .text (0×00401000)
;= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
; SYM:wmain <Int>
0×401000: 83EC6C                 SUB         ESP,0x6C           ; <==0×00401207(*+0×207)
0×401003: A100304000         MOV         EAX,DWORD PTR [__security_cookie <UInt>]; (0×403000); .data:0x4E 0xE6 0×40 0xBB
0×401008: 33C4                     XOR         EAX,ESP
0x40100A: 89442468             MOV         DWORD PTR [ESP+0x68],EAX
0x40100E: 56                          PUSH        ESI
0x40100F: 57                          PUSH        EDI
0×401010: 6A08                      PUSH        0×8
0×401012: E88F000000           CALL        __imp_??2@YAPAXI@Z ; (0x4010A6) ; operator new

In case you don’t know the internals of Win32 Structured Exception handling(SEH), please read A Crash Course on the Depths of Win32™ Structured Exception Handling by Matt Pietrek. You will notice the missing instruction to move DWORD pointer from FS:[0x0] register. FS:[0x00] -  4 bytes points to the current Structured Exception Handling (SEH) frame for each thread. Win32 SEH works on a per thread basis with each thread having its own exception handler callback function. On Intel machine, FS register points to the current TEB(Thread Environment Block). That’s why in SEH code, you will have a instruction to move DWORD ptr from FS:[0] register because that’s where you have a pointer to an EXCEPTION_REGISTRATION structure.

Why is it MISSING???

It turns out, visual studio 2008 default settings compiles with “Enable C++ Exception /EHsc” switch which will catch only c++ exceptions also called synchronous exceptions. More information on Exception handling model here http://msdn.microsoft.com/en-us/library/1deeycx5(VS.80).aspx Once you change the compiler switch to /EHa, everything is good as shown below

0×401000: 55                       PUSH        EBP                ; <==0×00401267(*+0×267)
0×401001: 8BEC                   MOV         EBP,ESP
0×401003: 6AFF                   PUSH        0xFF
0×401005: 681B194000       PUSH        __CxxFrameHandler3 + 0x001D; (0x40191B); .text:0x8B 0×54 0×24 0×08
0x40100A: 64A100000000  MOV         EAX,DWORD PTR FS:[0x0]

OutOfMemoryException in System.Drawing.Image, why??

Windows Forms application has been throwing OutOfMemoryException although there is enough virtual memory in the process.  When you have OutOfMemoryException usually callstack is not helpful unless of course you have blocked finalizer thread or some deadlock preventing objects from being gced but in this scenario whenever OutOfMemoryException is thrown it will always point to the same call stack.

at System.Drawing.Image.FromFile(String filename, Boolean useEmbeddedColorManagement)
 at System.Drawing.Image.FromFile(String filename)

Let’s look at System.Drawing.Image implementation using Reflector.  Just read the documentation below
Creates an Image from the specified file using embedded color management information in that file.

Exception Condition
OutOfMemoryException The file does not have a valid image format.-or- GDI+ does not support the pixel format of the file.
FileNotFoundException The specified file does not exist.


My first reaction to this is why? Why would you throw OutOfMemoryException if the file is corrupted or GDIPlus doesn’t support the pixel format.

I really don’t know why does System.Drawing library choose to throw OutOfMemoryException exception? This is only going to add to your worries.