visual studio 2008 memory leak/memory issue on x86 – the operation could not be completed.Not enough storage is available to complete this operation

And No I don’t have a solution for it and probably the only workaround is to make your visual studio Large Address Aware(3GB switch) on x86.

vs2008 error message

Steps to re-create

1. download and unzip http://debuggingblog.com/resources/transcripts.zip

2. open, close the xml file and try to load it the second time

3. If you load the xml file using IE8, you will see the followings once you close it

——————– State SUMMARY ————————–
TotSize (      KB)   Pct(Tots)  Usage
19e6f000 (  424380) : 20.24%   : MEM_COMMIT
f3b4000 (  249552) : 11.90%   : MEM_FREE
56dcd000 ( 1423156) : 67.86%   : MEM_RESERVE

Almost 1.4 GB Memory allocated in GC Segements for xml file is still reserved even after unloading the xml file.

However, visual studio 2008 is another story

0:000> !eeheap -gc
ephemeral segment allocation context: none
segment    begin allocated     size
01830000 01831000  027ecadc 0x00fbbadc(16497372)
12860000 12861000  137616c4 0x00f006c4(15730372)
………………………………………………………………………………..

We have bunch of 16MB GC segments and most of the objects are in gen 2.

0c55d1ec   739459     85777244 Microsoft.XmlEditor.XmlElement
0c559858  1496448     89786880 Microsoft.XmlEditor.Identifier
001f1918   105303     97472784      Free
793308ec  2369315    387475460 System.String
Total 9375571 objects

we have 90+ MB of free blocks and 380+MB in System.String. There are 2.36 million string objects, yeah so you don’t wanna pick each one of the string object to find GC root unless Microsoft or someone is paying you a dime to dump each object and aha a dump a day will make your day for sure.

0:000> !dumpheap -mt 0c9d4134
Address       MT     Size
018f241c 0c9d4134       68
5f610108 0c9d4134       68
total 2 objects
Statistics:
MT    Count    TotalSize Class Name
0c9d4134        2          136 Microsoft.XmlEditor.XmlDocumentProperties
Total 2 objects
0:000> !objsize 018f241c
sizeof(018f241c) =    507372388 (  0x1e3de364) bytes (Microsoft.XmlEditor.XmlDocumentProperties)
0:000> !objsize 5f610108
sizeof(5f610108) =    507371128 (  0x1e3dde78) bytes (Microsoft.XmlEditor.XmlDocumentProperties)

Did you just see that almost 1GB of virtual memory rooted in Microsoft.XmlEditor.XmlDocumentProperties? That’s just outrageous, I mean why would microsoft visual studio take up 1.2 GB of virtual memory to open a 58MB file, although It does make use of schema context cache.

0:000> !gcroot -nostacks 018f241c
DOMAIN(001EC570):HANDLE(RefCnt):16d1b20:Root:018f241c(Microsoft.XmlEditor.XmlDocumentProperties)

GCHandle of type RefCnt is keeping reference to Microsoft.XmlEditor.XmlDocumentProperties

There is an OutOfMemoryException thrown with the following callstack

Exception object: 5ed00a34
Exception type: System.OutOfMemoryException
Message: Insufficient memory to continue the execution of the program.
InnerException: <none>
StackTrace (generated):
SP       IP       Function
0012F5A0 0C97E8B3 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure(Int32, Int32[])+0x3b
0012F5AC 0C9E94BB Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.Source.GetText()+0x3c
0012F5DC 0C9E9360 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.Source.BeginParse()+0×55
0012F644 0C9ECF38 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.Source.OnIdle(Boolean)+0×80
0012F654 0C9ECE28 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.LanguageService.OnIdle(Boolean)+0xd8
0012F674 0C9ECCDD Microsoft_XmlEditor!Microsoft.XmlEditor.XmlLanguageService.OnIdle(Boolean)+0×35
0012F684 0C9ECC34 Microsoft_XmlEditor!Microsoft.XmlEditor.Package.FDoIdle(UInt32)+0xc4

Conclusion

I hope this is fixed in Visual Studio 2010, I do need to try it out.

.NET Crash/OutofMemoryException/Memory Leak – .NET windows forms and infragistics datagrid and why is System.Drawing.Image object not getting finalized??

Issue Description
Windows forms application has crashed with OOM exception. Before application crashes, cpu is almost pegged at 100% for a few minutes

Root Cause Analysis using WinDbg

Collect full memory dump at set intervals

  • You could get a crash dump and analyze the managed heap to find out rooted objects. But, since we have access to the system I prefer to get a dump at set intervals and compare the managed heap statistics because that makes it a little easier to find the objects which are surviving GC over a period of time.
  • We will use ADPlus to automate this task
  • I will run the script to get a full memory dump 4 times every 2 minutes
  • Command to automate this task is “cscript.exe adplus.vbs -hang -pn <myapp.exe> -quiet -r 4 120″
  • First Dump file size is around 800MB which also indicates process’s memory usage at that time
  • Second Dump file size is around 1.2 GB
  • Third Dump file size is around 1.6 GB and a little later application has crashed.

This is a pure .net application, so we are going to jump ahead and look at the managed heap stats, gc handles and the objects in finalize queue. We will use sos2.dll copied under the same folder as windbg executable, we will dump only pinned and strong gchanldes to identify gc handles increasing over the time because these handles could cause memory leak. Please note that,!gcht (gchandles by type) command is only available in our windbg extension sos2.dll. You could use sos.dll!gchandles to dump gchandles but it won’t give you the objects and their stats by type and you will have to figure out yourself probably by looking at the root.
GCHandles Stats from First Dump
0:000> .load sos2
0:000> !gcht -t p

Pinned GC Handle Statistics:
Pinned Handles: 60
Statistics:
………………………………………………….
Total 60 objects
0:000> !gcht -t s
Strong GC Handle Statistics:
Strong Handles: 185
Statistics:
………………………………………………………
Total 185 objects
GCHandles Stats from Second Dump
0:000> !gcht -t p
Pinned GC Handle Statistics:
Pinned Handles: 60
Statistics:
………………………………………………………………
Total 60 objects
0:000> !gcht -t s
Strong GC Handle Statistics:
Strong Handles: 186
Statistics:
……………………………………………………………..
Total 186 objects

Lets move over since we don’t see anything interesting with gchandles, no. of pinned gchandles remain same and strong gc handles count has increased only by one.

  • We will compare finalize queue stats in dumps, I am only including the interesting objects and the interesting comments for the sake of brevity

Finalize Queue in first dump

0:000> !finalizequeue
generation 2 has 9433 finalizable objects (05501508->0550a86c)
Ready for finalization 0 objects (0550af4c->0550af4c)
Statistics:
MT    Count    TotalSize Class Name
7ae3c9f8     1907 45768 System.Drawing.Bitmap
…………………………………………….
Total 9873 objects

Finalize Queue in second dump

0:000> !finalizequeue
generation 2 has 10545 finalizable objects (05501508->0550b9cc)
Ready for finalization 0 objects (0550bdac->0550bdac)
Statistics:
MT    Count    TotalSize Class Name
7ae3c9f8     2951 70824 System.Drawing.Bitmap
…………………………………………….
Total 10793 objects

Aha, Do we see something interesting here???? Of course, numbers of finalizable objects in generation 2 have increased by almost 1000 and on top of that number of objects ready to be finalized is 0. So why are these objects not getting finalized?

  • We have to find out why System.Drawing.Bitmap is not getting finalized.

As shown in above step,  generation 2 has 9433 finalizable objects (05501508->0550a86c).
We have finalizable objects starting from memory address 05501508 and ending at 0550a86c. You don’t want to dumpheap by type(System.Drawing.Bitmap) to look at the roots to this object, you will have to dump too many objects unless you get lucky. The better way is probably to display the memory and get the address of an object. Size of the System.Drawing.Bitmap object is 24 Bytes so we may be able to get the object address by specifying the address range ending with finalize queue @ 0550a86c. We will subtract 24*4 = 96 bytes(60) from 0550a86c which is 550A80C.
First column is the finalize queue address and the rest are the memory addresses of the objects
0:000> dd 550A80C 0550a86c

0550a80c  17b6e074 17b6e11c 17b6e1c4 17b6e26c
…………………………………………………………………………………….
0550a86c  17b76734
0:000> !do 17b6e074

Name: System.Drawing.Bitmap —-> Make sure this is System.Drawing.Bitmap
MethodTable: 7ae3c9f8
EEClass: 7ade4014
Size: 24(0×18) bytes

0:000> !gcroot -nostacks 17b6e704
DOMAIN(001581B0):HANDLE(Strong):ff11f8:Root:01981b64(System.Threading.Thread)->
………………………………………………………………………………………
01d00f54(MyApp.MyForm)->
160875cc(MyApp.Controls.MyControl)->
1618b578(Infragistics.Win.UltraWinGrid.UltraGridRow)->
14f1a7c0(Infragistics.Win.UltraWinGrid.CellsCollection)->
…………………………………………………………………………………….
17b6e674(Infragistics.Win.UltraWinGrid.UltraGridCell)->
17b6e6f4(Infragistics.Win.UltraWinGrid.UnBoundData)->
17b6e704(System.Drawing.Bitmap)

This is rooted in some strong handles so this is not rooted in finalization queue what that means is object is not ready to be finalized yet as we saw in finalizeQ stats. I am hiding the customer data so basically, we have a windows forms containing user control with infragistics UltraGrid and the System.Drawing.Bitmap is being set in a cell.

Let’s look at the sample code
foreach (UltraGridRow row in rows)
{
row.Cells[someindex] =<bitmap object>
}
This is where we have the problem because if there are let’s say 5000 rows then we are creating 5000 bitmap objects and as long as form is alive these objects will never be disposed. System.Drawing.Bitmap uses unmanaged GDIPlus library and this is not a lightweight object that’s why it was crashing with outofmemory exception and only in a particular scenario but this may go un-noticed during test cycle by QA team unless the test case covers this very particular scenario.
Resolution
I am sure there are many ways to fix it but one easy way to fix is create the drawing objects for rows visible in the client area and handle scroll/resize events to set the image and dispose the objects not in use.

Debugging memory leak in .NET Remoting Server

Memory leak in .net remoting server

Problem Description

.NET remoting application server crashes from time to time with out of memory exception.

Identifying the issue and the resolution

We have the crash dump. So lets open the crash dump using windbg and look at the managed heap stat

1.

0:000> .load clr20\sos.dll

0:000> !dumpheap -stat

——————————

Statistics:

MT Count TotalSize Class Name

7a774f6c 1 12 System.Net.WebRequest+WebProxyWrapper

7a774d9c 1 12 System.Net.Cache.HttpRequestCacheLevel

…………………………………………………………………………………………………………..

6c27b474 11622400

278937600

System.Threading.ReaderWriterCount

7912d8f8 157772 56244476 System.Object[]

2.

Dump “System.Threading.ReaderWriterCount” objects

0:000> !dumpheap -mt 6c27b474

——————————

Heap 0

Address MT Size

0107cdac 012f3654 24

3.

Find the references to this object

0:000> !gcroot -nostacks 012f3654

DOMAIN(00164F78):HANDLE(Pinned):6f13fc:Root:09070048(System.Object[])->

0107906c(System.Collections.Hashtable)->

45bae0d0(System.Collections.Hashtable+bucket[])->

0145ba30(System.Runtime.Remoting.ServerIdentity)->

012f0558(MemoryLeakApp.ServicesProvider)->

012f20d0(MemoryLeakApp.CacheRepository)->

012f20e0(MemoryLeakApp.Collections.ReaderWriterLockDictionary`2[[System.String, mscorlib],[System.Object, mscorlib]])->

012f20f4(System.Threading.ReaderWriterLockSlim)->

012f2134(System.Object[])->

012f3654(System.Threading.ReaderWriterCount)

Why are these objects rooted in System.Runtime.Remoting.ServerIdentity?

System.Runtime.Remoting.ServerIdentity is an internal class in System.Runtime.Remoting namespace. ServerIdentity derives from Identity and holds the extra server specific information associated with each instance of a remoted server object.

4.

In order to look at MemoryLeakApp.ServicesProvider, you can look at the method implementation with DumpIL command and a method description pointer. I prefer to save the assembly from a full dump and look at the class implementation using reflector. First I will run lm command to list loaded modules and then run sos.dll!savemodule command with the module address to save the assembly.

0:000> lm

start end module name

00400000 00486000 MemoryLeakApp(deferred)

0d070000 0d0b6000 RemotingLibrary (deferred)

0d220000 0d25a000 MemoryLeakApp_ServicesProvider(deferred)

0:000> !savemodule 0d220000 D:\temp\MemoryLeakApp_ServicesProvider.dll

3 sections in file

section 0 – VA=2000, VASize=325fc, FileAddr=200, FileSize=32600

section 1 – VA=36000, VASize=49c, FileAddr=32800, FileSize=600

section 2 – VA=38000, VASize=c, FileAddr=32e00, FileSize=200

5.

Now load “D:\temp\MemoryLeakApp_ServicesProvider.dll” in reflector

Class ServicesProvider : MyLifetimeMarshalByRefObject, IDisposable

{

}

public class MyLifetimeMarshalByRefObject: MarshalByRefObject

{


protected MyLifetimeMarshalByRefObject()

{

}

public override object InitializeLifetimeService()

{

return null;

}

}

MSDN description of InitializeLifetimeService is – “Obtains a lifetime service object to control the lifetime policy for this instance.” and when you override this method to return null that means this object will live forever.

You can read more on lifetime leases on http://msdn.microsoft.com/en-us/library/23bk23zc(VS.71).aspx

Resolution

According to MSDN, Marshal-by-reference objects (MBRs) do not reside in memory forever, whether server-activated Singleton objects or client-activated objects. Instead, unless the type overrides MarshalByRefObject.InitializeLifetimeService to control its own lifetime policies, each MBR has a lifetime that is controlled by a combination of leases, a lease manager, and some number of sponsors.

Using leases to manage the lifetime of remote objects is an alternative approach to reference counting, which can be complex and inefficient over unreliable network connections. Although leases can be configured to extend the lifetime of a remote object longer than is precisely required, the reduction in network traffic devoted to reference counting and pinging clients makes leasing an attractive solution when properly configured for a particular scenario.

You should not override this method to return null but implement lease management to suit your needs.