Debugging memory leak in .NET Remoting Server

Memory leak in .net remoting server

Problem Description

.NET remoting application server crashes from time to time with out of memory exception.

Identifying the issue and the resolution

We have the crash dump. So lets open the crash dump using windbg and look at the managed heap stat

1.

0:000> .load clr20\sos.dll

0:000> !dumpheap -stat

——————————

Statistics:

MT Count TotalSize Class Name

7a774f6c 1 12 System.Net.WebRequest+WebProxyWrapper

7a774d9c 1 12 System.Net.Cache.HttpRequestCacheLevel

…………………………………………………………………………………………………………..

6c27b474 11622400

278937600

System.Threading.ReaderWriterCount

7912d8f8 157772 56244476 System.Object[]

2.

Dump “System.Threading.ReaderWriterCount” objects

0:000> !dumpheap -mt 6c27b474

——————————

Heap 0

Address MT Size

0107cdac 012f3654 24

3.

Find the references to this object

0:000> !gcroot -nostacks 012f3654

DOMAIN(00164F78):HANDLE(Pinned):6f13fc:Root:09070048(System.Object[])->

0107906c(System.Collections.Hashtable)->

45bae0d0(System.Collections.Hashtable+bucket[])->

0145ba30(System.Runtime.Remoting.ServerIdentity)->

012f0558(MemoryLeakApp.ServicesProvider)->

012f20d0(MemoryLeakApp.CacheRepository)->

012f20e0(MemoryLeakApp.Collections.ReaderWriterLockDictionary`2[[System.String, mscorlib],[System.Object, mscorlib]])->

012f20f4(System.Threading.ReaderWriterLockSlim)->

012f2134(System.Object[])->

012f3654(System.Threading.ReaderWriterCount)

Why are these objects rooted in System.Runtime.Remoting.ServerIdentity?

System.Runtime.Remoting.ServerIdentity is an internal class in System.Runtime.Remoting namespace. ServerIdentity derives from Identity and holds the extra server specific information associated with each instance of a remoted server object.

4.

In order to look at MemoryLeakApp.ServicesProvider, you can look at the method implementation with DumpIL command and a method description pointer. I prefer to save the assembly from a full dump and look at the class implementation using reflector. First I will run lm command to list loaded modules and then run sos.dll!savemodule command with the module address to save the assembly.

0:000> lm

start end module name

00400000 00486000 MemoryLeakApp(deferred)

0d070000 0d0b6000 RemotingLibrary (deferred)

0d220000 0d25a000 MemoryLeakApp_ServicesProvider(deferred)

0:000> !savemodule 0d220000 D:\temp\MemoryLeakApp_ServicesProvider.dll

3 sections in file

section 0 – VA=2000, VASize=325fc, FileAddr=200, FileSize=32600

section 1 – VA=36000, VASize=49c, FileAddr=32800, FileSize=600

section 2 – VA=38000, VASize=c, FileAddr=32e00, FileSize=200

5.

Now load “D:\temp\MemoryLeakApp_ServicesProvider.dll” in reflector

Class ServicesProvider : MyLifetimeMarshalByRefObject, IDisposable

{

}

public class MyLifetimeMarshalByRefObject: MarshalByRefObject

{


protected MyLifetimeMarshalByRefObject()

{

}

public override object InitializeLifetimeService()

{

return null;

}

}

MSDN description of InitializeLifetimeService is – “Obtains a lifetime service object to control the lifetime policy for this instance.” and when you override this method to return null that means this object will live forever.

You can read more on lifetime leases on http://msdn.microsoft.com/en-us/library/23bk23zc(VS.71).aspx

Resolution

According to MSDN, Marshal-by-reference objects (MBRs) do not reside in memory forever, whether server-activated Singleton objects or client-activated objects. Instead, unless the type overrides MarshalByRefObject.InitializeLifetimeService to control its own lifetime policies, each MBR has a lifetime that is controlled by a combination of leases, a lease manager, and some number of sponsors.

Using leases to manage the lifetime of remote objects is an alternative approach to reference counting, which can be complex and inefficient over unreliable network connections. Although leases can be configured to extend the lifetime of a remote object longer than is precisely required, the reduction in network traffic devoted to reference counting and pinging clients makes leasing an attractive solution when properly configured for a particular scenario.

You should not override this method to return null but implement lease management to suit your needs.

Category: Debugging, WinDbg

Your email address will not be published. Required fields are marked *

*