Memory leak in .net remoting server
.NET remoting application server crashes from time to time with out of memory exception.
Identifying the issue and the resolution
We have the crash dump. So lets open the crash dump using windbg and look at the managed heap stat
0:000> .load clr20\sos.dll
0:000> !dumpheap -stat
MT Count TotalSize Class Name
7a774f6c 1 12 System.Net.WebRequest+WebProxyWrapper
7a774d9c 1 12 System.Net.Cache.HttpRequestCacheLevel
7912d8f8 157772 56244476 System.Object
Dump “System.Threading.ReaderWriterCount” objects
0:000> !dumpheap -mt 6c27b474
Address MT Size
0107cdac 012f3654 24
Find the references to this object
0:000> !gcroot -nostacks 012f3654
012f20e0(MemoryLeakApp.Collections.ReaderWriterLockDictionary`2[[System.String, mscorlib],[System.Object, mscorlib]])->
Why are these objects rooted in System.Runtime.Remoting.ServerIdentity?
System.Runtime.Remoting.ServerIdentity is an internal class in System.Runtime.Remoting namespace. ServerIdentity derives from Identity and holds the extra server specific information associated with each instance of a remoted server object.
In order to look at MemoryLeakApp.ServicesProvider, you can look at the method implementation with DumpIL command and a method description pointer. I prefer to save the assembly from a full dump and look at the class implementation using reflector. First I will run lm command to list loaded modules and then run sos.dll!savemodule command with the module address to save the assembly.
start end module name
00400000 00486000 MemoryLeakApp(deferred)
0d070000 0d0b6000 RemotingLibrary (deferred)
0d220000 0d25a000 MemoryLeakApp_ServicesProvider(deferred)
0:000> !savemodule 0d220000 D:\temp\MemoryLeakApp_ServicesProvider.dll
3 sections in file
section 0 – VA=2000, VASize=325fc, FileAddr=200, FileSize=32600
section 1 – VA=36000, VASize=49c, FileAddr=32800, FileSize=600
section 2 – VA=38000, VASize=c, FileAddr=32e00, FileSize=200
Now load “D:\temp\MemoryLeakApp_ServicesProvider.dll” in reflector
Class ServicesProvider : MyLifetimeMarshalByRefObject, IDisposable
public class MyLifetimeMarshalByRefObject: MarshalByRefObject
public override object InitializeLifetimeService()
MSDN description of InitializeLifetimeService is – “Obtains a lifetime service object to control the lifetime policy for this instance.” and when you override this method to return null that means this object will live forever.
You can read more on lifetime leases on http://msdn.microsoft.com/en-us/library/23bk23zc(VS.71).aspx
According to MSDN, Marshal-by-reference objects (MBRs) do not reside in memory forever, whether server-activated Singleton objects or client-activated objects. Instead, unless the type overrides MarshalByRefObject.InitializeLifetimeService to control its own lifetime policies, each MBR has a lifetime that is controlled by a combination of leases, a lease manager, and some number of sponsors.
Using leases to manage the lifetime of remote objects is an alternative approach to reference counting, which can be complex and inefficient over unreliable network connections. Although leases can be configured to extend the lifetime of a remote object longer than is precisely required, the reduction in network traffic devoted to reference counting and pinging clients makes leasing an attractive solution when properly configured for a particular scenario.
You should not override this method to return null but implement lease management to suit your needs.