LoadLibrary failed, Win32 error 0n193 “%1 is not a valid Win32 application.” Please check your debugger configuration and/or network access.

0:000> .loadby sos coreclr
The call to LoadLibrary(c:Program Files (x86)Microsoft Silverlight3.0.40624.0sos) failed, Win32 error 0n193
“%1 is not a valid Win32 application.”
Please check your debugger configuration and/or network access.

Make sure you are not using WinDbg 64 bit version. Silverlight is not 64 bit yet so even if you have a browser running on 64 bit os, sos dll for silverlight coreclr will fail to load on WinDbg 64 bit. Analyze your dump with WinDbg x86 version. I have WinDbg 32 bit and 64 bit both installed on my vista 64 bit os, although I still prefer XP or may be windows 7 from now on.

ProcDump sysiternals tool – really really helpful to create a memory dump based on CPU Usage

As described in Sysinternals documentation http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use) and unhandled exception monitoring. It also can serve as a general process dump utility that you can embed in other scripts.

You don’t need to write your own utility to create a memory dump by monitoring performance counter. Don’t forget to use the switch “-ma” to dump full memory(especially for .net app) because by default it only dumps thread and handle.

This is really helpful to get a memory dump based on CPU usage and we could probably get the memory dump without using ADPlus in most of the cases.

syntax to dump full memory given process id is

procdump <process id> -ma

syntax to dump full memory given process id and cpu usage 80%(threshold)

procdump <process id> -ma -c 80

Silverlight 3 OutOfBrowser(OOB) behind the scenes Explained, how to host a silverlight xap package OOB style without installing it – read/modify silverlight OOB metadata

You can find a sample on www.silverlight.net or google(bing) it. The purpose of this post is not to show you how to create a Out Of Browser(OOB) application using visual studio / Silverlight but what happens behind the scene and how you can re-use a silverlight xap package to deploy it on any machine by xcopy or a media drive and run it as a desktop application out of browser style.
First of All, creating a Silverlight Out of Browser Application is one line of code and the change in deployment manifest. Please refer to http://timheuer.com/blog/archive/2009/03/18/silverlight-3-offline-update-framework.aspx to understand silverlight offline/update behavior

In order to support Out of Browser in Silverlight, you just need to call Application.Current.Install() on user action as shown below.

private void Button_Click(object sender, RoutedEventArgs e)

When you install the application, it downloads the content(xap package) and generate the html file to host the silverlight app along with some metadata. Everything is generated at isolated storage, as shown below in Windows XP and server 2003

<systemdrive>:\Documents and Settings\<username>\Local Settings\Application Data\Microsoft\Silverlight\OutOfBrowser\<appid>

Continue reading

How to Revert back & forth between Silverlight 3 runtime and Silverlight 2 runtime – Ensuring That Your Silverlight 2 Applications Work with Silverlight 3

With the release of Silverlight 3, you may run into issue where silverlight 2 application may not work as expected. Please read this article on msdn Ensuring That Your Silverlight 2 Applications Work with Silverlight 3

You may need to revert to Silverlight 2 runtime to make sure that the issue has to do with the Silverlight 3 runtime. First of all in order for you to revert back and forth between Silverlight 2 and Silverlight 3 runtime, you should not uninstall Silverlight 2 runtime from your machine. Silverlight runtime gets loaded based on Silverlight plug in which is a COM dll(npctrl.dll).

COM Dll CLSID is “DFEAF541-F3E1-4c24-ACAC-99C30715084A” which remains same in silverlight 3 basically, you can’t run Silverlight 2 and Silverlight 3 runtime(CLR for Silverlight) side by side. Once you install Silverlight 3 runtime, browser running silverlight 2 app will load the Silverlight 3 runtime, you can verify that by looking at the loaded silverlight runtime dll in browser process space. By Default, Silverlight 3 will be installed in “C:\Program Files\Microsoft Silverlight\3.xxxx” so in order for you to revert back and forth all you need to do is register and unregister com dll.

You can simply create a batch file to do the job

for example

Batch file to revert to silverlight 2 runtime, click to download

regsvr32 /u /s “C:\Program Files\Microsoft Silverlight\3.0.40624.0\npctrl.dll”
regsvr32 /s “C:\Program Files\Microsoft Silverlight\2.0.40115.0\npctrl.dll”
Revert to silverlight 3 runtime, click to download

regsvr32 /u /s “C:\Program Files\Microsoft Silverlight\2.0.40115.0\npctrl.dll”
regsvr32 /s “C:\Program Files\Microsoft Silverlight\3.0.40624.0\npctrl.dll”

interesting windows service hang issue because of zone identifier data

The first time I came across this issue, my ex-boss Bill Vieux pointed it out immediately when we were looking at hang dump. I came across this issue again last week so I thought I will blog about it, just in case anyone finds it useful.


Windows Service is configured to watch a folder to export a file, when a file is written to disk it will launch an executable with parameters. First of all, yes it is always a bad idea to launch an exe from a windows service but anyways, sometime you don’t have a choice. Windows service was hanging with the release of new executable and we could never recreate this issue. We asked for hang dump and the following is what we found on a callstack.

0:012> ~6kb
ChildEBP RetAddr  Args to Child
00e8d07c 7e419418 7e42770a 00000000 00000000 ntdll!KiFastSystemCallRet
00e8d0b4 7e4249c4 003d060c 00000000 00000001 USER32!NtUserWaitMessage+0xc
00e8d0dc 7e424a06 03850000 03ab83f8 00000000 USER32!InternalDialogBox+0xd0
00e8d0fc 7e4247ea 03850000 03ab83f8 00000000 USER32!DialogBoxIndirectParamAorW+0×37
00e8d120 77fa9ef1 03850000 00001140 00000000 USER32!DialogBoxParamW+0x3f
00e8d188 7e2f3dc1 03850000 00001140 00000000 SHDOCVW!SHFusionDialogBoxParam+0x3a
00e8d1a8 7e2f43b4 00000000 00001140 00e8d1fc SHDOCVW!_ShowSafeOpenDialog+0×26
00e8f6d0 7ca4d2ec 00000000 0019400c 00194008 SHDOCVW!SafeOpenPromptForShellExec+0x2ce
00e8f6f0 7ca04173 0019400c 00000001 00194008 shell32!CShellExecute::_ZoneCheckFile+0×60
00e8f708 7ca040fa 0019400c 013ca5c0 00194008 shell32!CShellExecute::_VerifyZoneTrust+0x2a
00e8f72c 7ca03071 0019400c 00e8fa0c 00194008 shell32!CShellExecute::_VerifyExecTrust+0xa4
00e8f754 7ca02f6a 013ca5c0 00190f00 00e8fa0c shell32!ShellExecuteNormal+0×30
00e8f770 00a3b0aa 013ca5c0 11ad3aad 00000000 shell32!ShellExecuteExW+0x8d

If you are familiar with alternate data stream, you will notice VerifyZoneTrust immediately. When we dump the parameter passed in internaldialogbox, below is what we have

0:012> dc 03ab83f8 03ab83f8 + 512
03ab83f8  80c80ac0 00000000 00000011 01090000  …………….
03ab8408  000000a9 00000000 004d0008 00200053  ……….M.S. .
03ab8418  00680053 006c0065 0020006c 006c0044  S.h.e.l.l. .D.l.
03ab8428  00000067 50000080 00000000 000a000a  g……P……..
03ab8438  001400f5 ffff1141 00440082 0020006f  ….A…..D.o. .
03ab8448  006f0079 00200075 00610077 0074006e  y.o.u. .w.a.n.t.
03ab8458  00740020 0020006f 0070006f 006e0065   .t.o. .o.p.e.n.
03ab8468  00740020 00690068 00200073 00690066   .t.h.i.s. .f.i.
03ab8478  0065006c 0000003f 00000000 50000003  l.e.?……….P
03ab86f8  0069004c 006b006e 00540000 00690068  L.i.n.k…T.h.i.
03ab8708  00200073 00790074 00650070 006f0020  s. .t.y.p.e. .o.
03ab8718  00200066 00690066 0065006c 00630020  f. .f.i.l.e. .c.
03ab8728  006e0061 00680020 00720061 0020006d  a.n. .h.a.r.m. .
03ab8738  006f0079 00720075 00630020 006d006f  y.o.u.r. .c.o.m.
03ab8748  00750070 00650074 002e0072 004f0020  p.u.t.e.r… .O.
03ab8758  006c006e 00200079 00750072 0020006e  n.l.y. .r.u.n. .
03ab8768  006f0073 00740066 00610077 00650072  s.o.f.t.w.a.r.e.
03ab8778  00660020 006f0072 0020006d 00750070   .f.r.o.m. .p.u.
03ab8788  006c0062 00730069 00650068 00730072  b.l.i.s.h.e.r.s.
03ab8798  00790020 0075006f 00740020 00750072   .y.o.u. .t.r.u.
03ab87a8  00740073 0020002e 0041003c 0048003e  s.t… .<.A.>.H.
03ab87b8  0077006f 00630020 006e0061 00490020  o.w. .c.a.n. .I.
03ab87c8  00640020 00630065 00640069 00200065   .d.e.c.i.d.e. .
03ab87d8  00680077 00740061 00730020 0066006f  w.h.a.t. .s.o.f.
03ab87e8  00770074 00720061 00200065 006f0074  t.w.a.r.e. .t.o.
03ab87f8  00740020 00750072 00740073 003c003f   .t.r.u.s.t.?.<.
03ab8808  0041002f 0000003e 00000000 00000000  /.A.>………..
03ab8818  80c80ac0 00000000 00000011 01090000  …………….
03ab8828  000000a7 00000000 004d0008 00200053  ……….M.S. .
03ab8838  00680053 006c0065 0020006c 006c0044  S.h.e.l.l. .D.l.
03ab8848  00000067 50000080 00000000 000a000a  g……P……..
03ab8858  001400f5 ffff1141 00440082 0020006f  ….A…..D.o. .
03ab8868  006f0079 00200075 00610077 0074006e  y.o.u. .w.a.n.t.
03ab8878  00740020 0020006f 0070006f 006e0065   .t.o. .o.p.e.n.
03ab8888  00740020 00690068 00200073 00690066   .t.h.i.s. .f.i.
03ab8898  0065006c 0000003f 00000000 50000003  l.e.?……….P

We have a modal dialog box with message “Do you want to …, file can harm your computer…” and since windows service is not running in interactive desktop mode, you are not going to see the warning message.

on executing streams( downloadable from http://technet.microsoft.com/en-us/sysinternals/bb897440.aspx) on exe, we see the following output

D:\Tools\Streams>streams “d:\xxxxx.exe”

Streams v1.56 – Enumerate alternate NTFS data streams
Copyright (C) 1999-2007 Mark Russinovich
Sysinternals – www.sysinternals.com

:Zone.Identifier:$DATA       26

Zone.identifier is added as a security measure to every executable when downloaded from http or on a network share.

Since this application could be deployed using xcopy, so someone unzipped on a network share to copy the executable and of course windows service is now hung. Although, it is always a bad idea to launch exe but sometime there is no option so watch out for this scenario

Writing a dynamic, configurable windows service with better debugging experience

Download the source code

I was writing a media streaming service along with file watcher so I thought I will share this piece of code in case anyone finds it useful.  When developing Win32 Windows NT Service in old days, it used to be an exercise to debug a windows service from visual studio, I will always end up in writing a test harness to test those dependent dlls.

Although, managed code has made our life  easier but I still write a service with nothing in it other than service control manager startup.

Step 1 – Define a contract for dynamic and configurable service

public interface IDynamicService
void StartUp(NameValueCollection configData);
void Shutdown();
Step2 – Service is configurable from a config file

<sectionGroup name=”StreamingServices”>
<section name=”ServiceMap” type=”System.Configuration.NameValueSectionHandler” />
<!– List of services–>
<section name=”MediaService” type=”System.Configuration.NameValueSectionHandler” />
<section name=”MaintenanceService” type=”System.Configuration.NameValueSectionHandler” />
<add key=”AppName” value=”StreamingServices” />
<add key=”ServiceName” value=”MediaService” />
<!–Start MediaService only–>
<add key=”MediaService”          value=”TRUE” />
<!– type name implementing the service–>
<add key=”Type”             value=”StreamingMedia” />
<!– assembly name implementing the type, assembly metadata table only contains the assembly name with path or extension–>
<add key=”AssemblyName”          value=”MediaService” />
<!–path to the assembly–>
<add key=”AssemblyPath”            value=”MediaService.dll” />
<!–this service watches the folders for filesystem events–>
<add key=”Folders”                value=”D:\StreamingService\Host” />
<add key=”Type”                          value=”MaintenanceService” />
<add key=”AssemblyName”          value=”MaintenanceService” />
<add key=”AssemblyPath”            value=”MaintenanceService.dll” />

Step 3 – create a struct for serviceinstance metadata

struct ServiceInstance
public NameValueCollection Configuration { get; set; }
public IDynamicService DynamicServiceInstance { get; set; }

Step 4 – Implementing the dynamic service

/// <summary>
/// </summary>
/// <param name=”appName”></param>
public DynamicService(string appName)
//read the service map section to iterate through list of services
NameValueCollection serviceMap = ConfigurationManager.GetSection(appName + “/ServiceMap”)
as NameValueCollection;

foreach (string serviceName in serviceMap.Keys)
//if the service name is true in the config file, add it to the list
//<add key=”MediaService”    value=”TRUE” />

if (Convert.ToBoolean(serviceMap[serviceName]))
ServiceInstance serviceInstance = new ServiceInstance();
serviceInstance.Configuration = ConfigurationManager.GetSection(appName + “/” + serviceName)
as NameValueCollection;
catch (Exception ex)
//log it
throw ex;

public bool Start()
//iterate through each service set to true in config file
//<add key=”MediaService”    value=”TRUE” />

for (int i = 0; i < serviceList.Count; i++)
ServiceInstance serviceInstance = serviceList[i];
//try to load the service using reflection
serviceInstance.DynamicServiceInstance = LoadServices(serviceInstance.Configuration);

if (serviceInstance.DynamicServiceInstance != null)
//service implementing the contract IDynamicService will be started here
catch (Exception ex)
//log it
throw ex;
return true;
/// <summary>
/// Shutdown all the services
/// </summary>
public void Shutdown()
//shutting down services in reverse order
for (int i = serviceList.Count – 1; i >= 0; i–)
ServiceInstance serviceInstance = serviceList[i];

if (serviceInstance.DynamicServiceInstance != null)
catch (Exception ex)
//log it first
throw ex;
/// <summary>
/// this method is called recursively,  it  first loads the assembly as configured in .config
/// and then loads the type
/// </summary>
/// <param name=”assemblyPath”></param>
/// <param name=”typeName”></param>
/// <param name=”assemblyName”></param>
/// <returns></returns>

private object LoadType(string assemblyPath, string typeName, string assemblyName)
object objInstance = null;
bool assemblyLoaded = false;

//iterate through each assembly in the current domain to compare it with the
//configured assembly in app.config

Assembly[] assemblies = System.AppDomain.CurrentDomain.GetAssemblies();
foreach (Assembly assembly in assemblies)
//if assembly name is defined in the config file, load the type
//<add key=”AssemblyName”          value=”MediaService” />

if (assembly.GetName().Name == assemblyName)
assemblyLoaded = true;
Type[] types = assembly.GetTypes();
foreach (Type type in types)
if (type.Name == typeName)
//<add key=”Type”             value=”StreamingMedia” />
//if type name is defined, create the instance

objInstance = assembly.CreateInstance(type.FullName, false);
assemblyLoaded = true;
//called to load assembly from path
if (assemblyLoaded == false)
//<add key=”AssemblyPath”            value=”MediaService.dll” />

//recursive call here
objInstance = LoadType(assemblyPath, typeName, assemblyName);
catch (Exception ex)
//log message
throw ex;
return objInstance;
/// <summary>
/// read the config section to load the assembly and type
/// </summary>
/// <param name=”configData”></param>
/// <returns></returns>

private IDynamicService LoadServices(NameValueCollection configData)
IDynamicService objRet = null;

string assemblyPath = configData[ASSEMBLY_PATH];
string typeName = configData[TYPE_NAME];
string assemblyName = configData[ASSEMBLY_NAME];
objRet = LoadType(assemblyPath, typeName, assemblyName) as IDynamicService;
catch (Exception ex)
//log message
throw ex;
return objRet;

Step 5 – Since we are going to watch the file system, we should never process a file in the same thread as the filewatcher event handler(cause of missing events, blocking the thread for longer duration, buffer and etc), we should just follow a producer consumer model to queue the data and hand it over to consumer. we will create a generic thread safe queue  first

public class SyncQueue<T>
#region Private variables
private const int DEFAULT_CAPACITY = 25;
private object _syncRoot = null;
private Queue<T> _syncQueue = null;
private AutoResetEvent _resetEvent = new AutoResetEvent(false);

#region Constructor
public SyncQueue()

public SyncQueue(int capacity)
_syncQueue = new Queue<T>(capacity);
_syncRoot = ((ICollection)_syncQueue).SyncRoot;

#region public methods
public void Enqueue(T data)
if (data == null) new ArgumentNullException(“Null Argument”);
lock (_syncRoot)

public T Dequeue()
lock (_syncRoot)
return _syncQueue.Dequeue();

public AutoResetEvent Signal
get { return _resetEvent; }

public int Count
get { return _syncQueue.Count; }

public Queue<T> Queue
get { return _syncQueue; }

Step 6 – now we will define an abstract base class for File System Events, I have the FileSystemWatcher created in this base class to handle events but it may be better to have a composite type with filters and FileSystemWatcher created in a derived class for better reusability but it serves the purpose for me. I still prefer to suffix it with “Base” which is not preferred by many for abstract class

public abstract class FileSystemWatcherBase
#region Properties
private SyncQueue<string> FileQueue { get; set; }
private string RootPath { get; set; }
private string[] Filters { get; set; }

#region constructors
protected FileSystemWatcherBase() { }

public void Initialize(FSStruct fsData)
CheckForNullArguments(fsData.FileQueue, “DataQueue<string> fileQueue”);
CheckForNullArguments(fsData.RootPath, “string rootPath”);
CheckForNullArguments(fsData.Filters, “string[] filters”);
FileQueue = fsData.FileQueue;
RootPath = fsData.RootPath;
Filters = fsData.Filters;

#region public members
public virtual void Start()
string[] folders = Directory.GetDirectories(RootPath);
foreach (string folder in folders)
foreach (string filter in Filters)
CreateFSWatcher(folder, filter);

#region protected members
protected void CreateFSWatcher(string folder, string filter)
FileSystemWatcher watcher = new FileSystemWatcher();
// Create a new FileSystemWatcher and set its properties.
watcher.Path = folder;
watcher.IncludeSubdirectories = true;
watcher.NotifyFilter = NotifyFilters.FileName;
watcher.Filter = filter;

// Add event handlers.
watcher.Created += new FileSystemEventHandler(OnCreated);
watcher.Error += new ErrorEventHandler(OnError);
// Begin watching.
watcher.EnableRaisingEvents = true;

#region private methods

private void OnError(object source, ErrorEventArgs e)
Exception ex = e.GetException();
//to do—

private void OnCreated(object source, FileSystemEventArgs e)

private void CheckForNullArguments(object obj, string paramName)
if (obj == null)
throw new ArgumentNullException(paramName, “Null arguments in FileSystemWatcher Constructor”);


#region data structure
public struct FSStruct
public SyncQueue<string> FileQueue;
public string RootPath;
public string[] Filters;

Step 7 – Media Service to monitor and process the files

//implements the IDynamicService

public class MediaService : FileSystemWatcherBase, IDynamicService
private Thread workerThread;
private bool isRunning = false;
private SyncQueue<string> _fileQueue;
private AutoResetEvent _waitEvent;
public MediaService() { }

public void StartUp(NameValueCollection configData)
_waitEvent = new AutoResetEvent(false);
_fileQueue = new SyncQueue<string>();
string[] filters = { “*.mp4″, “*.mpeg” };
FSStruct fsData = new FSStruct()
FileQueue = _fileQueue,
Filters = filters,
RootPath = configData["Folders"]

public override void Start()
isRunning = true;
workerThread = new Thread(new ThreadStart(Run));

public void Shutdown()
isRunning = false;

private void Run()
while (isRunning)
_fileQueue.Signal.WaitOne(); // Wait for the file
while (_fileQueue.Count > 0)
//to do with file
catch (Exception ex)
//to do

Step 8 – Writing the service

Just 2 lines of code either in a console app or a Windows Service OnStart().

This will load numbers of services dynamically from a config file. Services can be removed or a new one can be added directly in a config file. You can debug it directly from a console appor a forms app making it reusable

protected override void OnStart(string[] args)
string appName = ConfigurationSettings.AppSettings["AppName"];
new DynamicService(appName).Start();


Download the source code

WinDbg meta-command tip to display all the extension commands exported by WinDbg extension

Usually, A WinDbg extension will have the !help command in case you need to look at the supported commands in an extension. However, not all commands may be documented or no documentation at all. In that case you can use Depends or any dissembler to look at the Export section.

But, with .extmatch command, you can achieve the same right in the debugger itself as shown below.

Below is the executed command to display all the extension commands supported by loaded SOS in CLR 4.0

remember the space between sos and *

0:020> .extmatch /D /e c:\WINDOWS\Microsoft.NET\Framework\v4.0.20506\sos *

Interesting WinDbg Extension SOS commands in CLR 4.0/.NET Framework 4.0 CTP, .NET runtime dll renamed and sos commands just got richer

We will review the WinDbg Extension SOS.dll in .NET Framework 4.0 CTP. CLR 4.0 has renamed runtime dll from mscorwks.dll to CLR.DLL, that’s really helpful.

loading SOS dll depending on the location of .net 4.0 runtime aka CLR.DLL, execute the following command

.loadby sos clr

1.  DML Support – YES, finally.  SOS supports DML in .NET 1.1 but it was gone in clr 2.0.  Silverlight CoreCLR supports DML and now .NET framework 4.0 supports it as well.

Execute the following command to turn on DMLfor every command or use /D option

0:003> .prefer_dml 1
DML versions of commands on by default

0:003> !dumpheap /D -type Exception -stat

For people new to WinDbg, Why am I so excited about DML support in SOS?
DML Snapshot

If you look at the above snapshot, you have the link for each MethodTable address which you can just click on to execute the command. No need to type, however not every commands will have the DML support but !dumpobject is another important one, you can just click on object address to dump an object from GC Heap.

2. The following additional extension commands are added

Examining code and stacks


Examining CLR data structures


Diagnostic Utilities

!ListNearObj (lno)
!AnalyzeOOM (ao)

Examining the GC history


!ThreadState Command

When you execute !threads command, you will see the similar output as shown below

PreEmptive   GC Alloc           Lock
ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
0    1  310 00161438      a020 Enabled  013b4c64:013b5fe8 00159230     1 MTA
2    2  8c4 0016dab0      b220 Enabled  00000000:00000000 00159230     0 MTA (Finalizer)

First column is your debugger thread id and the second column ID is ManagedThread ID, OSID column is OS thread ID so that means OSID column will be 0 or some garbage when a runtime uses Fiber.

You will see the State column which is a bit flag as shown below(taken from Shared CLI)

TS_Unknown                = 0×00000000,    // threads are initialized this way

TS_AbortRequested         = 0×00000001,    // Abort the thread
TS_GCSuspendPending       = 0×00000002,    // waiting to get to safe spot for GC
TS_UserSuspendPending     = 0×00000004,    // user suspension at next opportunity
TS_DebugSuspendPending    = 0×00000008,    // Is the debugger suspending threads?
TS_GCOnTransitions        = 0×00000010,    // Force a GC on stub transitions (GCStress only)

TS_SuspendUnstarted       = 0×00400000,    // latch a user suspension on an unstarted thread

TS_ThreadPoolThread       = 0×00800000,    // is this a threadpool thread?
TS_TPWorkerThread         = 0×01000000,    // is this a threadpool worker thread?

TS_Interruptible          = 0×02000000,    // sitting in a Sleep(), Wait(), Join()
TS_Interrupted            = 0×04000000,    // was awakened by an interrupt APC. !!! This can be moved to TSNC

TS_CompletionPortThread   = 0×08000000,    // Completion port thread

SOS in CLR4.0 has !threadstate command, which tells you exactly the state of the thread given the bit field, the following output shows you the threadstate bit for Worker Thread, Completion Port Thread and Finalizer Thread

0:000> !ThreadState 1009220
Legal to Join
CLR Owns
In Multi Threaded Apartment
Thread Pool Worker Thread
0:000> !ThreadState 800a220
Legal to Join
In Multi Threaded Apartment
Completion Port Thread
0:000> !ThreadState b220
Legal to Join
CLR Owns
In Multi Threaded Apartment

Other Important Commands

!findroots – This is a very powerful and interesting command, because it allows you to break into debugee when CLR garbage collect generational objects.

!GCWhere - tells you the generation number along with the GC heap segment, you no longer need to map the object address with the GC heap segment or use any other extension dll

!HeapStat- This is another cool command, this command displays the stat on generational heap including generation sizes

!AnalyzeOOM – displays the detailed informatin on Last System.OutOfMemoryException

I can’t do justice on detailed documentation for each of these commands because SOS !help documentation has done a very good job. You can either look at !help documentation  or read below. I am just copying and pasting from SOS Help documentation

0:020> !help ThreadState
!ThreadState value

The !Threads command outputs, among other things, the state of the thread.
This is a bit field which corresponds to various states the thread is in.
To check the state of the thread, simply pass that bit field from the
output of !Threads into !ThreadState.

0:003> !Threads
ThreadCount:      2
UnstartedThread:  0
BackgroundThread: 1
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
PreEmptive   GC Alloc           Lock
ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
0    1  250 0019b068      a020 Disabled 02349668:02349fe8 0015def0     0 MTA
2    2  944 001a6020      b220 Enabled  00000000:00000000 0015def0     0 MTA (Finalizer)
0:003> !ThreadState b220
Legal to Join
CLR Owns
In Multi Threaded Apartment

Possible thread states:
Thread Abort Requested
GC Suspend Pending
User Suspend Pending
Debug Suspend Pending
GC On Transitions
Legal to Join
Yield Requested
Hijacked by the GC
Blocking GC for Stack Overflow
CLR Owns
In Single Threaded Apartment
In Multi Threaded Apartment
Reported Dead
Task Reset
Sync Suspended
Debug Will Sync
Stack Crawl Needed
Suspend Unstarted
Thread Pool Worker Thread
Completion Port Thread
Abort Initiated
Failed to Start
0:020> !help DumpSigElem
!DumpSigElem <sigaddr> <moduleaddr>

This command dumps a single element of a signature object.  For most circumstances,
you should use !DumpSig to look at individual signature objects, but if you find a
signature that has been corrupted in some manner you can use !DumpSigElem to read out
the valid portions of it.

If we look at a valid signature object for a method we see the following:
0:000> !dumpsig 0x000007fe`ec20879d 0x000007fe`eabd1000
[DEFAULT] [hasThis] Void (Boolean,String,String)

We can look at the individual elements of this object by adding the offsets into the
object which correspond to the return value and parameters:
0:000> !dumpsigelem 0x000007fe`ec20879d+2 0x000007fe`eabd1000
0:000> !dumpsigelem 0x000007fe`ec20879d+3 0x000007fe`eabd1000
0:000> !dumpsigelem 0x000007fe`ec20879d+4 0x000007fe`eabd1000
0:000> !dumpsigelem 0x000007fe`ec20879d+5 0x000007fe`eabd1000

We can do something similar for fields.  Here is the full signature of a field:
0:000> !dumpsig 0x000007fe`eb7fd8cd 0x000007fe`eabd1000
[FIELD] ValueClass System.RuntimeTypeHandle

Using !DumpSigElem we can find the type of the field by adding the offset of it (1) to
the address of the signature:
0:000> !dumpsigelem 0x000007fe`eb7fd8cd+1 0x000007fe`eabd1000
ValueClass System.RuntimeTypeHandle

!DumpSigElem will also work with generics.  Let a function be defined as follows:
public A Test(IEnumerable<B> n)

The elements of this signature can be obtained by adding offsets into the signature
when calling !DumpSigElem:

0:000> !dumpsigelem 00000000`00bc2437+2 000007ff00043178
0:000> !dumpsigelem 00000000`00bc2437+4 000007ff00043178
Class System.Collections.Generic.IEnumerable`1<__Canon>

The actual offsets that you should add are determined by the contents of the
signature itself.  By trial and error you should be able to find various elements
of the signature.

0:020> !help VerifyObj
!VerifyObj <object address>

!VerifyObj is a diagnostic tool that checks the object that is passed as an
argument for signs of corruption.

0:002> !verifyobj 028000ec
object 0x28000ec does not have valid method table

0:002> !verifyobj 0680017c
object 0x680017c: bad member 00000001 at 06800184

0:020> !help FindRoots
!FindRoots -gen <N> | -gen any | <object address>

The “-gen” form causes the debugger to break in the debuggee on the next
collection of the specified generation.  The effect is reset as soon as the
break occurs, in other words, if you need to break on the next collection you
would need to reissue the command.

The last form of this command is meant to be used after the break caused by the
other forms has occurred.  Now the debuggee is in the right state for
!FindRoots to be able to identify roots for objects from the current condemned

!FindRoots is a diagnostic command that is meant to answer the following

“I see that GCs are happening, however my objects have still not been
collected. Why? Who is holding onto them?”

The process of answering the question would go something like this:

1. Find out the generation of the object of interest using the !GCWhere
command, say it is gen 1:
!GCWhere <object address>

2. Instruct the runtime to stop the next time it collects that generation using
the !FindRoots command:
!FindRoots -gen 1

3. When the next GC starts, and has proceeded past the mark phase a CLR
notification will cause a break in the debugger:
(fd0.ec4): CLR notification exception – code e0444143 (first chance)
CLR notification: GC – end of mark phase.
Condemned generation: 1.

4. Now we can use the !FindRoots <object address> to find out the cross
generational references to the object of interest.  In other words, even if the
object is not referenced by any “proper” root it may still be referenced by an
older object (from an older generation), from a generation that has not yet been
scheduled for collection.  At this point !FindRoots will search those older
generations too, and report those roots.
0:002> !findroots 06808094
older generations::Root:  068012f8(AAA.Test+a)->

0:020> !help HeapStat
!HeapStat [-inclUnrooted | -iu]

This command shows the generation sizes for each heap and the total, how much free
space there is in each generation on each heap.  If the -inclUnrooted option is
specified the report will include information about the managed objects from the
GC heap that are not rooted anymore.

Sample output:

0:002> !heapstat
Heap     Gen0         Gen1         Gen2         LOH
Heap0    177904       12           306956       8784
Heap1    159652       12           12           16
Total    337556       24           306968       8800

Free space:                                                 Percentage
Heap0    28           12           12           64          SOH:  0% LOH:  0%
Heap1    104          12           12           16          SOH:  0% LOH:100%
Total    132          24           24           80

0:002> !heapstat -inclUnrooted
Heap     Gen0         Gen1         Gen2         LOH
Heap0    177904       12           306956       8784
Heap1    159652       12           12           16
Total    337556       24           306968       8800

Free space:                                                 Percentage
Heap0    28           12           12           64          SOH:  0% LOH:  0%
Heap1    104          12           12           16          SOH:  0% LOH:100%
Total    132          24           24           80

Unrooted objects:                                           Percentage
Heap0    152212       0            306196       0           SOH: 94% LOH:  0%
Heap1    155704       0            0            0           SOH: 97% LOH:  0%
Total    307916       0            306196       0

The percentage column contains a breakout of free or unrooted bytes to total bytes.

0:020> !help GCWhere
!GCWhere <object address>

!GCWhere displays the location in the GC heap of the argument passed in.

0:002> !GCWhere 02800038
Address  Gen Heap segment  begin    allocated size
02800038 2    0   02800000 02800038 0282b740  12

When the argument lies in the managed heap, but is not a valid *object* address
the “size” is displayed as 0:

0:002> !GCWhere 0280003c
Address  Gen Heap segment  begin    allocated size
0280003c 2    0   02800000 02800038 0282b740  0

0:020> !help ListNearObj
!ListNearObj <object address>

!ListNearObj is a diagnostic tool that displays the object preceeding and
succeeding the address passed in:

The command looks for the address in the GC heap that looks like a valid
beginning of a managed object (based on a valid method table) and the object
following the argument address.

0:002> !ListNearObj 028000ec
Before: 0x28000a4           72 (0×48      ) System.StackOverflowException
After:  0×2800134           72 (0×48      ) System.Threading.ThreadAbortException
Heap local consistency confirmed.

0:002> !ListNearObj 028000f0
Before: 0x28000ec           72 (0×48      ) System.ExecutionEngineException
After:  0×2800134           72 (0×48      ) System.Threading.ThreadAbortException
Heap local consistency confirmed.

The command considers the heap as “locally consistent” if:
prev_obj_addr + prev_obj_size = arg_addr && arg_obj + arg_size = next_obj_addr
prev_obj_addr + prev_obj_size = next_obj_addr

When the condition is not satisfied:

0:002> !lno 028000ec
Before: 0x28000a4           72 (0×48      ) System.StackOverflowException
After:  0×2800134           72 (0×48      ) System.Threading.ThreadAbortException
Heap local consistency not confirmed.

0:020> !help AnalyzeOOM

!AnalyzeOOM displays the info of the last OOM occured on an allocation request to
the GC heap (in Server GC it displays OOM, if any, on each GC heap).

To see the managed exception(s) use the !Threads command which will show you
managed exception(s), if any, on each managed thread. If you do see an
OutOfMemoryException exception you can use the !PrintException command on it.
To get the full callstack use the “kb” command in the debugger for that thread.
For example, to display thread 3′s stack use ~3kb.

OOM exceptions could be because of the following reasons:

1) allocation request to GC heap
in which case you will see JIT_New* on the call stack because managed code called new.
2) other runtime allocation failure
for example, failure to expand the finalize queue when GC.ReRegisterForFinalize is
3) some other code you use throws a managed OOM exception
for example, some .NET framework code converts a native OOM exception to managed
and throws it.

The !AnalyzeOOM command aims to help you with investigating 1) which is the most
difficult because it requires some internal info from GC. The only exception is
we don’t support allocating objects larger than 2GB on CLR v2.0 or prior. And this
command will not display any managed OOM because we will throw OOM right away
instead of even trying to allocate it on the GC heap.

There are 2 legitimate scenarios where GC would return OOM to allocation requests -
one is if the process is running out of VM space to reserve a segment; the other
is if the system is running out physical memory (+ page file if you have one) so
GC can not commit memory it needs. You can look at these scenarios by using performance
counters or debugger commands. For example for the former scenario the “!address
-summary” debugger command will show you the largest free region in the VM. For
the latter scenario you can look at the “Memory\% Committed Bytes In Use” see
if you are running low on commit space. One important thing to keep in mind is
when you do this kind of memory analysis it could an aftereffect and doesn’t
completely agree with what this command tells you, in which case the command should
be respected because it truly reflects what happened during GC.

The other cases should be fairly obvious from the callstack.

Sample output:

0:011> !ao
———Heap 2 ———
Managed OOM occured after GC #28 (Requested to allocate 1234 bytes)
Reason: Didn’t have enough memory to commit
Detail: SOH: Didn’t have enough memory to grow the internal GC datastructures (800000 bytes) -
on GC entry available commit space was 500 MB
———Heap 4 ———
Managed OOM occured after GC #12 (Requested to allocate 100000 bytes)
Reason: Didn’t have enough memory to allocate an LOH segment
Detail: LOH: Failed to reserve memory (16777216 bytes)

0:020> !help FAQ
>> Where can I get the right version of SOS for my build?

If you are running version 1.1 or 2.0 of the CLR, SOS.DLL is installed in the
same directory as the main CLR dll (CLR.DLL). Newer versions of the
Windows Debugger provide a command to make it easy to load the right copy of

“.loadby sos clr”

That will load the SOS extension DLL from the same place that CLR.DLL is
loaded in the process. You shouldn’t attempt to use a version of SOS.DLL that
doesn’t match the version of CLR.DLL. You can find the version of
CLR.DLL by running

“lmvm clr”

in the debugger.  Note that if you are running CoreCLR (e.g. Silverlight)
then you should replace “clr” with “coreclr”.

If you are using a dump file created on another machine, it is a little bit
more complex. You need to make sure the mscordacwks.dll file that came with
that install is on your symbol path, and you need to load the corresponding
version of sos.dll (typing .load <full path to sos.dll> rather than using the
.loadby shortcut). Within the Microsoft corpnet, we keep tagged versions
of mscordacwks.dll, with names like mscordacwks_<architecture>_<version>.dll
that the Windows Debugger can load. If you have the correct symbol path to the
binaries for that version of the Runtime, the Windows Debugger will load the
correct mscordacwks.dll file.

>> I have a chicken and egg problem. I want to use SOS commands, but the CLR
isn’t loaded yet. What can I do?

In the debugger at startup you can type:

“sxe clrn”

Let the program run, and it will stop with the notice

“CLR notification: module ‘mscorlib’ loaded”

At this time you can use SOS commands. To turn off spurious notifications,

“sxd clrn”

>> I got the following error message. Now what?

0:000> .loadby sos clr
0:000> !DumpStackObjects
Failed to find runtime DLL (clr.dll), 0×80004005
Extension commands need clr.dll in order to have something to do.

This means that the CLR is not loaded yet, or has been unloaded. You need to
wait until your managed program is running in order to use these commands. If
you have just started the program a good way to do this is to type

bp clr!EEStartup “g @$ra”

in the debugger, and let it run. After the function EEStartup is finished,
there will be a minimal managed environment for executing SOS commands.

>> I have a partial memory minidump, and !DumpObj doesn’t work. Why?

In order to run SOS commands, many CLR data structures need to be traversed.
When creating a minidump without full memory, special functions are called at
dump creation time to bring those structures into the minidump, and allow a
minimum set of SOS debugging commands to work. At this time, those commands
that can provide full or partial output are:


For a minidump created with this minimal set of functionality in mind, you
will get an error message when running any other commands. A full memory dump
(obtained with “.dump /ma <filename>” in the Windows Debugger) is often the
best way to debug a managed program at this level.

>> What other tools can I use to find my bug?

Turn on Managed Debugging Assistants. These enable additional runtime diagnostics,
particularly in the area of PInvoke/Interop. Adam Nathan has written some great
information about that:


>> Does SOS support DML?

Yes.  SOS respects the .prefer_dml option in the debugger.  If this setting is
turned on, then SOS will output DML by default.  Alternatively, you may leave
it off and add /D to the beginning of a command to get DML based output for it.
Not all SOS commands support DML output.

0:020> !help HistInit

Before running any of the Hist – family commands you need to initialize the SOS
structures from the stress log saved in the debuggee.  This is achieved by the
HistInit command.

Sample output:

0:001> !HistInit
Attempting to read Stress log
facilitiesToLog  = 0xffffffff
levelToLog       = 6
MaxLogSizePerThread = 0×10000 (65536)
MaxTotalLogSize = 0×1000000 (16777216)
CurrentTotalLogChunk = 9
ThreadsWithLogs  = 3
Clock frequency  = 3.392 GHz
Start time         15:26:31
Last message time  15:26:56
Total elapsed time 25.077 sec
—————————- 2407 total entries —————————–

SUCCESS: GCHist structures initialized

0:020> !help HistStats

HistStat provides a number of garbage collection statistics obtained from the
stress log.

Sample output:

0:003> !HistStats
GCCount    Plugs Promotes   Relocs
2296        0       35       86
2295        0       34       85
2294        0       34       85

2286        0       32       83
2285        0       32       83
322        0       23       55
0        0        0        0
Root 01e411b8 relocated multiple times in gc 322
Root 01e411bc relocated multiple times in gc 322

Root 01e413f8 relocated multiple times in gc 322
Root 01e413fc relocated multiple times in gc 322

0:020> !help histroot
!HistRoot <root>

The root value obtained from !HistObjFind can be used to track the movement of
an object through the GCs.

HistRoot provides information related to both promotions and relocations of the
root specified as the argument.

0:003> !HistRoot 01e411b8
GCCount    Value       MT Promoted?                Notes
2296 028970d4 5b6c5cd8       yes
2295 028970d4 5b6c5cd8       yes
2294 028970d4 5b6c5cd8       yes
2293 028970d4 5b6c5cd8       yes
2292 028970d4 5b6c5cd8       yes
2291 028970d4 5b6c5cd8       yes
2290 028970d4 5b6c5cd8       yes
2289 028970d4 5b6c5cd8       yes
2288 028970d4 5b6c5cd8       yes
2287 028970d4 5b6c5cd8       yes
2286 028970d4 5b6c5cd8       yes
2285 028970d4 5b6c5cd8       yes
322 028970e8 5b6c5cd8       yes Duplicate promote/relocs

0:020> !help HistObj
!HistObj <obj_address>

This command examines all stress log relocation records and displays the chain
of GC relocations that may have led to the address passed in as an argument.
Conceptually the output is:

GenN    obj_address   root1, root2, root3,
GenN-1  prev_obj_addr root1, root2,
GenN-2  prev_prev_oa  root1, root4,

Sample output:
0:003> !HistObj 028970d4
GCCount   Object                                    Roots
2296 028970d4 00223fc4, 01e411b8,
2295 028970d4 00223fc4, 01e411b8,
2294 028970d4 00223fc4, 01e411b8,
2293 028970d4 00223fc4, 01e411b8,
2292 028970d4 00223fc4, 01e411b8,
2291 028970d4 00223fc4, 01e411b8,
2290 028970d4 00223fc4, 01e411b8,
2289 028970d4 00223fc4, 01e411b8,
2288 028970d4 00223fc4, 01e411b8,
2287 028970d4 00223fc4, 01e411b8,
2286 028970d4 00223fc4, 01e411b8,
2285 028970d4 00223fc4, 01e411b8,
322 028970d4 01e411b8,
0 028970d4

0:020> !help HistObjFind
!HistObjFind <obj_address>

To examine log entries related to an object whose present address is known one
would use this command. The output of this command contains all entries that
reference the object:

0:003> !HistObjFind 028970d4
GCCount   Object                                  Message
2296 028970d4 Promotion for root 01e411b8 (MT = 5b6c5cd8)
2296 028970d4 Relocation NEWVALUE for root 00223fc4
2296 028970d4 Relocation NEWVALUE for root 01e411b8

2295 028970d4 Promotion for root 01e411b8 (MT = 5b6c5cd8)
2295 028970d4 Relocation NEWVALUE for root 00223fc4
2295 028970d4 Relocation NEWVALUE for root 01e411b8

0:020> !help HistClear

This command releases any resources used by the Hist-family of commands.
Generally there’s no need to call this explicitly, as each HistInit will first
cleanup the previous resources.

visual studio 2008 memory leak/memory issue on x86 – the operation could not be completed.Not enough storage is available to complete this operation

And No I don’t have a solution for it and probably the only workaround is to make your visual studio Large Address Aware(3GB switch) on x86.

vs2008 error message

Steps to re-create

1. download and unzip http://debuggingblog.com/resources/transcripts.zip

2. open, close the xml file and try to load it the second time

3. If you load the xml file using IE8, you will see the followings once you close it

——————– State SUMMARY ————————–
TotSize (      KB)   Pct(Tots)  Usage
19e6f000 (  424380) : 20.24%   : MEM_COMMIT
f3b4000 (  249552) : 11.90%   : MEM_FREE
56dcd000 ( 1423156) : 67.86%   : MEM_RESERVE

Almost 1.4 GB Memory allocated in GC Segements for xml file is still reserved even after unloading the xml file.

However, visual studio 2008 is another story

0:000> !eeheap -gc
ephemeral segment allocation context: none
segment    begin allocated     size
01830000 01831000  027ecadc 0x00fbbadc(16497372)
12860000 12861000  137616c4 0x00f006c4(15730372)

We have bunch of 16MB GC segments and most of the objects are in gen 2.

0c55d1ec   739459     85777244 Microsoft.XmlEditor.XmlElement
0c559858  1496448     89786880 Microsoft.XmlEditor.Identifier
001f1918   105303     97472784      Free
793308ec  2369315    387475460 System.String
Total 9375571 objects

we have 90+ MB of free blocks and 380+MB in System.String. There are 2.36 million string objects, yeah so you don’t wanna pick each one of the string object to find GC root unless Microsoft or someone is paying you a dime to dump each object and aha a dump a day will make your day for sure.

0:000> !dumpheap -mt 0c9d4134
Address       MT     Size
018f241c 0c9d4134       68
5f610108 0c9d4134       68
total 2 objects
MT    Count    TotalSize Class Name
0c9d4134        2          136 Microsoft.XmlEditor.XmlDocumentProperties
Total 2 objects
0:000> !objsize 018f241c
sizeof(018f241c) =    507372388 (  0x1e3de364) bytes (Microsoft.XmlEditor.XmlDocumentProperties)
0:000> !objsize 5f610108
sizeof(5f610108) =    507371128 (  0x1e3dde78) bytes (Microsoft.XmlEditor.XmlDocumentProperties)

Did you just see that almost 1GB of virtual memory rooted in Microsoft.XmlEditor.XmlDocumentProperties? That’s just outrageous, I mean why would microsoft visual studio take up 1.2 GB of virtual memory to open a 58MB file, although It does make use of schema context cache.

0:000> !gcroot -nostacks 018f241c

GCHandle of type RefCnt is keeping reference to Microsoft.XmlEditor.XmlDocumentProperties

There is an OutOfMemoryException thrown with the following callstack

Exception object: 5ed00a34
Exception type: System.OutOfMemoryException
Message: Insufficient memory to continue the execution of the program.
InnerException: <none>
StackTrace (generated):
SP       IP       Function
0012F5A0 0C97E8B3 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure(Int32, Int32[])+0x3b
0012F5AC 0C9E94BB Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.Source.GetText()+0x3c
0012F5DC 0C9E9360 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.Source.BeginParse()+0×55
0012F644 0C9ECF38 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.Source.OnIdle(Boolean)+0×80
0012F654 0C9ECE28 Microsoft_VisualStudio_Package_LanguageService_9_0!Microsoft.VisualStudio.Package.LanguageService.OnIdle(Boolean)+0xd8
0012F674 0C9ECCDD Microsoft_XmlEditor!Microsoft.XmlEditor.XmlLanguageService.OnIdle(Boolean)+0×35
0012F684 0C9ECC34 Microsoft_XmlEditor!Microsoft.XmlEditor.Package.FDoIdle(UInt32)+0xc4


I hope this is fixed in Visual Studio 2010, I do need to try it out.