Interesting post on Microsoft Advanced Windows Debugging Blog
cool stuff from microsoft research team
Interesting post on Microsoft Advanced Windows Debugging Blog
cool stuff from microsoft research team
Ilya Ryzhenkov has reported that the command to dump gc handles by type (!gcht) hangs his visual studio. He has provided the call stack for all the threads when it hangs.
If any of you run into same issue, please do let me know and even better if you can provide a hang dump
. I will continue to use this command in post mortem debugging and at customer sites to fix any issue I encounter.
Click to Download access violation adplus configuration file
ADPlus Configuration file to the rescue
Someone asked me about getting a memory dump on breakpoints in production environment.
Usually, you won’t have the luxury of attaching a debugger and inserting a breakpoint in production environment. However, you can still get a memory dump under different conditions or execute a command line option using ADPlus configuration file. AdPlus does support -hang switch to take a memory dump of a process anytime but that’s not good enough if you need to take a memory dump on a particular first chance exception or even when a breakpoint is hit.
ADPlus script supports configuration file with -c switch to create a memory dump of a user mode win32 process. You should read more on ADPlus configuration file on WinDbg help.
Below is the example of a ADPlus configuration file, which will create a memory dump under the following conditions
<ADPlus>
<!– RunMode could be crash or hang, Quiet suppresses the warning message box–>
<Settings>
<RunMode> CRASH </RunMode>
<Option> Quiet </Option>
<ProcessName> <process name><!–e.g. cmd.exe–> </ProcessName>
</Settings>
<!–
PreCommands is included to change the symbol path for kernel32.dll, the first command .sympath sets the symbol path to c:\windows\system32(kernel32.dll location and the 2nd command reload the kernel32.dll defaulting to export symbols.
The reason for loading the export symbols has to do with setting a breakpoint in kernel32.dll functions as described in my last blog entry
–>
<PreCommands>
<Cmd> .sympath c:\windows\system32 </Cmd>
<Cmd> .reload /f kernel32.dll </Cmd>
</PreCommands>
<Exceptions>
<Config>
<Code>0×80000001</Code>
<Actions1> MiniDump </Actions1>
<Actions2> FullDump </Actions2>
</Config>
</Exceptions>
<Breakpoints>
<NewBP>
<Type> BM </Type>
<Address> kernel32.dll!UnhandledExceptionFilter </Address>
<Actions> FullDump</Actions>
<CustomActions> r </CustomActions>
</NewBP>
<NewBP>
<Type> BM </Type>
<Address> kernel32.dll!SetUnhandledExceptionFilter </Address>
<Actions> MiniDump </Actions>
<CustomActions> r </CustomActions>
</NewBP>
</Breakpoints>
</ADPlus>
adplus command to execute configuration file(exception.cfg)
cscript.exe adplus.vbs -c exception.cfg
While debugging crash dump generation issue as described in blog on Dr Watson gotcha, I noticed that you can’t set a breakpoint on kernerl32 functions since microsoft symbols server gives you access to stripped public symbols only. This is one of those scenario where you would rather have export symbols.
Scenario
While doing live debugging or attaching a debugger to generate a dump when it hits a breakpoint on kernel32!SetUnhandledExceptionFilter
Steps using WinDbg
Run the following command
0:021> bm kernel32!SetUnhandledExceptionFilter
You can use bm to set a symbol breakpoint that matches the pattern.
Gotcha
If you have symbol server path set correctly pointing to microsoft public symbol server, WinDbg will display the following message and it suggest you to switch to export symbols
No matching code symbols found, no breakpoints set.
If you are using public symbols, switch to full or export symbols.
How to Switch to Export Symbols to set a breakpoint?
Run the following commands
Symbol search path is: c:\windows\system32
This will set your symbol path to kernel32.dll which should be under your windows system folder in my case it is “c:\windows\system32″
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Windows\system32\kernel32.dll -
Don’t worry about ERROR message because this is what we want, we want it be to set to export symbols. .reload command will reload the symbols for kernerl32.dll defaulting to export symbols
breakpoint 6 redefined
6: 7617d16f @!”kernel32!SetUnhandledExceptionFilter”
And your breakpoint is set using export symbols and of course you can use depends for all the exported symbols.
Scenario
A win32 user mode Application is crashing and support is not able to get a crash dump either using adplus.vbs script or using CDB or NTSD debugger. It failed to generate crash dump even after setting WinDbg as the default post mortem debugger.
Steps to resolution
1) Crate a adplus configuration file to generate a minidump on the following breakpoints
Analyze the stack trace
1) Open the first crash dump with breakpoint on kernel32!UnhandledExceptionFilter, run the following command
0:000> kb
ChildEBP RetAddr Args to Child
0012ebf4 6da6c8ac 0012ec54 002ef7f0 c0000417 kernel32!UnhandledExceptionFilter
0012ef2c 6da302fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xf9
0012ef74 6da30366 047d0030 ffffffff 00000001 msvcr90!fread_s+0×4a
Something interesting, why is it calling _invoke_watson?
2) Open the 2nd crash dump with breakpoint on kernel32!SetUnhandledExceptionFilter and run the following command
0:000> kb
ChildEBP RetAddr Args to Child
0012ebf4 6da6c89f 00000000 002ef7f0 c0000417 kernel32!SetUnhandledExceptionFilter
0012ef2c 6da302fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xec
0012ef74 6da30366 047d0030 ffffffff 00000001 msvcr90!fread_s+0×4a
Interesting, SetHandledExceptionFilter is passed NULL . MSDN documentation says, A value of NULL for this parameter specifies default handling within UnhandledExceptionFilter so that means a pointer to a top-level exception filter function is nulled out.
stack trace
002eebb4 6dcbc8ac 002eec14 0040f7f0 c0000417 kernel32!UnhandledExceptionFilter
002eeeec 6dc802fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xf9
002eef34 6dc80366 045b0030 ffffffff 00000001 msvcr90!fread_s+0×4a
002eef50 6ef3105e 045b0030 00000001 000001f4 msvcr90!fread+0×18
Root Cause Analysis
1) Basically, CRT is hardcoded to invoke drwatson, in our case msvcr90!fread under a few scenario
2) Crash dump was not getting created because it was using fread with invalid runtime parameter resulting in calling hardcoded dr watson
3) Event log also shows the Exception Code = 0xC0000417, defined as #define STATUS_INVALID_CRUNTIME_PARAMETER ((NTSTATUS)0xC0000417L) in “NTSTATUS.h”
4) This was confirmed in the following article on microsoft web site A proposal to make Dr.Watson invocation configurable
When any of CRT invariants are violated (invalid arguments passed, buffer overrun detected, etc.) CRT needs to report somehow about that violation. VC++ 2005 uses the following strategy:
1. Display (optionally) a message box with error description.
2. Invoke Dr.Watson to create crash dump.
(All this code is located in abort.c, gs_report.c, and invarg.c files).
prior to windows vista, DrWatson is installed as the default post mortem debugger. But you always have an option to set WinDbg or NTSD or any other debugger to change post mortem debugger. However, you don’t have Dr Watson in Windows Vista ( now you have windows error reporting service). So, We ran into an interesting issue while trying to get a crash dump for a user mode win32 process. You have to hook your own unhandled exception filter to create a dump
Download pdf version from here
Introduction
WinDBG is a debugging tool from Microsoft for user and kernel mode debugging. WinDBG is a GUI interface built on CDB, NTSD and KD along with debugging extensions. We will mainly be using WinDBG along with SOS extension for managed code debugging.
WinDBG is a very powerful tool for debugging and it allows you to set a breakpoint, view source code using symbol files, view stack trace, parameters to a method, memory and registers. Please refer to the below link to download WinDBG
http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx
Once you have installed Debugging Tools for Windows, you can instantiate WinDBG from Start->Programs->Debugging Tools for Windows(x86)->WinDbg
We recommend you to read and read and read WinDBG Help that will be your only source on documentation.
Below are the options you have to access WinDbg Help
Start->Programs->Debugging Tools for Windows(x86)->Debugging Help (on x86)


Please note that when you see a command starting with kd> on help, these commands are for kernel debugging. Those KD commands will not run in user mode debugging. They can’t be used to debug a win32 user mode process.
Memory Dumps
Memory dump is a snapshot of the memory and register at the time of crash. You can also collect a memory of a process to get the snapshot of a process’s memory at a particular instance.
User Mode Dump
User Mode Dump includes the virtual memory information, process environment block, process and thread structures to assist in debugging.
Full Dump – includes all the information about a process(exe) including committed virtual memory page, thread and process data structures, handle data, loaded and unloaded modules and their addresses.
Mini Dump – includes only specific memory snapshot of a process. Although, the name is minidump but it can be larger than full dump depending on the dump option you choose.
As described in WinDBG help
The name “minidump” is misleading, because the largest minidump files actually contain more information than the “full” user-mode dump. For example, .dump /mf or .dump /ma will create a larger and more complete file than .dump /f. For this reason, .dump /m[MiniOptions] recommended over .dump /f for all user-mode dump file creation.
If you are creating a minidump file with the debugger, you can choose exactly what information to include. A simple .dump /m command will include basic information about the loaded modules that make up the target process, thread information, and stack information.
What dump to collect for a Win32 User Mode Process?
As described in ADPlus.vbs
ADPlus has 3 modes of operation: ‘Hang‘, ‘Quick‘ and ‘Crash‘ mode.
Hang Mode
In ‘Hang’ mode, ADPlus assumes that a process is hung and it will attach a debugger to the process(s) specified on the command line. After the debugger is attached to the process(s) ADPlus will dump the memory of each process to a .dmp file for later analysis with a debugger (such as WinDBG). In this mode, processes are paused briefly while their memory is being dumped to a file and then resumed.
Quick Mode
In ‘-quick’ mode ADPlus assumes that a process is hung and it will attach a debugger to the process(s) specified on the command line with either the ‘-p’ or ‘-pn’ or ‘-iis’ switches. After the debugger is attached to the process(s) ADPlus will create mini dumps for each process, containing commonly requested debug information, rather than dumping the entire memory for each process.’Quick’ mode is generally faster than ‘Hang’ mode, but requires symbols to be installed on the server where ADPlus is running.
Crash Mode
In ‘Crash’ mode, ADPlus assumes that a process will crash and it will attach a debugger to the process(s) specified on the command line with either the ‘-p’ or ‘-pn’ or ‘-iis’ switches. After the debugger is attached to the process(s)ADPlus will configure the debugger to log ‘first chance’ access violations (AV’s) to a text file. When a ‘second chance’ access violation occurs, the processes memory is dumped to a .dmp file for analysis with a debugger such as WinDBG. In this mode, a debugger remains attached to the process(s) until the process exits or the debugger is exited by pressing CTRL-C in the minimized debugger window. When the process crashes, or CTRL-C is pressed, it will need to be re-started.
Hang Dump Check List
Memory Leak?
High CPU Usage, Performance?
Process Hang/Deadlock?
Analyze the state of an application at a particular instance?
Crash Dump Check List
Application crashing due to unhandled exception?
Unexpected behavior due to a handled exception or a first chance exception?
How to Collect Memory Dump using ADPlus(visual basic script)
ADPlus.vbs file is installed in a default location with Debugging tools for Windows

Steps to collect hang memory dump using ADPlus
If WinDBG is not installed on a system, follow http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx to download WinDbg for 32bit OS or 64 bit OS depending on the version of windows OS .
Please note that WinDBG supports XCOPY so you can either install it directly on a system or you can choose to install it on a test system and copy the installed folder on a production machine. Sometime XCOPY is desirable for security reasons and if you don’t want to get a permission to install a software. By Default, Debugging Tools for Windows get installed under “\Program Files\Debugging Tools for Winodws\”
cscript.exe – to specify cscript.exe script interpreter to run vbscript.exe otherwise it will try to set cscript.exe as default script interpreter
adplus.vbs – vbscript file to generate memory dump
-hang – As described in adplus.vbs file(copy and paste), ADPlus has 3 modes of operation: ‘Hang’, ‘Quick’ and ‘Crash’ mode. In ‘Hang’ mode, ADPlus assumes that a process is hung and it will attach a debugger to the process(s) specified on the command line with either the ‘-p’ or ‘-pn’ or ‘-iis’ switches. After the debugger is attached to the process(s) ADPlus will dump the memory of each process to a .dmp file for later analysis with a debugger (such as WinDBG). In this mode, processes are paused briefly while their memory is being dumped to a file and then resumed.
-pn – Adplus supports -pn and -p switch to specify the name or process id of an application
notepad.exe – process to be debugged to generate memory dump
-quiet – ADPlus support quiet mode to suppress the dialog boxes as shown below in Figure 2 and Figure 3

Figure 1

Figure 2

Figure 3
The name of executable specified in -pn switch must not contain drive or folder path. Process name should be specified as shown in the Task Manager.
By default, the Adplus script will create subfolder similar to the following Hang_Mode__Date_mm-ddyyyy__Time_hh-mm-ssAM/PM. This subfolder will be created under C:\WinDBG\, folder will contain debugger logs and a memory dump.

Important Note
Steps to collect crash memory dump using ADPlus
same as above just replace the -hang switch to -crash.
Followings are the additional switches you should know to collect crash dump
Continue – WinDBG commands and Dump Analysis in Beginner’s Guide to WinDBG – Part 2
I went through the pain of dumping objects after objects just to see the member instance. I took a leaf out from John Robbin’s book on Managed Debugging. He also mentioned that he didn’t appreciate the fact that microsoft removed the recursive dump in .net 2.0. I wrote sos2.dll (a WinDbg extension) which supports command to dump the objects and its members recursively. It also supports 2 other commands one to dump heap by generation number and another to get the object’s generation. Please note that dumping the heap by generation is as fast as sos.dll since it loads the sos.dll to dump the sorted heap. This is really useful when you have more than one heap because you won’t have to figure out the generation start and end address for all the heaps.
Object Inspection
—————————–
ObjectGeneration (og)
DumpGenHeap (dg)
RecursiveDumpObject (rdo)
——————————
Find an Object’s GC Generation
——————————
ObjectGeneration (og)
!og <object address>
0:000> !og 023a2eac
GC Heap Generation # 2
——————————
Dump Generational Heap
——————————
DumpGenHeap (dg)
!dg -stat <generationnumber>
Dump all the objects on heap for Generation # 1
0:000> !dg 1
Dump the statistical summay for Generation # 2
0:000> !dg -stat 2
Dump the statistical summay for Generation # 2
0:000> !dg -stat
Dump all the objects on heap for Generation # 2
(By Default, it will dump all the objects on generation #2 because that’s what usually we look at)
——————————
Object Inspection Recursively
——————————
RecursiveDumpObject (rdo)
!rdo [-nofields] <object address> < recursion level>
This command allows you to examine the fields of an object, as well as view the properties of the object such as the EEClass, the MethodTable, and the size. This command works same as !DumpObj from sos.dll but it also allows you to dump the member of the objects with recursion level option, otherwise you will have to keep running !DumpObj command with member instance address If you specify no recursion level or 0, it will just dump an object on managed heap same as DumpObj as shown below
0:000> !rdo 01cd17ec
Name: SOS2TestHarness.Oregon
MethodTable: 001d311c
EEClass: 001d1410
Size: 20(0×14) bytes
(C:\Dev\SOS2TestHarness.exe)
Fields:
MT Field Offset Type VT Attr Value Name
790fd8c4 4000001 4 System.String 0 instance 01cd1680 _name
001d31bc 4000002 8 …TestHarness.Ocean 0 instance 01cd1a28 _ocean
79101fe4 4000003 c …ections.Hashtable 0 instance 01cd1800 _keyValuePair
As shown Below when you specify the recursion level, it will also dump the member objects
0:000> !rdo 01cd17ec 1
Name: SOS2TestHarness.Oregon
MethodTable: 001d311c
EEClass: 001d1410
Size: 20(0×14) bytes
(C:\Dev\SOS2TestHarness.exe)
Fields:
MT Field Offset Type VT Attr Value Name
790fd8c4 4000001 4 System.String 0 instance 01cd1680 _name
****** Dump _name(member instance) *******
Name: System.String
MethodTable: 790fd8c4
EEClass: 790fd824
Size: 32(0×20) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Pacific
Fields:
MT Field Offset Type VT Attr Value Name
……………………………………………………………….
001d31bc 4000002 8 …TestHarness.Ocean 0 instance 01cd1a28 _ocean
****** Dump _ocean(member instance) *******
Name: SOS2TestHarness.Ocean
MethodTable: 001d31bc
EEClass: 001d1474
Size: 12(0xc) bytes
(C:\Dev\SOS2TestHarness.exe)
Fields:
MT Field Offset Type VT Attr Value Name
001d324c 4000004 4 …TestHarness.River 0 instance 01cd1a34 _river
79101fe4 4000003 c …ections.Hashtable 0 instance 01cd1800 _keyValuePair
****** Dump _keyValuePair(member instance) *******
Name: System.Collections.Hashtable
MethodTable: 79101fe4
EEClass: 79101f74
Size: 56(0×38) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
7912d9bc 400092b 4 …ashtable+bucket[] 0 instance 01cd1838 buckets
79102290 400092c 1c System.Int32 0 instance 2 count
………………………………………………………………….