Dr Watson will be there untill the end of this world

Scenario

A win32 user mode Application is crashing and support is not able to get a crash dump either using adplus.vbs script or using CDB or NTSD debugger. It failed to generate crash dump even after setting WinDbg as the default post mortem debugger.

Steps to resolution

1) Crate a adplus configuration file to generate a minidump on the following breakpoints

  • kernel32!UnhandledExceptionFilter
  • kernel32!SetUnhandledExceptionFilter

Analyze the stack trace

1)  Open the first crash dump with breakpoint on kernel32!UnhandledExceptionFilter, run the following command

0:000> kb
ChildEBP RetAddr  Args to Child
0012ebf4 6da6c8ac 0012ec54 002ef7f0 c0000417 kernel32!UnhandledExceptionFilter
0012ef2c 6da302fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xf9
0012ef74 6da30366 047d0030 ffffffff 00000001 msvcr90!fread_s+0×4a

Something interesting, why is it calling _invoke_watson?

2) Open the 2nd crash dump with breakpoint on kernel32!SetUnhandledExceptionFilter and run the following command

0:000> kb
ChildEBP RetAddr  Args to Child
0012ebf4 6da6c89f 00000000 002ef7f0 c0000417 kernel32!SetUnhandledExceptionFilter
0012ef2c 6da302fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xec
0012ef74 6da30366 047d0030 ffffffff 00000001 msvcr90!fread_s+0×4a

Interesting, SetHandledExceptionFilter is passed NULL . MSDN documentation says, A value of NULL for this parameter specifies default handling within UnhandledExceptionFilter so that means a pointer to a top-level exception filter function is nulled out.

stack trace

002eebb4 6dcbc8ac 002eec14 0040f7f0 c0000417 kernel32!UnhandledExceptionFilter

002eeeec 6dc802fd 00000000 00000000 00000000 msvcr90!_invoke_watson+0xf9

002eef34 6dc80366 045b0030 ffffffff 00000001 msvcr90!fread_s+0×4a

002eef50 6ef3105e 045b0030 00000001 000001f4 msvcr90!fread+0×18

Root Cause Analysis

1) Basically, CRT is hardcoded to invoke drwatson, in our case msvcr90!fread under a few scenario

2) Crash dump was not getting created because it was using fread with invalid runtime parameter resulting in calling hardcoded dr watson

3) Event log also shows the Exception Code = 0xC0000417, defined as #define STATUS_INVALID_CRUNTIME_PARAMETER ((NTSTATUS)0xC0000417L) in “NTSTATUS.h”

4) This was confirmed in the following article on microsoft web site A proposal to make Dr.Watson invocation configurable

When any of CRT invariants are violated (invalid arguments passed, buffer overrun detected, etc.) CRT needs to report somehow about that violation. VC++ 2005 uses the following strategy:
1. Display (optionally) a message box with error description.
2. Invoke Dr.Watson to create crash dump.

(All this code is located in abort.c, gs_report.c, and invarg.c files).

prior to windows vista, DrWatson is installed as the default post mortem debugger. But you always have an option to set WinDbg or NTSD or any other debugger to change post mortem debugger. However, you don’t have Dr Watson in Windows Vista ( now you have windows error reporting service). So, We ran into an interesting issue while trying to get a crash dump for a user mode win32 process. You have to hook your own unhandled exception filter to create a dump