Do you dislike a situation where applications on your machine have been working fine for weeks and months at a time and all of a sudden they begin to run very slowly? Well, I do too, especially when it comes to applications I use all day & every day.
At any given point in time I usually have Visual Studio and/or WinDbg running and today I noticed that my Visual Studio 2010 started to consume more CPU cycles than it does usually, and it was loading debug symbols for a release mode postmortem dump very, very slowly. First thing I did was to open the same dump in WinDbg, and guess what? WinDbg also was acting exactly the same. It was consuming approximately 30 to 40 percent of CPU cycles while in .reload /f /v and it was taking its sweet time.
On my machine, I use _NT_SYMBOL_PATH environment variable that controls my path to local symbol cache and remote symbol servers, and it is applicable to all debuggers that I use. I must point out that while I did come across several blogs that recommend against using this variable, it isn’t the root cause of the problem I had experienced today. After all, my debuggers were working fine last week (Friday) and not today (Monday).
So, what changed?
While I didn’t change anything, my machine is a member of a corporate domain so it is quite possible that “something” was changed on my machine for me.
Troubleshooting approach: simultaneous sniffer and process monitor capture on my machine while reloading symbols in VS 2010 or WinDbg.
Observations:
I noticed 3 to 7 second delays originating from my machine between each request to a remote symbol server (in this case: Microsoft public symbol server). Meaning: WinDbg or VS 2010 were blocked / busy doing something else before they even get to the point where they ask remote symbol server for a file, if it is missing in my cache.
I also noticed excessive registry activity surrounding the registry keys shown below.
