Do you dislike a situation where applications on your machine have been working fine for weeks and months at a time and all of a sudden they begin to run very slowly? Well, I do too, especially when it comes to applications I use all day & every day.
At any given point in time I usually have Visual Studio and/or WinDbg running and today I noticed that my Visual Studio 2010 started to consume more CPU cycles than it does usually, and it was loading debug symbols for a release mode postmortem dump very, very slowly. First thing I did was to open the same dump in WinDbg, and guess what? WinDbg also was acting exactly the same. It was consuming approximately 30 to 40 percent of CPU cycles while in .reload /f /v and it was taking its sweet time.
On my machine, I use _NT_SYMBOL_PATH environment variable that controls my path to local symbol cache and remote symbol servers, and it is applicable to all debuggers that I use. I must point out that while I did come across several blogs that recommend against using this variable, it isn’t the root cause of the problem I had experienced today. After all, my debuggers were working fine last week (Friday) and not today (Monday).
So, what changed?
While I didn’t change anything, my machine is a member of a corporate domain so it is quite possible that “something” was changed on my machine for me.
Troubleshooting approach: simultaneous sniffer and process monitor capture on my machine while reloading symbols in VS 2010 or WinDbg.
I noticed 3 to 7 second delays originating from my machine between each request to a remote symbol server (in this case: Microsoft public symbol server). Meaning: WinDbg or VS 2010 were blocked / busy doing something else before they even get to the point where they ask remote symbol server for a file, if it is missing in my cache.
I also noticed excessive registry activity surrounding the registry keys shown below.
So, overall behavior can be described as follows:
1.WinDbg / VS 2010 checks local symbol cache.
2.If symbols are missing for a given module, delays occur during lots of registry activity under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip and subsequent DNS queries.
3.HTTP request finally goes out to download debug symbols and WinDbg / VS 2010 stored those in your local cache.
Clearly, majority of the observed delays I experienced today came from step 2 above. Upon review of the call stacks for the above mentioned registry activity I noticed that jsproxy.dll (JScript Proxy Auto-Configuration DLL) was present in the stack with an InternetInitializeAutoProxyDll() function offset.
Subsequently, I’ve reviewed my proxy setting. I expected to see my manual proxy configuration pointing to our corporate proxy server and a fairly extensive bypass list that I’ve spent lots of time tweaking to ensure optimal performance. To my big surprise I saw that my custom settings were deleted and overwritten with “Automatically detect settings”.
There are many articles out there that describe similar slowness when you use “Automatically detect settings”, and I’m not going to regurgitate that info. I restored manual proxy settings, restarted VS 2010 and WinDbg and the issue was resolved.
Who pushed this domain policy to my machine and why is a topic for another day.
Until next time,