This problem was the most infuriating and time consuming problem that I’ve ever dealt with. The first part of this post might be a bit of rambling, so if you want to jump to the details, just click here.
I noticed the first symptoms when I couldn’t debug a web site from within Visual Studio 2010. The below error was being shown when the local development web server was trying to load.
WebDev.WebServer40.exe has encountered a problem and needs to close. We are sorry for the inconvenience.
I initially thought it was a bug in Visual Studio or the .NET Framework, so I verified all of the patches and services packs. After downloading and installing the Cassini development web server replacement for the default Visual Studio web server, I continued to get the same error.
The system event logs showed two error occurring relating to the vshost.exe process.
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791A8BBD (79140000) with exit code 80131506.
Faulting application MyWebSite.vshost.exe, version 10.0.30319.1, stamp 4ba2084b, faulting module clr.dll, version 4.0.30319.1, stamp 4ba1d9ef, debug? 0, fault address 0x00068bbd.
I tried uninstalling all of the .NET Frameworks using the .NET Framework Cleanup Tool. Then I reinstalled all of the Frameworks, but that didn’t change the problem.
The Microsoft .NET CLR team was able to use the dmp files that I uploaded via Microsoft Connect to inform me that there was a problem with the WebDev.WebServer40.exe file, but after they inspected the file, they found that the error didn’t match the file. The dmp file said that the initial bits of the exe were 0s, but the file’s initial bits weren’t. This lead them to believe there was a RootKit installed that was interfering with the execution of the application.
The Microsoft .NET CLR tech support responded saying:
1. When loaded into your process, 8 bytes at offset 0x168 into the .exe files are being set to 0. This is the issue that is causing the CLR to fail. These bits are set correctly in the file you provided to me.
2. C:WindowsSystem32Detoured.dll is loaded into the process. This means to me that the detours library is being used on your machine to modify the behavior of these processes.
Meanwhile, I discovered that I could not run any .NET application – either built by myself or third-party. Even some commercial applications such as Toad and Ad-aware would crash when opening. These third party applications crashed and left the following errors in the system event log:
Source: Application Popup
Description: Application popup: #APPNAME#.exe – Application Error : The application failed to initialize properly (0xc000007b). Click on OK to terminate the application.
Applications built using version 4 of the .NET Framework would return the following error in the system event log:
Source: .NET Runtime
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791A8BBD (7914000) with exit code 80131506.
Applications built using version 2 of the .NET Framework would return the following error in the system event log:
Source: .NET Runtime
Description: .NET Runtime version 2.0.50727.3053 – Fatal Execution Engine Error (7A097706) (80131506)
Suspect #1 Bit9 Parity Agent
At first, we believed the Rootkit that was messing up the execution was the corporate installed/mandated Bit9 Parity Agent. Parity Agent is an application white listing Windows Service that restricts execution of applications. After turning off Bit9 Parity Agent service, I continued to get the same errors.
Suspect #2 Symantec Anti-Virus
After reading the Symantec forums, I figured that the Dell’s Embassy Trust Suite must be the problem, so I uninstalled and rebooted. That didn’t fix the problem.
The Symantec forum user also recommended uninstalling the “Firewall and Intrusion Prevention” and the “Application and Device Control” modules. To do this, use the Add or Remove Software control panel option and click the “Change or Modify” option of Symantec Endpoint Protection software.
After disabling the Application and Device Control Symantec Endpoint software, I continued to receive the same errors.
Safe Mode vs Normal Mode
I discovered that when I booted into safe mode, I am able to run .NET applications without any problems. This means the .NET Framework and the applications weren’t the problems. After spending most of the day trying to figure out what’s different between Safe Mode and Normal Mode, I was unable to pinpoint the problem.
“Run As” another user, as myself
I also discovered that if I right click on a .NET application and choose “Run As” then enter my current network logon (the same identity that I was already logged in as), then the application works. I am able to run WinForms and console application and the Cassini Web Server.
Sys Internals and Debugging
I tried renaming c:windowssystem32detoured.dll, but that just caused more headaches. Dr Watson hung and wasn’t able to submit the debug crash dmp files to Microsoft, other apps just hung and were unable to be killed.
Using DebugView Monitoring Tool
I downloaded DebugView, but I was unable to see anything other than the application crashing. It didn’t show what else might be causing the crash.
Using Rootkit Revealer
I downloaded RootkitRevealer and it found a handful of discrepancies with a description of “Error dumping hive: The system cannot find the file specified.”
Using Process Monitor
I downloaded Process Monitor and found it quite useful. I was able to filter and find all of the processes that use c:windowssystem32detoured.dll. The result was the following processes (in order):
So my .NET application successfully calls detoured.dll, then svchost.exe, then DW20.exe (Dr Watson error is generated), then svchost.exe. Nothing out of the ordinary…
When I use “Run As” and watch the process calls, the application loads and the following processes call c:windowssystem32detoured.dll
- svchost.exe (about 10 seconds after application UI loads)
- All .NET applications including Microsoft Development WebServer, Toad, Ad-Adware, and any developed console or WinForms applications crash when trying to run.
- These same applications will work if right click and choose “Run As” and use same user credentials as currently logged on user.
- These same applications will work if booted into “Safe Mode.”
Suspected Applications (Cleared, no fault)
- Bit9 Parity Agent
- Symantec Anti-Virus
- .NET Framework
- Microsoft Visual Studio
- Use “Run As” to run .NET applications
I have been unable to determine a solution.
- WebDev.WebServer40.exe has encountered a problem and needs to close (Microsoft Connect)
- .NET Framework Cleanup Tool (Aaron Stebner’s WebLog)
- Endpoint kills my app (Symantec Forums)
- DebugView for Windows v4.76 (Microsoft TechNet, Sys Internals)
- RootkitRevealer (Microsoft TechNet, Sys Internals)
- Process Monitor (Microsoft TechNet, Sys Internals)