Jim's Blog Ramblings about novels, comics, programming, and other geek topics

30Sep/107

Troubleshooting Internal Error in the .NET Runtime

Google AdSense

This problem was the most infuriating and time consuming problem that I’ve ever dealt with.  The first part of this post might be a bit of rambling, so if you want to jump to the details, just click here.

I noticed the first symptoms when I couldn’t debug a web site from within Visual Studio 2010. The below error was being shown when the local development web server was trying to load.

WebDev.WebServer40.exe has encountered a problem and needs to close. We are sorry for the inconvenience.

I initially thought it was a bug in Visual Studio or the .NET Framework, so I verified all of the patches and services packs. After downloading and installing the Cassini development web server replacement for the default Visual Studio web server, I continued to get the same error.

The system event logs showed two error occurring relating to the vshost.exe process.

Event 1:
Application: MyWebSite.vshost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791A8BBD (79140000) with exit code 80131506.

Event 2:
Faulting application MyWebSite.vshost.exe, version 10.0.30319.1, stamp 4ba2084b, faulting module clr.dll, version 4.0.30319.1, stamp 4ba1d9ef, debug? 0, fault address 0x00068bbd.

 

I tried uninstalling all of the .NET Frameworks using the .NET Framework Cleanup Tool. Then I reinstalled all of the Frameworks, but that didn’t change the problem.

The Microsoft .NET CLR team was able to use the dmp files that I uploaded via Microsoft Connect to inform me that there was a problem with the WebDev.WebServer40.exe file, but after they inspected the file, they found that the error didn’t match the file. The dmp file said that the initial bits of the exe were 0s, but the file’s initial bits weren’t. This lead them to believe there was a RootKit installed that was interfering with the execution of the application.

The Microsoft .NET CLR tech support responded saying:

1. When loaded into your process, 8 bytes at offset 0x168 into the .exe files are being set to 0. This is the issue that is causing the CLR to fail. These bits are set correctly in the file you provided to me.

2. C:WindowsSystem32Detoured.dll is loaded into the process. This means to me that the detours library is being used on your machine to modify the behavior of these processes.

Meanwhile, I discovered that I could not run any .NET application – either built by myself or third-party. Even some commercial applications such as Toad and Ad-aware would crash when opening. These third party applications crashed and left the following errors in the system event log:

Source: Application Popup
Description: Application popup: #APPNAME#.exe - Application Error : The application failed to initialize properly (0xc000007b). Click on OK to terminate the application.

Applications built using version 4 of the .NET Framework would return the following error in the system event log:

Source: .NET Runtime
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 791A8BBD (7914000) with exit code 80131506.

Applications built using version 2 of the .NET Framework would return the following error in the system event log:

Source: .NET Runtime
Description: .NET Runtime version 2.0.50727.3053 – Fatal Execution Engine Error (7A097706) (80131506)

 

Suspect Applications

Suspect #1 Bit9 Parity Agent

At first, we believed the Rootkit that was messing up the execution was the corporate installed/mandated Bit9 Parity Agent. Parity Agent is an application white listing Windows Service that restricts execution of applications. After turning off Bit9 Parity Agent service, I continued to get the same errors.

 

Suspect #2 Symantec Anti-Virus

After reading the Symantec forums, I figured that the Dell’s Embassy Trust Suite must be the problem, so I uninstalled and rebooted. That didn’t fix the problem.

The Symantec forum user also recommended uninstalling the “Firewall and Intrusion Prevention” and the “Application and Device Control” modules. To do this, use the Add or Remove Software control panel option and click the “Change or Modify” option of Symantec Endpoint Protection software.

After disabling the Application and Device Control Symantec Endpoint software, I continued to receive the same errors.

 

Plot Twists

Safe Mode vs Normal Mode

I discovered that when I booted into safe mode, I am able to run .NET applications without any problems. This means the .NET Framework and the applications weren’t the problems. After spending most of the day trying to figure out what’s different between Safe Mode and Normal Mode, I was unable to pinpoint the problem.

 

“Run As” another user, as myself

I also discovered that if I right click on a .NET application and choose “Run As” then enter my current network logon (the same identity that I was already logged in as), then the application works. I am able to run WinForms and console application and the Cassini Web Server.

 

Sys Internals and Debugging

Renaming detoured.dll

I tried renaming c:windowssystem32detoured.dll, but that just caused more headaches. Dr Watson hung and wasn’t able to submit the debug crash dmp files to Microsoft, other apps just hung and were unable to be killed.

 

Using DebugView Monitoring Tool

I downloaded DebugView, but I was unable to see anything other than the application crashing. It didn’t show what else might be causing the crash.

 

Using Rootkit Revealer

I downloaded RootkitRevealer and it found a handful of discrepancies with a description of “Error dumping hive: The system cannot find the file specified.”

 

Using Process Monitor

I downloaded Process Monitor and found it quite useful. I was able to filter and find all of the processes that use c:windowssystem32detoured.dll. The result was the following processes (in order):

  1. MyApplication.exe
  2. svchost.exe
  3. DW20.exe
  4. svchost.exe

So my .NET application successfully calls detoured.dll, then svchost.exe, then DW20.exe (Dr Watson error is generated), then svchost.exe. Nothing out of the ordinary...

When I use “Run As” and watch the process calls, the application loads and the following processes call c:windowssystem32detoured.dll

  1. MyApplication.exe
  2. svchost.exe (about 10 seconds after application UI loads)

 


Quick Notes

Symptoms

  • All .NET applications including Microsoft Development WebServer, Toad, Ad-Adware, and any developed console or WinForms applications crash when trying to run.
  • These same applications will work if right click and choose “Run As” and use same user credentials as currently logged on user.
  • These same applications will work if booted into “Safe Mode.”

Suspected Applications (Cleared, no fault)

  • Bit9 Parity Agent
  • Symantec Anti-Virus
  • .NET Framework
  • Microsoft Visual Studio

Work-Arounds

  • Use “Run As” to run .NET applications

 

Solution

I have been unable to determine a solution.

 

Resources

James Welch

James Welch is a software engineer in Vermont working for a large information technology company and specializing in .NET. Additionally, he holds a Master’s Degree in Software Engineering and a Bachelor of Science Degree in Computer Science. Jim also enjoys local craft beer, comic books, and science-fiction and fantasy novels, games, and movies.

Twitter Google+ 

Comments (7) Trackbacks (0)
  1. have you ever find a solution for this?

    I’m having the same error and can’t track the cause of it.

    tnx,
    Shlomi

  2. No. I never did figure out the root cause. My corporate laptop lease expired around the same time this happened, so I ended up getting a new laptop before I could figure it out.

  3. Do you remember/know what type of AD your company was using if any? (2003,2008,2008R2) I am fighting this on a wide scale on my servers that are in house. All of my research and tinkering has pushed me to permissions as the culprit somewhere. Just hoping that your network environment is similar to mine so I can know I am going down a legit road.

    • We are running 2008R2 right now. I don’t remember what we were using when I posted this blog entry and had the problems. I’d imagine we were up to date on the releases, so if R2 was out in Sep 2009, then we were probably on R2 then too.

  4. I had the same problem. The cause was the SCardSvc service for a smart card reader my Dell Latitude laptop has. I don’t use the card reader, so i disabled the service and no more problems.

  5. This is an old post, but I was having the same issue and found an MSDN post that references the same symptoms, but also has what looks like a reliable fix. In case it helps anyone, here’s that post: http://blogs.msdn.com/b/carloc/archive/2009/03/11/fatal-execution-engine-error-on-x64-framework.aspx

  6. Hello, Just a small 2c
    I found myself in a similar situation. I did the following things:
    – Repair .NET 4 Installation
    – Repair VS2010 Installation
    – Re-apply SP1 for VS2010 (VS2010 crashed during this bit)
    At the end I still had all the errors but then: I tried your RunAs suggestion and it worked (with VS2010, the rest wasn’t affected for me)! So I went and deleted everything in the folder:
    – \Documents and Settings\\Application Data\Microsoft\VisualStudio\10.0
    And now VS2010 works again! :)
    Hope it helps you…


Leave a Reply

No trackbacks yet.