PSA: Clearing global debugger properties

Or: Why is my UnhandledExceptionFilter not firing?

# TLDR

So your UnhandledExceptionFilter is just not being called? Maybe you program is being run under a debugger without you even knowing. Check the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ registry key and make sure it does not specify any special treatment for your .exe.

# The whole story

Working at sentry, one of my responsibilities is to take care of the native SDK. And yesterday I was in a really strange situation. For unknown reasons, all my integration tests using the crashpad backend were failing. Admittedly, those were the only tests I was running since I didn’t touch any other code. But it was strange and it didn’t make any sense. I tried the master branch, and it was the same, even though everything was working as expected on CI.

A quick internet search didn’t give me any good ideas at first. But I noticed that log output was missing from our FirstChanceHandler, which I added on Windows in addition to Linux. Its a piece of code that can still run in-process at the time of a crash to flush any internal state. It is based around the SetUnhandledExceptionFilter mechanism, which it seems is the Windows way of handling native crashes in the process.

This refined my search a bit, and lead me to a SO post highlighting a paragraph from the official documentation.

[…] if an exception occurs in a process that is not being debugged […] the exception makes it to the unhandled exception filter […]

So having an unhandled exception filter, and a debugger is mutually exclusive. But I was not using a debugger. At this point I was also testing all the other sentry backends, none of which was working.

And then I remembered, I was running the Application Verifier, which does not really do anything by itself, but it apparently through some kind of global magic enables additional checks when running programs in the Visual Studio Debugger. But I was not doing that in this case, how come my exceptions still didn’t reach the handler?

Another round of searching got me to a page that mentioned the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ which can be used to attach a debugger automatically right when spawning a process, and its documentation mentions something along those lines as well.

And sure enough, there was an entry for sentry_example.exe. Just deleting that registry key solved the whole problem. All my integration tests returned to running correctly. So it seems all of this is a bit of magic on Windows, and sadly not really well documented. But now I’m eager to learn more about how all this works.