6

I have a huge code-base I am unfamiliar with, and the program terminates abnormally because a thread somewhere is calling __fastfail. This is based on the message, which ends with

... Fatal program exit requested.

The call stack has no symbols because it's inside C++ 2015 runtime (ucrtbase.dll). The call appears to be made on a thread other than my main thread. This mysterious thread only starts just before the issue happens so I can't catch the act of it starting in the debugger - I don't know what's starting it, and what causes the whole process in the first place.

I have SEH in my main() using __try/__catch, so any unhandled exceptions should be getting trapped there. Instead I am guessing something somewhere bubbles up to the runtime and results in __fastfail.

I tried peppering all my threads with SEH just like main(), tried hooking abort(), exit() and terminate(), and can't find the problem. How do I debug this, any tips?

Thomas Weller
  • 55,411
  • 20
  • 125
  • 222
Valeriy Novytskyy
  • 775
  • 1
  • 7
  • 18

2 Answers2

3

WinDbg

I would say this is a nice task for WinDbg. WinDbg is part of the Debugging Tools for Windows and it's free. Install both versions, x64 and x86 so that you can debug any kind of application.

  1. Start WinDbg, use the correct bitness
  2. Run your executable under WinDbg (File/Open executable). It will stop at the initial breakpoint.
  3. Set up the symbols, at least .symfix c:\debug\symbols and .reload. As mentioned by @James McNellis, the symbols are available and this will download them when needed.
  4. Continue running the application with g
  5. Reproduce the issue
  6. When WinDbg stops,
    • create a crash dump with .dump /ma c:\debug\mydump.dmp so you can analyze it later
    • get information about the exception with .exr -1
    • switch to the thread that caused the exception with ~#s
    • look at the callstack with k

Learning WinDbg is a hard task, since most things are done via cryptic commands and not via UI, but it can do almost everything.

For more specific problems when you have some more clues, ask additional questions using the tag.

Visual Studio

Also Visual Studio can download symbols (PDB files; callstack information) from Microsoft Servers.

  1. Go to Tools | Options ... in the main menu
  2. Select Debugging | Symbols from the options menu
  3. Enter a directory name if you want the file to be stored in a specific directory.

Here it is how it looks like in Visual Studio 2015 Community Edition:

Symbols in Visual Studio 2015 Community Edition

Community
  • 1
  • 1
Thomas Weller
  • 55,411
  • 20
  • 125
  • 222
  • It shouldn't take this much work. When the `__fastfail` interrupt occurs, the system will halt execution of all threads in the process and then notify the debugger if one is attached. As long as one has symbols loaded for ucrtbase.dll, it should be fairly straightforward to walk up the stack. Either the Visual Studio debugger or windbg will work just fine here. – James McNellis Sep 03 '15 at 22:01
1

Hans and James are right, getting a readable call stack was essential in solving this case.

The codebase installed a pure call handler which was performing a fast fail. Pure handlers fall through SEH. One of the 40-something threads in the program had a pure call error because some other thread partially cleaned up program state which the problem thread then tried to access. Once I had the symbols Visual Studio broke into C++ runtime library at a place where it was calling the purecall handler, and I traced it back into the program where they installed their own.

The way they did fast fail was convoluted and resolved into some RtlXXX function that apparently could not be trapped with signal(SIGABRT) because I did try that before.

Thanks again for the help everyone!

Valeriy Novytskyy
  • 775
  • 1
  • 7
  • 18