17

There is an ongoing issue where a bad CrowdStrike update has caused systems worldwide to fail to boot Windows and blue screen to WinRE after the failed boot attempts

For machines affected by this, how to resolve the issue and have them boot normally?

JW0914
  • 9,096
peterK88
  • 313

3 Answers3

22

In many corporate networks, end users are not expected/allowed to maintain their own systems and there may be potential wrinkles in corporate environments (compliance requirements, BitLocker, etc.)

  • Please determine what the appropriate steps are for your environment and whether rectification of the issue is self-service, requires you do it yourself, or if it's even necessary.


Fixes:


Microsoft Recovery Tool:

Microsoft has released a tool that builds a bootable recovery system, requiring a USB drive and Admin privledges on a 64bit Windows OS with 8GB of free space:

  • Bitlocker Only:
    Recovery key for each machine required
    1. Get the Recovery Key ID from WinRE:
      1. See advanced repair options → Troubleshoot → Advanced options → Command Prompt
    2. Go to the Microsoft account's Device List
      1. Log in with work/school credentials
      2. Select: Windows PC → View BitLocker Keys
      3. Find matching: Recovery ID → Get: Recovery Key

  • Reported Issues:
    See comments on linked Microsoft Article
    • If WDK is installed, do the recovery system generation from a clean system


Manual Removal:

CrowdStrike's Director of Overwatch provides this workaround for the "faulty channel file":

  1. Boot to Safe Mode or WinRE's Command Prompt from the WinRE blue screen:
    1. Safe Mode:
      See advanced repair options → Troubleshoot → Advanced options → Startup Settings → Restart → Options menu: Safe Mode F4 / 4

    2. WinRE: Command Prompt
      See advanced repair options → Troubleshoot → Advanced options → Command Prompt
      • BitLocker Only:
        1. Get the Recovery Key ID from the screen
        2. Go to the Microsoft account's Device List
          1. Log in with work/school credentials
          2. Select: Windows PC → View BitLocker Keys
          3. Find matching: Recovery ID → Get: Recovery Key
        3. Enter Recovery Key within WinRE → Enter

      • If a machine is stuck on a BSOD and not auto-booted to WinRE:
        1. Reboot machine: Press and hold power button to power off → Release → Power on
        2. Once Windows' bootloader begins loading Windows, repeat Step 1 twice in a row
          • Windows will auto-boot to WinRE upon two failed attempts by the Windows bootloader to load Windows

  2. Delete file matching C-00000291*.sys within: 
    %WinDir%\System32\drivers\CrowdStrike
    1. Safe Mode:
      1. Open an Admin terminal:
        WinKey+R → Open: powershell → Ctrl+Shift+OK
      2. Delete file: 
        Remove-Item -Path "$env:WinDir\System32\drivers\CrowdStrike\C-00000291*.sys" -Force
      3. Reboot: 
        Shutdown /f /r /t 0

    2. WinRE Command Prompt:
      (C: is usually not the OS partition mount point in WinRE)
      1. Obtain mount point of the OS partition: 
        ::# Launch DiskPart:
    DiskPart

        rem # List all volumes [partitions]:
        Lis Vol
        

        rem # Close DiskPart:
        Exit
      2. Delete file: 
        Del /f /q "<OSdriveLetter>:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"
      3. Reboot: Close Command Prompt → Continue to Windows <#>

  3. Normally boot to Windows

  • It's mentioned some machines may self-recover between auto-booting to WinRE since a fix has been pushed out
    • Azure's help pages recommends rebooting VMs up to 15x as a potential workaround

  • It might be possible to boot a WinPE [Windows installer] or WinRE CD/DVD/PXE/USB to do the necessary changes, but this isn't guaranteed since these are business machines
    • Windows installer: Open a terminal once the GUI loads via Shift+F10
JW0914
  • 9,096
Journeyman Geek
  • 133,878
5

While the community wiki answer outlines the basic steps for the workaround/repair, they may be too difficult or time-consuming to perform on a team- or org-wide basis. Our company just added the CrowdStrike BSOD fix to our EasyRE live repair CDs/USBs, and can fix this with only one click (full repair is under 60 seconds, boot time included). The important part is that it doesn't require any credentials (as it's not Windows-based) and will work even if you can't boot into Safe Mode (as some orgs have enabled the CrowdStrike deployment option that blocks booting with CS integration disabled for further security).

If you need to deploy this org-wide, it's possible to boot into the ISO via a PXE network server, too.

Here's a screenshot of the automated repair picking up on this particular bad update and rolling it back:

EasyRE rolling back CrowdStrike update to work around BSOD

Disclosure: worked on this fix and affiliated with the company in question (NeoSmart Technologies), but AFAIK SuperUser accepts such answers as long as the disclosure is in place!

Journeyman Geek
  • 133,878
Mahmoud Al-Qudsi
  • 3,993
0

I've created an automated PowerShell script based on the CrowdStrike's documentation to fix the BSOD issue. It will wait for the machine to be online, check for the relevant files, reboot into safe mode, delete the files, reboot out of safe mode and verify that the files are gone. I hope it helps and would love feedback.

https://github.com/systemfrontier/Automated-CrowdStrike-Falcon-BSOD-Remediation-Tool

As someone else mentioned, this won't work if BitLocker prevents a normal boot into Windows.

Jay Adams
  • 172