Navigating the CrowdStrike Falcon Incident: Recovery Guide and Solutions

By /

Incident Overview

KEPPTIC has observed a critical incident affecting Virtual Machines (VMs) running Windows Client and Windows Server with the CrowdStrike Falcon agent installed. This incident, identified by CrowdStrike, has caused crashes and blue screen errors (BSOD) starting from approximately 19:00 UTC on July 18, 2024.

CrowdStrike and Microsoft Response

CrowdStrike promptly addressed the issue by reverting the problematic channel file associated with the Falcon Sensor update. Microsoft has also provided specific guidance for Azure VMs experiencing these issues, including steps for utilizing the Serial Console for recovery.

Incident Details and Impact

  • Affected Systems: Windows hosts with the Falcon Sensor update channel file timestamped at 0409 UTC or later.
  • Unaffected Systems: Windows hosts offline during the affected period or brought online after 0527 UTC, and Mac/Linux-based hosts.

Recommended Actions for Affected Customers

Based on our experience and guidance from CrowdStrike and Microsoft, KEPPTIC recommends the following actions for affected customers:

  1. Reboot Operations:
    • It may be necessary to attempt multiple restart operations on affected VMs. Customers have reported success after up to 15 reboots in some cases.
  2. Restoration from Backup:
    • Restore affected VMs from backups taken before 19:00 UTC on July 18, 2024, if feasible.
  3. Disk Repair and File Deletion:
    • Option 1: Disk Repair via New Virtual Server
      • Detach the operating system disk volume from the impacted VM.
      • Create a snapshot or backup of the disk volume as a precaution.
      • Attach or mount the volume to a new virtual server.
      • Navigate to %WINDIR%\System32\drivers\CrowdStrike/ on the attached volume.
      • Locate and delete files matching C00000291*.sys.
      • Detach the volume from the new virtual server.
      • Reattach the fixed volume to the impacted VM.

Azure VM Recovery Steps via Serial Console

For Azure VMs affected by the Falcon Sensor update, follow these steps using the Serial Console:

  1. Access Serial Console:
    • Login to the Azure portal, navigate to Virtual Machines, and select the affected VM.
    • Click on “Connect” > “More ways to Connect” > “Serial Console.”
  2. Initiate Safe Boot:
    • Once Serial Console has loaded, type cmd and press Enter.
    • Type ch -si 1 and press any key (space bar). Enter Administrator credentials if prompted.
    • Execute the following commands:
      • bcdedit /set {current} safeboot minimal
      • bcdedit /set {current} safeboot network
    • Restart the VM.
  3. Confirm Boot State:
    • After restarting, confirm the boot state by running
      • wmic COMPUTERSYSTEM GET BootupState

Leveraging KEPPTIC Managed IT Services

KEPPTIC is committed to assisting clients in navigating and mitigating IT incidents like the Falcon Sensor update issue. Our Managed IT Services include:

  • Proactive Monitoring: Continuous monitoring to detect and respond to issues promptly.
  • Patch Management: Timely application of updates and patches to maintain system security and stability.
  • Incident Response: Swift response capabilities with tailored recovery strategies to minimize downtime and impact.

Conclusion

With CrowdStrike’s corrective actions and Microsoft’s guidance, KEPPTIC stands ready to support our clients in resolving this incident effectively. For ongoing support, guidance, or to learn more about our Managed IT Services, please visit KEPPTIC’s website or contact us directly.

Stay resilient with KEPPTIC—your trusted partner in Managed IT Services.