CrowdStrike Outage Explained – What Happened & What Can be Done to Prevent Future Issues?

2 August 2024

A routine software update should enhance security, not plunge the world into chaos. However, on July 19, 2024, CrowdStrike’s Falcon Sensor update did just that, causing millions of Windows computers to crash. This unexpected glitch, which has been described as one of the largest IT outages in history, caused significant disruptions worldwide, affecting healthcare, banking, airlines, and more.

The ripple effect of this mishap highlighted the vulnerabilities in our interconnected world and raised pressing questions about the reliability and robustness of cybersecurity measures. Below, we look at CrowdSrike’s outage, including what happened, why it occurred, who was affected, and what steps can be taken to prevent such future issues.

The Incident: What Happened?

On July 19, 2024, a routine software update by CrowdStrike, a leading cybersecurity firm, went disastrously wrong. The update, part of the Falcon Sensor platform aimed at enhancing the detection of novel threats, was pushed to millions of computers globally.

This update, however, contained a critical flaw that caused Windows operating systems to crash, resulting in the notorious “Blue Screen of Death.” The issue stemmed from a failure in CrowdStrike’s quality control mechanism, specifically a bug in the Content Validator, which allowed the faulty update to be released. According to CrowdStrike’s initial report on the incident, the problematic update was published at 04:09 UTC and rolled back at 05:27 UTC. Still, by then, millions of Windows devices had already been affected.

The Cause: Why Did It Happen?

The root cause of the outage was a bug in the Content Validator, a key component of CrowdStrike’s quality control mechanism. This bug allowed the faulty update to pass validation checks and be released to the public. The update contained a rapid response content configuration meant to gather telemetry on potential threat techniques.

Unfortunately, the configuration led to an ‘out-of-bounds memory read,’ causing system crashes on Windows devices that received the update. CrowdStrike’s routine tests failed to catch this error due to the flaw in their validation system, leading to widespread havoc.

Impact: Who Was Affected?

According to analysis, the CrowdStrike outage sent shockwaves across various industries, causing significant disruptions and financial losses. The fallout was widespread, affecting key sectors and countless individuals and businesses. Below is a detailed look at who was impacted by this event:

Healthcare sector: Hospitals and clinics experienced severe disruptions, with systems crashing and patient records becoming inaccessible. Many appointments and procedures were delayed or cancelled, leading to an estimated loss of $1.94 billion.
Banking and finance: Financial institutions were hit hard, with an estimated $1.15 billion in losses. The outage affected transaction processing, online banking services, and ATM operations. Customers faced difficulties accessing their accounts and completing financial transactions, leading to widespread frustration and potential financial losses.
Airlines: Major airlines faced massive disruptions. Thousands of flights were cancelled or delayed, causing chaos for travellers worldwide. The collective losses for the airline industry were estimated at $860 million.
Fortune 500 Companies: Numerous Fortune 500 companies relying on CrowdStrike’s cybersecurity software experienced significant operational disruptions. The outage affected business operations, leading to an estimated $5.4 billion in direct losses.

Response and Recovery Efforts

In the wake of the outage, CrowdStrike moved quickly to address the issue. The company reverted the defective update within an hour and a half of its release.

However, the recovery process proved to be labour-intensive, requiring manual intervention to delete the faulty file from the affected systems. This process was complicated because as many as 8.5 million devices needed to be reset.

CrowdStrike is enhancing its quality control processes and implementing additional safeguards to prevent future incidents. The company has pledged to improve its testing and validation systems to catch potential problems before updates are released. This includes developing new checks and balances within the Content Validator to ensure that only thoroughly vetted updates are deployed.

Moreover, CrowdStrike aims to provide customers with more granular control over when updates are installed, reducing the risk of widespread disruption. The company is also considering a staggered release strategy, where updates are rolled out gradually rather than all at once, allowing for more effective monitoring and rapid response to any issues.

Preventive Measures: What Can Be Done to Prevent Future Issues?

It’s crucial to implement robust strategies to prevent similar incidents in the future. Here are key measures that can help safeguard against such widespread disruptions:

Implementing stricter quality control measures in the software development and update process to catch potential issues before they reach end-users
Deploying updates in phases rather than all at once to minimise the impact of any unforeseen issues
Investing in advanced monitoring tools to detect anomalies in real-time
Conducting regular audits and security assessments of systems and processes to identify vulnerabilities and areas for improvement
Allowing users more control over when and how updates are installed to reduce the risk of disruptions
Holding vendors accountable for the reliability and security of their products to drive better practices

Partner with ICT Solutions

The CrowdStrike outage is a vital reminder of the fragility of our interconnected digital ecosystems and the far-reaching consequences of cybersecurity failures. As businesses continue to navigate these challenges, it is essential to learn from such events and fortify defences to prevent future disruptions.

If you’re looking to strengthen your organisation’s defences against the ever-evolving cyber-attack threats, partner with ICT Solutions. We provide comprehensive IT support and cybersecurity services in Liverpool and across the UK, ensuring your business is protected on all fronts.

News Security