Global PC outage: Handwritten boarding cards, failed payments & a messy software

The widespread impact was on businesses that are Microsoft’s Windows PC and services customers, meaning banks and payment gateways couldn’t process transactions

Put everything on the cloud, they said. Whilst a robust idea allowing seamless access to data, software and services, it also amplifies the magnitude of any failure, bringing everything down like a pack of cards. That’s something the world experienced this Friday, when Microsoft’s services suffered an outage, which has since been resolved in phases, with Azure services back online first, followed by Windows and Microsoft 365.

Microsoft said it was investigating issues with its cloud services in the Central US region. (Reuters file photo)

The widespread impact was on businesses that are Microsoft’s Windows PC and services customers, meaning banks and payment gateways couldn’t process transactions, airlines in many countries had to ground flights, TV channels went off air, whilst employees in many organisations globally couldn’t log into Microsoft 365 cloud services or access third-party productivity software that uses Azure services or, worse still, were unable to log in to their Windows computing devices, greeted instead by the dreaded BSOD, or blue screen of death.

“A configuration change in a portion of our Azure backend workloads caused interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services dependent on these connections,” was an explanation from Microsoft regarding the outage. They didn’t name the offender in a statement, but the cause of this global pandemonium sits at the doorstep of US cybersecurity company CrowdStrike.

Why did the world’s computers crash? Microsoft isn’t to be blamed for this outage, though it is Microsoft’s services and their consumers who have borne the brunt. A security update by CrowdStrike, for its Falcon endpoint security suite, caused the cloud-connected and local systems to crash. In a notification to customers in the early hours of the outage, CrowdStrike confirmed that BSOD errors and bug-check issues were indeed related to the Falcon Sensor software.

“CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes,” the notification read. They suggest the faulty update has been reverted, but computing devices and systems knocked offline didn’t register those recalibrations.

“Whilst the issue is associated with Windows systems, it does not appear to be an issue with Microsoft Windows, but rather, security software installed on millions of Windows computers worldwide. Because this is security software, it requires a higher level of privileges to the underlying operating system, so a bad or faulty security update can result in a catastrophic impact,” Satnam Narang, Senior Staff Research Engineer at cybersecurity company Tenable, tells HT.

To understand CrowdStrike’s role in Microsoft’s services, we don’t need to travel too far back in time. In May, the security company announced CrowdStrike Falcon for Defender, to be deployed by organisations alongside the Microsoft Defender product (this is also part of Windows 11) to enhance security threat identification, reduce chances of a threat bypass and manage updates for large teams.

“Security professionals saddled with deploying Microsoft Defender through E3 and E5 licensing have long asked us for help. Falcon for Defender fills critical security gaps at a disruptive price point,” Michael Sentonas, President at CrowdStrike, had said at the time. On a broader enterprise level, globally more than 3,500 companies use CrowdStrike’s software as an endpoint protection tool on their network devices, including computing devices given to employees.

Can you fix BSOD or reboot loop issue on your PC? Whilst it will be up to Microsoft and specific affected platforms to ensure services knocked offline are back in the green, CrowdStrike has suggested a few workaround steps including booting Windows in Safe Mode or Recovery Environment, navigating to the “CrowdStrike” folder in the local drive and deleting a file with the name “C-00000291.sys”. However, the limitation here is, enterprises usually lock user access to system folders on the PCs and laptops they assign to employees.

At each organisation affected, chances are there will be the requirement for manual, or in some cases batch, updates to remove the offending file. That could take a significant amount of time. It is impossible to remotely update a computing device or a machine via the internet if it cannot boot up and be connected to the internet.

From what is understood thus far, Windows PCs and laptops which are set up for personal use haven’t seen any impact of the CrowdStrike configuration error. HT can confirm that Windows 11’s most recent machine, the Asus Vivobook S 15 which is labelled a Copilot+ PC, doesn’t have any CrowdStrike software installed alongside Windows Defender.

How bad was the impact? Airline boarding passes written by hand and electronic flight status terminals replaced by a whiteboard and a human being with a marker pen. TV channels off air temporarily. Banks unable to process transactions. Globally, businesses and organisations struggled, including Visa Inc. which struggled to get payments through on their network. Throughout the day, online platform DownDetector indicated spikes in outage monitoring for various enterprises, businesses and services globally, including parts of Amazon Web Services.

“We can confirm reports of connectivity issues and reboots of Windows Instances, Windows Workspaces and Appstream Applications related to a recent update to the Crowdstrike agent (csagent.sys), which is resulting in a stop error (BSOD) within the Windows operating system,” AWS confirmed in an update, once their services resumed normalcy. Google Cloud didn’t report any issues for their enterprise customers.

IndiGo airlines issued a notification suggesting their systems across the network were impacted. “Our digital team are coordinating closely with Microsoft Azure to resolve these issues swiftly,” it read. Other airlines including Vistara, SpiceJet and Akasa Air reported struggles, whilst the Delhi Airport also issued an official notification that services were impacted. Globally too, airports in the US and Europe were affected as airlines had their systems knocked offline.

The Paris Olympics committee confirmed their systems were affected too, but contingency plans were in place.

It may still be a while before the financial cost of an incorrect security software update that messed up computers globally is totalled. Last year, VPN service provider AtlasVPN had simulated the cost of one day of a global internet shutdown. That’d be around $43 billion. Whilst AtlasVPN shuttered services this summer and moved users to NordVPN, their estimates give us a fair idea of the still-counting cost of the July 19 PC outage.