Microsoft And CrowdStrike outage

nRiTeCh said:
It is due to the enabling of windows updates users are getting bsods both at home and corporate. It has nothing to do with crowdstrike in the first place.
Click to expand...
It is caused by CrowdStrike update. MS updates don't go out worldwide at the same time as the organizations (clients) have granular control over when the updates get pushed.

Meanwhile, someone in reddit posted CrowdStrike is not worth 83B dollars and is a sitting duck if something is to happen as they are having a large client base all over the world. 4 hrs later, largest outage in IT history happens. You can't make this shit up!

https://www.reddit.com/r/wallstreetbets/comments/1e6ms9z
 
n1r0 said:
Has any malware caused chaos on this scale before?
Click to expand...
WannaCry ransomware with far larger scale.

Also this was NOT a malware attack. Neither was caused by Microsoft.
Crowdstrike has an endpoint protection software which is used by many companies including Government organizations. They pushed an update of their software which included a faulty kernel mode driver which failed to load in Windows causing a bugcheck (BSOD).
It was sheer incompetence by them as this could have easily been prevented if the patch was tested even on a single windows machine. They pushed the update without any testing whatsoever.
 
enthusiast29 said:
WannaCry ransomware with far larger scale.

Also this was NOT a malware attack. Neither was caused by Microsoft.
Crowdstrike has an endpoint protection software which is used by many companies including Government organizations. They pushed an update of their software which included a faulty kernel mode driver which failed to load in Windows causing a bugcheck (BSOD).
It was sheer incompetence by them as this could have easily been prevented if the patch was tested even on a single windows machine. They pushed the update without any testing whatsoever.
Click to expand...
Crowd strike has posted updated info on the issue. It was "just" a configuration update or equivalent to definition update, backed by the fact that you could just delete the C-00000291-*.sys file in Safe Mode and get the system working again.

They push it through several times a day for every novel attack and in that case it can be assumed that testing is automated. Probably will have to add more safeguards for corrupt updates.

www.crowdstrike.com

Technical Details on July 19, 2024 Outage | CrowdStrike

Learn more about the July 19, 2024 CrowdStrike outage and the technical details related to it.
www.crowdstrike.com www.crowdstrike.com
 
t3chg33k said:
Crowd strike has posted updated info on the issue. It was "just" a configuration update or equivalent to definition update, backed by the fact that you could just delete the C-00000291-*.sys file in Safe Mode and get the system working again.
Click to expand...
That sys file is a kernel mode driver. Those are breaking changes that have to go through an effective regression testing suite. They simply didn't and this is the reason why chaos happened.
Any bad kernel mode driver will trigger a bugcheck (BSOD) on boot.

They're trying to make it sound not so serious when in-fact this was a colossal incompetence incident.
 
www.tomsguide.com

Massive Microsoft and CrowdStrike outage live updates — airlines, banks and more taken down

Windows machines around the world are experiencing a significant IT issue
www.tomsguide.com www.tomsguide.com
> Meanwhile, Microsoft, one of the largest affected companies, seems to have suffered a separate outage that mainly affects Microsoft 365 apps and services due to a configuration change in its backend Azure settings. The company says it has now fixed these.

msft 365 issue got overshadowed by crowdstrike mega issue.
 
enthusiast29 said:
That sys file is a kernel mode driver. Those are breaking changes that have to go through an effective regression testing suite. They simply didn't and this is the reason why chaos happened.
Any bad kernel mode driver will trigger a bugcheck (BSOD) on boot.

They're trying to make it sound not so serious when in-fact this was a colossal incompetence incident.
Click to expand...
1721494475831.png

Crowdstrike specifically clarified that in the article I linked previously. It is a misconception being spread by those going by the file extension.
 
t3chg33k said:
View attachment 202042
Crowdstrike specifically clarified that in the article I linked previously. It is a misconception being spread by those going by the file extension.
Click to expand...
Yeah even the directory name "drivers" must be mislabeled. Now this specific file could be a loader as they might say and that loader could be bad but it doesn't matter because there's no other way to trigger a bugcheck in windows. It's always either a faulty driver or failing/unstable hardware.

Don't trust every word of a corporation especially the one which caused the damage. They have a conflict of interest here to protect their reputation.
 
enthusiast29 said:
That sys file is a kernel mode driver. Those are breaking changes that have to go through an effective regression testing suite. They simply didn't and this is the reason why chaos happened.
Any bad kernel mode driver will trigger a bugcheck (BSOD) on boot.

They're trying to make it sound not so serious when in-fact this was a colossal incompetence incident.
Click to expand...
The worse and the unprofessional part is they seem to have forgotten to conduct a version/patch test in their labs before rolling out to the public. Sheer negligence by a security based co. Wonder why was there a hurry to release without proper testing.

1721502429243.jpeg
 
Last edited:
CrowdStrike’s faulty update caused a worldwide tech disaster that affected 8.5 million Windows devices on Friday, according to Microsoft. Microsoft says that’s “less than one percent of all Windows machines,” but it was enough to create problems for retailers, banks, airlines, and many other industries, as well as everyone who relies on them.
Click to expand...

www.theverge.com

CrowdStrike’s faulty update crashed 8.5 million Windows devices, says Microsoft

Here are the details about what went wrong on Friday.
www.theverge.com www.theverge.com
enthusiast29 said:
Don't trust every word of a corporation especially the one which caused the damage. They have a conflict of interest here to protect their reputation.
Click to expand...
They are under the radar of every single customer company and govt. org right now, I highly doubt they’re that stupid.

But yeah, I do think it might be an underlying bug in their kernel components as many are presuming that went unnoticed until a bad payload exposed it in a glorious fashion.

Even the “null file” theory seems to not track as folks reported seeing differing file contents.

threadreaderapp.com

Thread by @patrickwardle on Thread Reader App

@patrickwardle: I don't do Windows but here are some (initial) details about why the CrowdStrike's CSAgent.sys crashed Faulting inst: mov r9d, [r8] R8: unmapped address ...taken from an array of pointers (held in RA...…
threadreaderapp.com threadreaderapp.com
 
Last edited:
enthusiast29 said:
Yeah even the directory name "drivers" must be mislabeled. Now this specific file could be a loader as they might say and that loader could be bad but it doesn't matter because there's no other way to trigger a bugcheck in windows. It's always either a faulty driver or failing/unstable hardware.

Don't trust every word of a corporation especially the one which caused the damage. They have a conflict of interest here to protect their reputation.
Click to expand...
In case of security software, obscurity is part of the security and hence you will have to take the maker's word for it as it is not possible to determine what each component does.

There is no question that Crowdstrike uses kernel-mode drivers as Windows provides that as a feature.
learn.microsoft.com

Overview of Early Launch AntiMalware - Windows drivers

Top-level page for documentation that provides information about developing Early Launch Antimalware (ELAM) drivers for Windows operating systems.
learn.microsoft.com

However, it is not a driver update that caused this issue. The configuration file determines how the Falcon sensor operates and in this case it contained a logic error which caused a memory allocation error.

The kernel-mode driver itself was flawed in the sense that it didn't validate the memory allocation prior to execution and ended up causing the BSOD.

The only point is that a relatively trivial configuration update caused the entire issue and not a core software update, although the flaw did exist in the software to enable this to occur.
 
Autoupdates in Windows are horrible. I remember putting everything at Pause/disabled and yet the Window was downloading the updates and always waiting to get installed with next restart. No one knows what Windows server is putting on your computer in each forced update.
 
rocking said:
Autoupdates in Windows are horrible. I remember putting everything at Pause/disabled and yet the Window was downloading the updates and always waiting to get installed with next restart. No one knows what Windows server is putting on your computer in each forced update.
Click to expand...
That is why corporates have a policy to delay major updates by 1-2 years. However, the question remains as to what do you do with configuration updates like these that are delivered multiple times a day and are essential to thwarting 0-day exploits.
 
It seems that the CEO will now have the unenviable record of large-scale outages following him wherever he goes :blackalien:

MSN: This is the 2nd time CrowdStrike CEO George Kurtz has been at the center of a global tech failure

www.zdnet.com

What caused the great CrowdStrike-Windows meltdown of 2024? History has the answer

When a trusted software provider delivers an update that causes PCs to immediately stop working across the world, chaos ensues. Last week's incident wasn't the first such event. Here's how to make sure it doesn't happen again.
www.zdnet.com www.zdnet.com
 
Back
Top