The world’s biggest IT catastrophe forced airlines, hospitals, 911 call centers, and banks (among other businesses) to their knees last month. The costs from this massive outage are estimated to reach several billion dollars. We’re all going to be talking about it for several more months, and we’ll likely be seeing some high-level litigation. This blunder could potentially change the way we all think about IT and security, as well as create new laws.

The craziest thing (so far) about the entire CrowdStrike outage is how incredibly fast it crashed the entire world. I think this is the most important lesson for business owners to learn from.

It Was Practically a Doomsday Button

The massive CrowdStrike Outage happened early Friday morning, on July 19th. CrowdStrike is a cybersecurity firm that provides enterprise-level cybersecurity software. CrowdStrike has been around since 2011, and has had a pretty big impact on the cybersecurity industry. According to the company, they have around 29,000 business customers, and over 500 of them are on the Fortune 1000 list. 

The issue occurred when a faulty software update was released and pushed to 8.5 million computers. This update caused PCs to crash, and the fix required some manual work and wasn’t discovered until later on in the day. The way that the affected PCs crashed didn’t give a lot of clues as to what was causing the outage, so for a short time on Friday, it was just being referred to as the Windows Blue Screen of Death issue, and many attributed the problem to Microsoft, when it was really this CloudStrike update.

It was a chaotic situation as the world struggled to get back to normalcy as quickly as possible.

We’ve already seen plenty of cases where popular enterprise software is targeted by cybercriminals over the last few years. In fact, some of the software that IT departments and IT companies use to remotely manage hundreds of PCs at once has been targeted. The results of these types of attacks are disastrous, so it’s extremely important that businesses work with IT firms that take cybersecurity very seriously. Cybercriminals know that if they can exploit these tools to gain access to hundreds, thousands, or even millions of endpoints—it’s basically the motherlode.

The CloudStrike Outage wasn’t a cyberthreat or an attack. It was a mistake.

The outage was severe, but it could have been worse if it was caused intentionally by cybercriminals. The point I’m making here isn’t to play a game of “what if,” but to convey just how serious all of this is when it comes to your technology. This stuff happens fast. The next big thing could be worse, but the point is that if you don’t have contingencies, your business will have to suffer through it.

It Happened In Minutes

As mentioned, the flawed update that caused the world’s largest IT kerfuffle was only live and available for about 78 minutes. The update was released just a few minutes after midnight (Eastern time) on Friday morning and then a little over an hour later it was reverted when CrowdStrike realized what was going on. That’s all the time needed for 8.5 million devices to be affected worldwide. It happened while most of us were asleep.

I can’t stress just how fragile all of this is. Yes, shame on CrowdStrike for not being more thorough in their testing, but the entire business tech world is built on a foundation of millions of tiny, shifting parts. Think of it like a big Jenga tower, where each block is a different technology or code project. 

It reminds me of a 2020 comic by XKCD:

How to Prepare for the Unexpected IT Disaster

We can all learn a lot from this outage.

1. Carefully vet your vendors. CrowdStrike didn’t have any major pockmarks in its image, especially not compared to other massive cybersecurity vendors, but there are definitely vendors out there that aren’t as diligent or responsive. This outage could have been a whole lot worse.
2. Test updates before deploying them to a live environment. IT is complex. Your network and all of the things running within are going to be a little different than anyone else’s network. You never know when an update that works for everyone else might cause havoc for you. Even so, you need to keep up on your security updates, because not doing so will open you up for even bigger risks.
3. Have a good backup solution, and test it regularly. If you don’t test your data backup solution, you can’t be certain it works. It’s a pain, but knowing you can run and survive off of your backup temporarily will make a real problem that much easier to get through.
4. Have an emergency/disaster plan. Your plan should cover the unexpected, and assume the worst. Have important phone numbers documented, establish responsibilities and lines of communication, and make sure you can communicate to your staff and customers even if your technology is down. Build your disaster plan as if it were for a massive power outage.
5. Have a managed IT provider you can trust. The real heroes of the CrowdStrike outage are the IT professionals who signed in at 1am Friday morning and tirelessly worked (in some cases, through the weekend) to put the world back together. This outage is living proof that a software bug can cause a massive problem, and having the support in your back pocket to help your business through it can make all the difference.
6. Understand the value of your IT. I think everyone affected by the CrowdStrike outage are coming to terms with just how important their technology truly is, and how expensive downtime can be. It’s worth it to be proactive.
7. Look into business insurance and cybersecurity insurance. It’s a good time to review your business insurance and see what it covers. It’s estimated that insured losses from the CrowdStrike outage could be up to $1.5 billion dollars.

Get a Handle on Your IT So Your Business Isn’t Left in the Dark

Technology is only going to get more complicated. That’s why Maryland businesses trust Dresner Group for all of their IT. We are like having a dedicated in-house IT department that truly understands your business and prioritizes your technology to meet your goals. To get started, give us a call at (410) 531-6727.