{"id":4022,"date":"2024-07-20T19:30:00","date_gmt":"2024-07-20T19:30:00","guid":{"rendered":"https:\/\/www.discovery.org\/tech\/?p=4022"},"modified":"2024-10-15T21:56:16","modified_gmt":"2024-10-15T21:56:16","slug":"what-we-can-learn-from-the-crowdstrike-fiasco","status":"publish","type":"post","link":"https:\/\/www.discovery.org\/tech\/2024\/07\/20\/what-we-can-learn-from-the-crowdstrike-fiasco\/","title":{"rendered":"What We Can Learn From the Crowdstrike Fiasco"},"content":{"rendered":"<div class=\"hailed-articles \" data-frequency=\"https:\/\/mindmatters.ai\/wp-json\/signal\/articles?&id=32874&format=div&refresh=3600\" refresh-needed=\"1\" echo-since=\"380631\"><mark id=\"32874\" class=\"hailed content\" data-frequency=\"https:\/\/mindmatters.ai\/wp-json\/signal\/articles?&id=32874&format=div&refresh=3600\" refresh-needed=\"1\" echo-since=\"380631\" category=\"Computer Security\"><\/mark>\n<p>The <a href=\"https:\/\/www.crowdstrike.com\/platform\/\">Crowdstrike platform<\/a> is a piece of cybersecurity software that has been deployed to millions of computers worldwide.&nbsp; While it supports several different operating systems, it is primarily used on Windows computers.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"embed-frame\"><iframe loading=\"lazy\" title=\"CrowdStrike update causes blue screen error for systems running Microsoft Windows\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/IDvQ3-vqs8Y?start=19&#038;feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n\n\n\n\n<p><strong>What happened and why<\/strong><\/p>\n\n\n\n<p>On July 19, an update was pushed to Windows computers running Crowdstrike that caused them to completely fail.&nbsp; This disrupted many different sectors, but the worst impact was on airlines, banks, and healthcare.&nbsp; The problem was quickly fixed at Crowdstrike, but the damage was done. Each computer required manual intervention to get working again.<\/p>\n\n\n\n<p>This obviously points to problematic internal controls at Crowdstrike. First, any update should have been tested internally before deployment. Second, deployments at this scale should be done in a rolling fashion with feedback mechanisms to prevent system-wide catastrophes such as this one.&nbsp;<\/p>\n\n\n\n<p>However, I want to take a look more broadly at this problem and what it might be teaching us about how we look at technological solutions generally.<\/p>\n\n\n\n<p><strong>The risk of removing all risk<\/strong><\/p>\n\n\n\n<p>In modern society, we oftentimes try to remove risk altogether.&nbsp; We buy insurance for everything under the sun, we save for retirement, and we have lots of rules to make sure nobody gets hurt.&nbsp; The question, though, is whether we are actually removing risk, or just moving it somewhere else.&nbsp; <a href=\"https:\/\/www.fooledbyrandomness.com\/\">Nicholas Nassim Taleb<\/a>, investor and author of a number of books on business, finance, and investing, has been warning the world for years about the fact that a lot of things that we try to do to remove risk actually makes the problem <em>worse<\/em> but also less visible.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"alignright size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"789\" height=\"1200\" src=\"https:\/\/mindmatters.ai\/wp-content\/uploads\/sites\/2\/2024\/07\/Black-Swan.jpg\" alt=\"\" class=\"wp-image-32877\" style=\"width:392px;height:auto\" srcset=\"https:\/\/mindmatters.ai\/wp-content\/uploads\/sites\/2\/2024\/07\/Black-Swan.jpg 789w, https:\/\/mindmatters.ai\/wp-content\/uploads\/sites\/2\/2024\/07\/Black-Swan-649x987.jpg 649w, https:\/\/mindmatters.ai\/wp-content\/uploads\/sites\/2\/2024\/07\/Black-Swan-768x1168.jpg 768w\" sizes=\"auto, (max-width: 789px) 100vw, 789px\" \/><\/figure>\n<\/div>\n\n\n<p>When we think about risk, we normally think about \u201cnormal\u201d distributions, where risk is spread out pretty evenly across the spectrum.&nbsp; The more catastrophic the event is, the less likely it is to occur.&nbsp; Our expectation is that, if we remove risk in the ordinary things, we are <em>also<\/em> removing risk at the extremes.&nbsp; However, the fact of the matter is that the opposite is true.&nbsp; When we remove risk in the ordinary things, we are often <em>adding<\/em> risk at the extreme ends.&nbsp; This creates distributions that have what are known as \u201cfat tails\u201d\u2014an increased probability that extreme events will happen.<\/p>\n\n\n\n<p>If everybody buys insurance, what happens when the insurance company goes broke? &nbsp;If everybody \u201cplays it safe\u201d in ordinary life, what happens when extreme acts of heroism are required and nobody is up to the task?&nbsp; Oftentimes, what we gain in de-risking the short and medium term shows up as fat tails causing extreme failures to become more likely.&nbsp; We can convince ourselves that they won\u2019t happen because they don\u2019t happen <em>often<\/em>, and then when they do occur we can just act like it is one of those things that nobody can control.&nbsp; <\/p>\n\n\n\n<p><strong>The tail risk<\/strong><\/p>\n\n\n\n<p>Our society is laser-focused on near-term, first-order effects of actions, and almost entirely blind to their larger-scale and second-order effects. In the case of Crowdstrike, people are de-risking their day-to-day security operations by giving it over to a third party company to do it for them.&nbsp; Additionally, auditors take this to be a positive thing, oftentimes bypassing large swaths of questions just by knowing that a company puts its computers under the control of Crowdstrike.&nbsp; What they are missing is the <em>tail risk<\/em> that is added by doing this.<\/p>\n\n\n\n<p>In this case, it was a faulty update.&nbsp; But there are other tail risks to consider.&nbsp;&nbsp; What happens if a bad actor gets a prominent place at Crowdstrike (or a similar firm)?&nbsp; What happens if someone figures out a vulnerability in Crowdstrike that causes computers which run it to be less safe?&nbsp;<\/p>\n\n\n\n<p>In short, many managers have spent their time only considering near-term risks.&nbsp; It is time for IT managers to also consider the \u201ctail risks\u201d of their decisions.&nbsp; We need to make tail risks an ordinary part of our vocabulary and considerations. Maybe the amount of effort saved in the short run makes disasters like this worthwhile and then perhaps we should just plan for them.&nbsp; But this should be a deliberate decision, made with full knowledge of how near-term decisions affect long-term tail risks.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Saving effort in the short term by running these risks is a decision that should be made with full knowledge of how near-term decisions affect long-term risks.<\/p>\n","protected":false},"author":370,"featured_media":4023,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[93],"tags":[138,139],"coauthors":[143],"class_list":["post-4022","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cybersecurity","tag-crowdstrike","tag-cybersecurity"],"acf":[],"author_names":["Jonathan Bartlett"],"_links":{"self":[{"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/posts\/4022","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/users\/370"}],"replies":[{"embeddable":true,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/comments?post=4022"}],"version-history":[{"count":0,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/posts\/4022\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/media\/4023"}],"wp:attachment":[{"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/media?parent=4022"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/categories?post=4022"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/tags?post=4022"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/coauthors?post=4022"}],"wp:action-assign-author":[{"href":"https:\/\/www.discovery.org\/tech\/wp-json\/wp\/v2\/post\/4022"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}