Yesterday’s Widespread Internet Outage, for non-geeks

See the end of this post for updates.

Yesterday, Friday October 21st, 2016, Internet users in the US (initially the East Coast) experienced issues accessing many popular websites, including Twitter, Etsy, Spotify, CNN, Amazon and others. This also affected numerous companies and services used heavily in the technology industry for everything from storing and sharing source code to monitoring the health of services and alerting on-call technical staff.

The cause of the problems was an outage of Dyn, one of the leading providers of DNS services, the system which translates human-readable domain names (like “blog.jasonantman.com”) to the numeric IP addresses actually used by computers to communicate. Aside from the immediate take-away - that so many high-traffic websites are relying on a single DNS provider, and therefore a single point of failure - there are a number of important points about what happened:

1) This was not an accidental outage, it was an intentional attack by a malicious party; specifically a distributed denial of service (DDoS) attack, where an attacker takes control of tens or hundreds of thousands, or millions, of computers, and uses them to bombard the target with data. Assuming enough data is generated, as was the case yesterday, this will overwhelm the target, preventing it from fulfilling the requests of legitimate users. This was a very visible, intentional attack on critical Internet infrastructure.

2) Some of the leading experts on computer security, people who consult for Fortune-500 companies and have substantial access to non-public information, believe that this attack was one of a series carried out by an advanced, well-funded nation state attacker (likely either China or Russia, with evidence pointing to the former). Furthermore, evidence points to this attack not being the end-goal, but rather part of an escalating test to determine the point of failure of critical Internet infrastructure. The cyber equivalent of test-firing missiles and flying over enemy territory to map defenses, in preparation for an attack. Homeland Security and the FBI are investigating, but have not speculated publicly on the source of the attack.

3) The source of the flood of traffic that caused the outage was a botnet, a large network of tens of millions of computers that were hijacked by the attacker and used to send massive amounts of data to Dyn’s systems. More specifically, the computers in this instance were “Internet of Things” (IoT) Internet-connected devices, mainly IP-based video/surveillance cameras and video recorders; everything from warehouse and corporate surveillance cameras to “nanny cams” and video baby monitors. These devices pose a significant risk to the Internet, and therefore to our economic and physical infrastructure; they’re often manufactured by companies that provide little to no support (the most common manufacturer provides none, selling their goods “white box” to be relabeled by distributors) and sold to users with little to no technical expertise. Unfortunately, there’s also no legal requirement in the US for them to be secure, generally no procedure for software updates, and usually no way for users to enhance the security of the devices.

The first two pieces of information should be deeply disturbing to all of us; in an age when “cyber warfare” is constantly in the media, yesterday’s even is the modern equivalent of a foreign power sending fighter planes over Washington, DC, to see how we react. There’s no plausible explanation for this, short of testing our defenses for an imminent or possible attack. It’s also important to note the implications of this beyond Twitter and Etsy; an attack of this magnitude launched against more than one - or all - of the leading DNS providers at once would cripple everything from banking, payments and credit card processing to healthcare and travel, and possibly large portions of our transportation, power and utility infrastructure. All areas of modern American life have become inextricably linked to the health of the Internet, whether we realize it or not (ever tried to eat at a restaurant when their “computers were down,” or fill your car with gas when there was a local Internet outage?) And while the original core of the network that became the Internet was designed to withstand a cold-war nuclear attack, the modern infrastructure relies increasingly on a small number of private companies. An attack of the scale possible from a nation-state has never happened before, and it’s not likely that it would be handled well.

On a separate note, the use of IoT “smart” devices in this attack is particularly unsettling. While most people expect that their desktop and laptop computers and smartphones will receive regular software updates from a company that cares about their continued functionality, and expect that their devices won’t be hijacked to attack critical infrastructure, the same is not true of the exploding field of “smart” devices. Robotic vacuum cleaners, Internet-based baby monitors, light bulbs that can be dimmed from your phone and networked thermostats are often sold as appliances at face-value, with very little ongoing support. Most people who buy these devices expect them to just work as advertised, and to be secure - they expect their baby monitor to not let hackers watch inside their homes and they expect their light bulbs to not attack critical Internet infrastructure on behalf of a third party. Perhaps the worst part is that, as many manufacturers of such devices are neither “computer companies” nor terribly interested in long-term customer relationships, every insecure device that’s sold will likely be in operation for the next five to ten years, or more. We need both a legislative solution to this, as well as a consumer-focused solution; perhaps the modern equivalent of Underwriter’s Laboratory for computerized devices. Without a solution, we may well end up in the world where, as suggested by some experts, it will be up to the Internet Service Provider of befuddled users to disconnect their service when a foreign power turns their toaster into a weapon.

Update Sunday October 23rd, 2016: I didn’t find it until now, but yesterday Dyn published an initial statement on the attack on their Blog. The details confirm what I read earlier and said above, but only add one new bit of information beyond what I’d read earlier: the botnet(s) acting as the source of the attack were vastly larger than originally stated; “tens of millions of IP addresses” rather than the approximately 500,000 originally suspected. I’ll be keeping an eye out for a more detailed follow-up from Dyn.