A Typo Broke The Internet, Amazon Bungled Bigly

Recently, huge parts of the internet went down for a period of 4 hours because of an issue with the cloud computing service by Amazon called S3. The company finally got to the bottom of the issue and it would seem that the reason for the blackout is good old-fashioned human error. It was caused by a typo, to be more specific, which is a small misstep that had huge consequences.

According to a post-mortem report that Amazon posted, the internet outage was caused by an incorrect line of code. The person responsible was supposed to get rid of only a small fraction of the servers that are tied to S3. The coding error ended up decimating a considerable number of servers instead.

“At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process,” the report reads. “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”

The blunder resulted in Amazon needing to restart the systems that were affected, which took much longer than expected. During that time, S3 was unable to accept requests from the list of clients that were affected by the particular servers that went down, Ars Technica reports.

It’s worth noting that the biggest contributor to the slow restart is the considerable growth of S3. This resulted in much bigger systems that needed to reboot, which does lead to more data that needs processing and functions that needed executing. More to the point, Amazon also wrote that it hadn’t had to restart most of the servers that went down in years, which arguably made things more cumbersome.