Incident: One mistyped input, and no way to say so

Module 14 · Incident file · About 30 minutes

What happened

On the morning of 28 February 2017, Amazon Simple Storage Service (S3) in AWS's Northern Virginia region (US-EAST-1) stopped serving requests. AWS published what happened in its own summary (AWS, "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region", 2017). Everything below comes from that page; times are US Pacific (PST).

The S3 team was debugging a billing system that was running slowly. At 9:37 AM "an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers". "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended." Those servers supported two other subsystems, the index and the placement subsystems, and both needed "a full restart". While they restarted, "S3 was unable to service requests". Services that rely on S3 in the region were hit too, among them "the S3 console", new EC2 instance launches and AWS Lambda. S3 was "operating normally" again at 1:54 PM, 4 hours 17 minutes after the command, and some services then had a backlog to clear.