Incident: A missed server and a rollback that made it worse

Module 13 · Incident file · About 35 minutes

What happened

You met Knight Capital in Module 10 Lesson 5, from the testing side; this file looks at the same morning from the deploy and rollback side.

On 16 October 2013 the US Securities and Exchange Commission published its order against Knight Capital Americas LLC (Release No. 34-70694). It describes the morning of 1 August 2012. Knight's automated order router, SMARS, received 212 small customer orders and "routed millions of orders into the market over a 45-minute period", obtaining "over 4 million executions in 154 stocks for more than 397 million shares". The order states that "Knight lost over $460 million from these unwanted positions." In the SEC's press release the same day, "SEC Charges Knight Capital With Violations of Market Access Rule", Knight "agreed to pay $12 million to settle charges".

The cause was a deployment. From 27 July Knight placed new code on its eight SMARS servers over several days, but "one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers." The SEC adds: "Knight did not have a second technician review this deployment", and it "had no written procedures that required such a review." From about 8:01 a.m. ET that day, an internal system sent 97 automated e-mails mentioning "Power Peg disabled" before the 9:30 market open. They were not designed as alerts, and nobody acted on them.