Incident: One extra second made servers spin

Module 4 · Incident file · About 30 minutes

What happened

Clocks around the world keep UTC, Coordinated Universal Time, the world's reference clock, which computers keep in step with. The Earth's spin is slightly irregular, so now and then UTC adds a leap second to stay in step with it. Instead of going from 23:59:59 to 00:00:00, the clock shows one extra second, 23:59:60. That day has 86,401 seconds instead of the usual 86,400.

One leap second was added at the end of 30 June 2012. It triggered a bug in the way Linux, the operating system that runs most web servers, handled time. Processes on affected machines began to spin: they used the processor at full power while doing no useful work. Several large websites had outages (widely reported, July 2012).

No program crashed with an error message. Each one kept doing exactly what its code told it, round after round, as Lesson 3's date loop did on the last day of a leap year. That is why the failure was hard to see from inside the program, and why it belongs in a module about loops.