CourseAI System Design: The Complete Course · Module 15: Communicating a Design · part 105 of 107
Part 105 · Module 15: Communicating a Design

Incident: One shared dependency, and a blind monitor

10 min read·
Module 15 · Incident file · About 30 minutes

What happened

TIMELINE 73 hours to bring players back ROBLOX, OCTOBER 2021, PACIFIC TIME 27 Oct 14:00 streaming on routing service 28 Oct 13:37 Vault slow Consul CPU high 28 Oct 16:35 players at 50% then all down 30 Oct 15:51 streaming off writes recover 31 Oct 16:45 all players back, 73 hours review published 20 January 2022; no user data lost; no cost given
73 hours from the first players lost (red) to all players back (green), most of it spent finding two causes.

From 28 to 31 October 2021 Roblox was down for 73 hours. On 20 January 2022 Daniel Sturman, writing with Scott Delap, Max Ross and others from Roblox and HashiCorp, published the review: "Fifty million players regularly use Roblox every day", on "over 18,000 servers and 170,000 containers" (Roblox, "Roblox Return to Service", 20 January 2022). Times below are Pacific.

At 13:37 on 28 October, Vault was slow and one Consul server had high CPU; players were not yet affected. At 16:35 the number of players online fell to half, then the whole service went down. The team tried four diagnoses in turn: failing hardware, then faster 128-core machines, then a reset from a snapshot, then cutting Consul's load. The cause was found on 30 October; at 16:45 on 31 October all players were let back in. No user data was lost; the review gives no cost.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan