Incident: A distributed design hit its limit at 5% load

Module 4 · Incident file · About 30 minutes

What happened

On 22 March 2023 Marcin Kolny, a senior engineer at Prime Video, published "Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%" on the Prime Video Tech blog. The original address now redirects to an Amazon page; the post is read here from the Internet Archive's copy. On 5 May 2023 Tim Anderson reported it for DevClass ("Reduce costs by 90% by moving from microservices to monolith: Amazon internal case study raises eyebrows"), with the developer community's reaction.

Prime Video wanted a tool that would "monitor every stream viewed by customers" and catch quality problems such as "block corruption or audio/video sync problems". The team's existing tool was never "designed to run at high scale", and the target was "thousands of concurrent streams". While onboarding more streams they found that "running the infrastructure at a high scale was very expensive", and found scaling bottlenecks. The design, in the post's words, "caused us to hit a hard scaling limit at around 5% of the expected load", and "the overall cost of all the building blocks was too high to accept the solution at a large scale". The post gives no dates for the build or the move.

This is not an AI system: the defect detectors analyse video frames, but what failed was the architecture around them. The arithmetic is the same one an agent design must pass.