CourseAI Engineering: The Complete Course · Module 11: Data Stores · part 88 of 105
Part 88 · Module 11: Data Stores

Incident: A two-millisecond migration stalled the API

10 min read·13 Oct 2026
Module 11 · Incident file · About 35 minutes

What happened

In May 2015 Chris Sinjakli of GoCardless, a payments company, published "Zero-downtime Postgres migrations - the hard parts". It opens: "A few months ago, we took around 15 seconds of unexpected API downtime during a planned database migration." For a payments API that matters: "if we're not accepting requests, our merchants are losing money."

GOCARDLESS, LATE JANUARY (PUBLISHED MAY 2015) before long read on hot table deploy migration queues 0 to 15 s API requests time out about 15 s migration goes through later backup: few 100 ms
A long read was running (grey), the migration queued (amber), API requests timed out for about 15 seconds (red), then the migration went through (green). On a backup it took a few hundred milliseconds (blue).

In late January the team was building invoicing. They renamed some tables that were empty and unused in production, then dropped and recreated their foreign key constraints to fix the names. "Just after the schema migration started, we started getting alerts about API requests timing out. These lasted for around 15 seconds, at which point the migration went through and our API came back up." Rerun against a backup from earlier that day, the same migrations "went through in a few hundred milliseconds." The slow part was not the change. It was the wait.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside