CourseAI System Design: The Complete Course · Module 10: Deployment and Serving · part 77 of 107
Part 77 · Module 10: Deployment and Serving

Incident: One bad config value crashed every pod

10 min read·
Module 10 · Incident file · About 35 minutes

What happened

CASE STUDY ChatGPT, 15 July 2025 (PDT) 7:43 PM errors begin 8:38 PM full availability about 55 minutes revert, resync, rolling restart a small percentage of users; no counts published
The incident as OpenAI's write-up gives it: errors for a small percentage of users from 7:43 PM (red) until full availability at 8:38 PM (green), about 55 minutes.

On 15 July 2025 ChatGPT had "reduced availability between 7:43 PM and 8:38 PM PDT", according to OpenAI's write-up on its status page, "Elevated Error Rate on ChatGPT". The impact: "Increased error rates for ChatGPT conversations for a small percentage of users for approximately 55 minutes". The cause was "an invalid configuration value used by a core service", which "caused new server instances to fail when restarting".

OpenAI resolved it "by restoring the correct configuration and forcing an immediate resynchronization of affected services", with "A rolling restart of affected pods" to speed recovery and "full availability restored by 8:38 PM PDT". The write-up gives the revert time as 7:12 PM PDT, earlier than the start it reports; this file uses only the start, the end and the 55 minutes. It gives no count of users or requests. Everything here is from that write-up.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan