CourseAI System Design: The Complete Course · Module 9: Agentic Architecture · part 68 of 107
Part 68 · Module 9: Agentic Architecture

Incident: An agent's cleanup destroyed production

20 min read·
Module 9 · Incident file · About 35 minutes

What happened

TIMELINE About 24 hours down DATATALKS.CLUB, FEBRUARY 2026 (HIS TIMES, TO SCALE UNTIL 2 AM) Thu 26 Feb Fri 27 Feb ~10 PM no state file ~11 PM destroy runs 12:00 AM support upgrade 12:30 AM snapshot exists 1 to 2 AM call, escalated ~10 PM restored about 24 hours down; 1,943,200 rows back in one table
Grigorev's timeline: deploying without the state file (amber), the destroy an hour later (red), AWS support through the night (blue), and the database restored about 24 hours after it was deleted (green).

On 6 March 2026 Alexey Grigorev, founder of DataTalks.Club, published "How I Dropped Our Production Database and Now Pay 10% More for AWS" on his newsletter, Alexey On Data. He was moving his AI Shipping Labs website to AWS with Claude Code, and "I was overly reliant on my Claude Code agent, which accidentally wiped all production infrastructure for the DataTalks.Club course management platform". That platform held "2.5 years of all submissions: homework, projects, leaderboard entries".

His timeline starts on Thursday 26 February at about 10 PM, deploying with Terraform without the state file, "as it was on my old computer". At about 11 PM a Terraform auto-approve command "inadvertently wiped out all production infrastructure, including the Amazon Relational Database Service (RDS)", and "all snapshots were also deleted". He upgraded to AWS Business Support at midnight, which "costs me an extra 10%"; AWS found a snapshot on its side, and by about 10 PM on Friday the database was back, with "1,943,200 rows in the courses_answer table alone". He writes: "This incident was my fault". Everything here is from his post.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan