CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 9: Scale and Distributed Training · part 69 of 111
Part 69 · Module 9: Scale and Distributed Training

Incident: Faulty chips that trained models quietly wrong

11 min read·13 Oct 2026
Module 9 · Incident file · About 35 minutes

What happened

THE STUDY, IN ORDER stress test 15 nodes fail held out temporarily compared vs 15 healthy findings drift, spikes Feb 2025 paper posted nodes then repaired and returned to the fleet
The study took nodes a fleet had already rejected (red), held them briefly (grey), and compared them with healthy ones (blue) before they went back for repair.

In February 2025 Jeffrey Ma, Hengzhi Pei, Leonard Lausen and George Karypis (Harvard and Amazon Web Services) published "Understanding Silent Data Corruption in LLM Training" (arxiv.org/abs/2502.12340, ACL 2025). Silent data corruption (SDC) is "hardware produces incorrect computations without explicit failure signals": no crash, no error, a wrong number.

Their material was real faulty hardware. Working "with a cloud computing platform", they used 15 nodes that had "failed production margin stress tests"; the team held them "only temporarily" before repair and compared each with healthy nodes, using deterministic execution so that any difference was the hardware's.

WHAT THE FAULTY NODES DID (CITED) gradient error up to 5.1% pretraining same loss, weights drift fine-tuning 3 loss spikes one fine-tuned model: 0.00% test accuracy no error message in any of these
The faults ranged from small gradient noise (amber) to a fine-tuned model left at zero accuracy (red), all without an error (cited from the paper).

What they found is what makes SDC dangerous. On the worst node, the gradient error was "5.1% of that of the ground-truth gradients". In pretraining, losses stayed nearly identical to the healthy run while the weights drifted apart; one node showed no difference for 450 steps, then began to drift. In fine-tuning, three nodes produced loss spikes, and on one "the loss spike occurs near the end of fine-tuning, which leads to the resulting model having zero test accuracy". The paper's introduction cites the scale: six unplanned interruptions blamed on SDC in Llama 3's 54-day snapshot, and Google's estimate of an SDC event "every week or two" in Gemini training.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside