CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 5: Building a Transformer from Scratch · part 35 of 111

Part 35 · Module 5: Building a Transformer from Scratch

Lesson 1: The whole model, traced shape by shape

29 min read·13 Oct 2026

Module 5 · Lesson 1 · About 50 minutes · Needs: Module 3's lab (tinyformer at v0.3.0); Module 2, Lesson 4 (the causal mask)

The situation

Sol 409 at Aurora Station. Kofi Mensah, the Reliability Engineer, opens tinyformer/model.py for review. "Every tensor in here is (32, 128, 128). Which 128 is time and which is width? If you swapped them, would anything tell you?"

This lesson answers him: the model end to end, one batch traced shape by shape, named axes, a free shape checker, and three bugs that keep every shape right. The running example is the first validation batch, whose row 0 begins "airlock motor. Bruno Hoffman works in th".

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399 See what is inside