CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 5: Building a Transformer from Scratch · part 40 of 111
Part 40 · Module 5: Building a Transformer from Scratch

Lab: A tested model that runs in 16 bits

25 min read·13 Oct 2026
Module 5 · Lab · About 70 minutes · Needs: lessons 1 to 5 of this module, and Module 4's lab

What you are building

TINYFORMER-LAB AFTER MODULE 5 four repairs small exact edits init.py schemes, GPT-2 v0.4.0 files float32 unchanged test_m05_model 16 tests, cheapest first m05_precision 3 formats x 3 seeds git tag v0.5.0
The lab repairs the package and adds init.py (blue) on v0.4.0 (grey), pins everything with sixteen tests (green), runs one seeded experiment (amber) and tags v0.5.0.

Sol 409 at Aurora Station, end of Kofi Mensah's review. The lessons left four repairs and a pile of checks in notebook cells: a bfloat16 model crashes (Lesson 3), a float64 model returns a float32 loss (Lesson 5), GPT-2 small loads only after a hand patch (Lesson 2), and nothing refuses a batch of the wrong shape (Lesson 1).

Today you make the four repairs, each with a test, prove no float32 number moved, answer Hana Aoki's 16-bit question with three seeds, and tag v0.5.0.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside