How This Course Works
One lab, built piece by piece
You do not study fifteen separate subjects. You build one thing: tinyformer-lab, a small transformer in which every design choice is a switch you can flip and measure. Each module adds a part to it. By the end you have one repository that builds, trains, serves, adapts and takes apart a transformer, with a test behind every claim.
The lab is set at Aurora Station, a fictional research station on Mars whose slow link to Earth means it must run its own small models on its own CPUs. The setting, its people, its data and its timeline are in 03-the-tinyformer-lab.md. It is made up; the real cases that open each lesson are not.
A mechanism course
Most explanations of transformers stop at a diagram of attention. This course goes through the machinery: you derive each piece, build it, run it, break it on purpose, and measure what changed. Where something only shows up at the scale of a frontier model, the lesson says so, cites the published measurement, and runs the small version honestly.
Every number in the course is one of three kinds, and the lesson says which:
| Kind | Means |
|---|---|
| Measured here | Printed by the lesson's own code. Timings also say which machine |
| Cited | From a named, dated source: a paper, a datasheet, a model card |
| Arithmetic | Worked out from a formula on the page |
Pictures first
Every lesson is a walk through one example, one picture at a time. A figure shows an idea; a short paragraph says what the figure shows in plain words; a short block of code proves it; a walkthrough explains every line of that code. Then the next figure. A lesson has fifteen to thirty figures and takes thirty to forty-five minutes to read properly.
The same colours mean the same things in every figure:
| Colour | Means |
|---|---|
| Blue | Your code and the normal data path: tensors flowing through the lab |
| Amber | What the model learnt or predicts: weights, scores, probabilities, anything uncertain |
| Green | Correct, passed, matches the reference |
| Red | Failure, a bug, a wrong answer |
| Grey | Optional, masked out, ignored |
The shape of every lesson
| Part | What happens | Why it is there |
|---|---|---|
| The situation | A moment at Aurora Station, with a named person and a reason to care | You learn the mechanism because you need it |
| A real case, first | Something that really happened, from a published source, cited | So you know the idea matters outside a course |
| The idea | Figures and short paragraphs, one running example throughout | The picture comes before the vocabulary and the formula |
| Run it | A short piece of code, pasted into Colab, with a code walkthrough | You see it work, and you understand every line |
| Break it | You make it fail on purpose: a missing mask, a wrong scale, a bad seed | See the failure here before you meet it in a real run |
| In production | What changes at real scale, and what real teams do about it | The part most courses leave out |
| Check yourself | Three to five questions, answers hidden until you click | You find out whether it stuck |
| Exercise | One task, with a check you can run | You do it, not just read it |
| Key terms and sources | A short glossary and the sources the lesson used | So you can go further |
Three depths of topic
Each topic is marked in its heading:
| Mark | Meaning | What you get |
|---|---|---|
| (Essential) | You cannot understand or build transformers without it | Figures, code and a "break it" step |
| (Recommended) | Skip it only if you have a reason | A figure and a short section, often with code |
| (Optional) | Knowing it exists is enough | An "awareness card": what it is, when you would need it |
If you are short of time, do the Essential sections of every module first and come back for the rest.
What every module contains
| File | What it is |
|---|---|
README.md | What the module is for, what you need to know first, the lesson list |
| Numbered lessons | One file per group of topics |
...-module-lab.md | A guided build that adds this module's piece to tinyformer-lab, with a runnable check |
...-incident-file.md | A real, published failure, then the same failure replayed in the lab and fixed |
Labs. Each lab runs top to bottom in one fresh Colab notebook, saves its files into the tinyformer/ package, adds a test, and ends with a check that prints whether you passed.
Incident files. Each one starts with a documented failure from the outside world, cited: a training run that diverged, a bug in a widely used library, a published post-mortem. It explains why it happened, replays the same failure in the lab, and shows the fix and the test that would have caught it.
Skip what you know
You may already know some of this. Each module README starts with What you need first, so you can check whether it is safe to jump ahead. If you have done the AI Engineering course, its Module 6 (Transformers) gave you the overview; this course goes underneath it, and the module READMEs say which sections you can skim.
How to run the code
All code runs in Google Colab, on the free CPU runtime. Open a new notebook, paste each code block into a cell in order, and run it. Nothing needs installing for most lessons, because Colab already has PyTorch.
- Most lessons start with the short torch setup cell and paste the slice of shared data they need from 06-shared-data.md. All of that data is generated by code; nothing is downloaded.
- A few lessons look inside a real, published model, GPT-2 small or SmolLM2-135M, downloaded once from Hugging Face. No account or key is needed.
- A few lessons compare the lab's model with a large hosted model. Those start with the standard provider setup cell, which lets you choose Groq or Gemini in Colab (free key) or Ollama on your own computer, by changing one word.
Both cells, the reference models and running locally: 02-setup-colab-and-llm-providers.md.
Most lesson training runs are sized to finish in a few minutes on a Colab CPU; a few Module 12 runs take longer, and the lesson says so before the cell. Timings in lessons are from the author's Mac with one CPU thread. A GPU is never required; where one makes a measurement more interesting, the lesson says "optional GPU". Things that cannot run in a notebook, such as a multi-GPU cluster or a production inference server, are given as exact instructions, stated as not run, with what was checked against the official documentation.
The tools
Everything in the course is open source, apart from the hosted model services, which are shown as options. You build each mechanism by hand in plain PyTorch first, check it against PyTorch's own version where one exists, and only then use a library: Hugging Face transformers to load published models, tokenizers to train a tokeniser, peft for LoRA.
Your numbers will be close to ours, not identical
The code is seeded, so losses, counts and accuracies should match the course's to the digits shown when you use the same versions. Timings will not match: they depend on your machine, and every timing in the course says which machine it came from. When a hosted model is called, its words will differ from run to run. Every number printed in a lesson came from running that lesson's code on the date shown.