Transformers: From Attention to Trained, Served and Interpreted Models
Most explanations of transformers stop at a diagram of attention. This course goes all the way through: you derive the mechanics, build the model, train it, scale it, serve it cheaply, adapt it, and then take it apart to see what it learnt.
Across 14 modules and a capstone you build one project: tinyformer-lab, your own configurable transformer, in which every architectural choice is a switch you can flip and measure. It is set at Aurora Station, a fictional research station on Mars whose slow link to Earth means it must run its own small models on its own CPUs. Everything runs in Google Colab on the free CPU runtime.
Who this course is for
| You are... | Start at |
|---|---|
| Comfortable with Python and NumPy, have trained one neural network, new to transformers | Module 1 |
| Have done the AI Engineering course up to its Module 6 | Module 1, skimming the sections each module README marks as covered there |
| Already build transformers at work | Any module: each README's What you need first says whether it is safe to jump in |
What you need
- A Google account, to run code in Google Colab. No GPU and no installation for most lessons.
- Python and NumPy, basic calculus (the chain rule) and linear algebra (matrix products, transposes).
- Having trained at least one neural network before. If you have not, do the AI Engineering course's Module 4, Deep Learning Basics first: it covers backpropagation, optimisers, schedules and initialisation.
- Helpful but not required: the AI Engineering course's Module 5 (tokenisation and why recurrent models struggle) and Module 6 (the transformer in overview, model families, fine-tuning and inference). This course goes underneath all of them.
- A free Groq or Gemini key only for the few lessons that compare the lab's model with a large hosted one.
Setup takes about ten minutes: course-guide/02-setup-colab-and-llm-providers.md.
The modules
| # | Module | What you add to tinyformer-lab |
|---|---|---|
| 1 | The Road to Attention | The repository and its data; a GRU raced against a transformer on one task |
| 2 | Self-Attention Mechanics | Attention by hand, masks, multi-head, grouped and multi-query heads |
| 3 | Anatomy of a Transformer Block | The config, feed-forward variants, norms, the residual stream, embeddings and head |
| 4 | Positional Information | Sinusoidal, learned, relative, ALiBi and rotary positions; context extension |
| 5 | Building a Transformer from Scratch | The finished model, numerics, and the tests that catch real bugs |
| 6 | Training Dynamics | The trainer, schedules, stability, and the base recipe |
| 7 | Architecture Families and Design Choices | Encoder, decoder and encoder-decoder presets; mixture of experts |
| 8 | Pretraining | Corpus report, deduplication, contamination check, tokeniser, objectives, packing |
| 9 | Scale and Distributed Training | A scaling-law fit, memory and cost arithmetic, parallelism sketches |
| 10 | Efficiency and Long Context | The KV cache, sparse, linear and IO-aware attention |
| 11 | Inference and Serving | Decoders, constrained and speculative decoding, batching, paged cache, quantisation |
| 12 | Adaptation and Alignment | Fine-tuning, LoRA, reward model, DPO, and the log-card ship decision |
| 13 | Beyond Text | A vision transformer, CLIP and a captioner on generated shapes |
| 14 | Analysis, Interpretability, and Frontiers | Hooks, ablation, probes, logit lens, patching; a state space model |
| 15 | Capstone Project | The honest write-up, with a run manifest and cost account for every result |
How each lesson works
Every lesson is a walk through one example from the lab, one picture at a time: a figure, a short paragraph that says what it shows, a short block of code that proves it, and a walkthrough of every line. Each lesson opens with a real, cited case, then runs the mechanism, breaks it on purpose, says what changes at real scale, and ends with questions, one exercise with a runnable check, key terms and sources. Each module ends with a lab and an incident file.
Every number is labelled: measured here (from the lesson's code, with the machine named for timings), cited (from a named, dated source) or arithmetic (from a formula on the page).
More detail: course-guide/01-how-this-course-works.md.
The tools you will use
Plain PyTorch first, for every mechanism. Hugging Face transformers to load two small published models (GPT-2 small, MIT licence; SmolLM2-135M, Apache 2.0), tokenizers to train a tokeniser, peft for LoRA once you have built it by hand. Versions are pinned in the setup guide.
Course guide
| File | Read it when |
|---|---|
| 01-how-this-course-works.md | Before your first lesson |
| 02-setup-colab-and-llm-providers.md | Before you run any code |
| 03-the-tinyformer-lab.md | To see what you are building, and at Aurora Station for whom |
| 06-shared-data.md | To see the generated data and the reference model every module reuses |
| 04-style-guide.md | Only if you are writing or editing lessons |
| 05-image-style-guide.md | Only if you are drawing the course figures |