The Project: tinyformer-lab at Aurora Station

Every lesson in this course adds to one project: your own configurable transformer lab, tinyformer-lab. Every design choice in it is a switch you can flip and measure: the position method, the normalisation, the activation, the number of key/value heads, tied embeddings, sliding windows, experts. You finish with one repository that builds, trains, serves, adapts and takes apart a transformer, with a test for every claim it makes.

The lab has a setting, so that its data, its people and its deadlines stay the same from Module 1 to Module 15. Aurora Station and everyone on it are made up. All names, logs, numbers and events in this setting are fictional and simulated. The real cases that open each lesson are separate, cited, and marked as real.

The setting

Aurora Station is a fictional research station on Mars with a crew of about forty. It keeps a written record of everything: maintenance logs, assignment sheets, crew conversations, greenhouse reports and procedures. That record, up to sol 399, is the course corpus (a sol is a Martian day).

The station has one problem that shapes the whole course: its link to Earth is slow, narrow and sometimes down. A message takes several minutes each way and bandwidth is rationed. Large hosted models on Earth can be asked a question only now and then, through a relay, and not at all during an outage. So the station runs its own small models, on its own small CPU servers. That is why every lesson in this course runs on a CPU, why every model is small enough to train in minutes, and why efficiency, quantisation and honest measurement matter here.

You are the station's machine-learning engineer. You own tinyformer-lab.

The people

All five names come from the corpus's own name lists, so they can turn up in generated documents too. That is a coincidence of the generator, not a story event. Minor characters may appear in one lesson; the main cast above does not change.

The data

Everything is generated by code in 06-shared-data.md, from one seed. Nothing is downloaded except the two small published models below.

Two real, published models are used where a lesson needs to look inside a model someone else trained: GPT-2 small (openai-community/gpt2, MIT licence), the course's reference model, and SmolLM2-135M (HuggingFaceTB/SmolLM2-135M, Apache 2.0), the example of the modern default stack. Details and how to load them: 02-setup-colab-and-llm-providers.md, section 6.

Fixed facts

These are set once here so that every module states them the same way.

• Course date: Friday 9 October 2026. Versions, model names, prices and documentation were checked on this day. Lessons say "(checked October 2026)".
• Currency: US dollars, for every cost in the course, because the hardware prices the course cites are published in dollars.
• The data snapshot is sol 400. Every generated document is dated sol 1 to 399. The story starts on sol 400.
• The seed is 11 (SEED in 06). Datasets that must not overlap use SEED + 1, SEED + 2 and so on, as 06 shows.
• Validation loss always means the average over VALID_BATCHES (eight fixed batches of 32 windows of 128 tokens) in nats per token, unless a lesson says otherwise.
• Every number is labelled as one of three kinds: measured here (from running the lesson's code, with the machine named), cited (from a named source, with a date) or arithmetic (worked out from a formula on the page). Timings are always "measured here" and always say which machine.
• Hosted model prices are kept in one place: the table in course-guide/02, section 4. GPU hire prices and hardware datasheet figures are kept in one place: the table in Module 9, Lesson 4, section 8 ("Costing a training run"), dated there (Lambda prices, October 2026; the H100 dense figure from NVIDIA's datasheet). Module 10's memory-bandwidth figures should refer to it. Other lessons refer to those two places and do not restate the numbers.
• Hana's 1,000-times model is big: width 2,048, 16 layers, 16 heads, context 2,048, 805,777,408 parameters (about 1,007 times base). Module 9, Lesson 4 defines it and the Module 9 lab saves it as scaling.BIG. Earlier Module 9 lessons round it to "about 800 million" or 1,000 times base (800,512,000); big is the one to quote.
• The station's model servers are small CPU machines, the same shape as a free Colab CPU runtime. A lesson's training run should finish in a few minutes there (a Module 12 LoRA run at the 448-character card window takes 3 to 4 minutes on the author's Mac, one thread).

Key events

Lessons refer to these events. They are fixed, so every module tells the same story. They happen in sols after the snapshot.

One repository: which module adds which file

Everything you keep goes in one repository, tinyformer-lab, around a single Python package called tinyformer/. Lessons run in Colab, so their code sits in cells; each module's lab saves the pieces into the package (with %%writefile) and adds a test. Push the repository to your own GitHub account after each lab, because Colab's disk is wiped when the runtime stops.

Each module's experiment script reproduces the module's main table. Results go in runs/, never inside tinyformer/.

Two things about the order. The Module 9 and Module 10 labs both start from v0.8.0 and touch separate files, so you can do either first; the Module 11 lab rebuilds both sets of files (each written only if missing) before it adds its own. From Module 11 on, a lab that finds no repository of yours rebuilds a stand-in of every earlier file it needs, so you can start at any later module.

What you keep from the old version of this course

This is the second version of the course. The first assumed a lab folder, a shipped pretrained model (models/base), a 9 MB generated corpus and an image dataset that learners did not have. This version keeps the ideas and makes everything reproducible inside a Colab notebook:

What "done" means

At the end of the course your repository can answer Kofi's question about any claim in it: show me the test, show me the run, show me the noise floor. If it can, you understand transformers well enough to build one, change one and say honestly what your change did.