Lesson 1: The config object and the feed-forward network
27 min read·13 Oct 2026
Module 3 · Lesson 1 · About 50 minutes · Needs: Module 2, Lessons 1 to 4 (attention, heads, masks); Module 1, Lesson 1 (tensors, matrix products)
The situation
Sol 406 at Aurora Station. Attention is tested. Kofi Mensah points at one line of the base recipe, d_ff = 344. "Why not 512? Why three matrices? If two thirds of the weights sit in this layer, show me what it does."
This lesson starts the block: ModelConfig, which holds every switch, for tinyformer/config.py, and the feed-forward network for tinyformer/layers.py. The running example is the first maintenance log in the validation set, Ana Dubois's entry for sol 286.