CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 7: Architecture Families and Design Choices · part 54 of 111
Part 54 · Module 7: Architecture Families and Design Choices

Lab: Presets, wirings and an expert switch, tested

29 min read·13 Oct 2026
Module 7 · Lab · About 60 minutes · Needs: lessons 1 to 4 of this module, and Module 6's lab

What you are building

TINYFORMER-LAB AFTER MODULE 7 moe.py experts, counts presets.py wirings, 8 presets config, model 4 fields, make_ffn v0.6.0 package: data, attention, trainer, unchanged test_m07_families 9 tests runs/m07_dense, m07_moe 2 arms x 2 seeds git tag v0.7.0
The lab adds moe.py, presets.py and an expert switch (blue) to the unchanged v0.6.0 package (grey), pins them with nine tests (green), records one seeded comparison (amber) and tags v0.7.0.

Sol 419 at Aurora Station. The four lessons left their work in notebook cells: Lesson 1's encoder and encoder-decoder, Lesson 2's published configurations, Lesson 3's ablation path, Lesson 4's experts. Kofi Mensah's question is the one from Module 6: does the package say what the lessons said? Hana Aoki wants the total and active weight counts of any expert model the lab can build, before anyone sizes a server for it.

Today you save all of it into tinyformer-lab: tinyformer/moe.py, tinyformer/presets.py, four new config fields and one changed line in the block. You pin them with tests that check presets against published file headers, then rerun Lesson 4's dense-against-experts comparison from the package with two seeds, and tag v0.7.0.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside