CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 7: Architecture Families and Design Choices · part 53 of 111

Part 53 · Module 7: Architecture Families and Design Choices

Lesson 4: Mixture of experts, more weights for the same work

27 min read·13 Oct 2026

Module 7 · Lesson 4 · About 55 minutes · Needs: Module 6, Lessons 1 and 4 (the proxy, train with loss_fn); Module 3, Lesson 1 (the SwiGLU feed-forward)

The situation

Sol 418 at Aurora Station. Hana Aoki has been reading model cards. "Mixtral has 47 billion weights but uses 13 billion per token. Could our model hold more knowledge for the same CPU time per character?" Kofi Mensah adds his usual condition: seeds, a matched comparison, and a test.

This lesson builds a mixture-of-experts (MoE) feed-forward layer into the lab as a ModelConfig switch, counts it, trains it and breaks it. The running example is one character: the "B" of "Bruno" in the first validation window, "airlock motor.\nBruno Hoffman works in th".

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399 See what is inside