CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 2: Self-Attention Mechanics · part 15 of 111

Part 15 · Module 2: Self-Attention Mechanics

Lesson 3: Multi-head attention, split, joined and switched off

21 min read·13 Oct 2026

Module 2 · Lesson 3 · About 50 minutes · Needs: Module 2, Lessons 1 and 2 (naive_attention with scale, the Maya trace, the GPT-2 head survey)

The situation

Sol 405 at Aurora Station. naive_attention now has a test and a scale switch. Kofi Mensah opens the base configuration: width 128, 4 heads of width 32. "Why four? One head of 128 has the same weights. And Lesson 1 found 144 heads in GPT-2. If one does nothing, show me."

This lesson adds MultiHeadAttention to tinyformer/attention.py, races one wide head against several at equal size, and switches GPT-2's heads off one at a time and together. If you met heads in the ai-engineering course (Module 6, Lesson 1), skim section 1.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399 See what is inside