Lesson 3: Normalisation, and where the norm sits

Module 3 · Lesson 3 · About 50 minutes · Needs: Lessons 1 and 2 of this module (the config, the feed-forward network, the residual stream); Module 2 (attention)

The situation

Sol 407 at Aurora Station. The block from Lesson 2 trains. Kofi Mensah reads Block.forward and stops at self.norm1(x). "Why there, before attention, and not after the addition like the 2017 paper? And why RMSNorm? Show me what breaks if you move it."

This lesson adds the norms to tinyformer/layers.py and two switches to ModelConfig, norm_place and qk_norm. The running example is Lessons 1 and 2's line: "Log 4770, sol 286, the night shift."