Lesson 3: Numerics, what 16 bits keep and what they lose

Module 5 · Lesson 3 · About 50 minutes · Needs: Module 4's lab (tinyformer at v0.4.0); Module 3's incident file (float16 NaNs in T5)

The situation

Sol 409 at Aurora Station. Hana Aoki, the Compute Officer, wants the next training runs in 16 bits: half the memory per number, and faster arithmetic on Earth's hired GPUs. Kofi Mensah objects. "Gemma lost its positions in bfloat16. Ours is rotary too. Show me ours does not, and where every other number could break."

The running example is row 0 of the copy task: 16 random tokens, a separator, the same 16 again. At position 31 the model reads token 20 and must predict token 4.