Lesson 3: Attention that respects memory, and the KV cache

Module 10 · Lesson 3 · About 60 minutes · Needs: Module 2, Lesson 5 (KV cache, GQA, MQA, MLA); Module 5, Lesson 4 (fused attention, checkpointing); Module 8's package as pasted in Module 8, Lesson 4

The situation

Sol 431 at Aurora Station. Tariq's relay splits long transmissions from Earth into 128-character pieces, because base reads only 128 at a time. Lesson 1 priced reading one 4,096-character transmission whole. Hana Aoki asks two things. "Can attention at 4,096 avoid a 4,096 by 4,096 grid of scores? And when the station serves many readers at once, what does each cost me in memory?"

You answer both with the same transmission: first attention without the grid, then the keys and values a server keeps for it, shrunk four ways.