Lesson 4: Paged cache memory and who runs next

Module 11 · Lesson 4 · About 55 minutes · Needs: Module 10, Lesson 3 (KV cache arithmetic and batch size); Lesson 1 of this module (prefill, decode and the cached decoding loop); Module 8's package as pasted in Module 8, Lesson 4

The situation

Sol 434 at Aurora Station, and the Earth link is still down. Every question the crew would have sent to Earth now goes to base on the station's CPU servers. Hana Aoki has set aside 4 MiB of one server's memory for the KV cache: how many crew can it serve at once? Tariq Haddad adds a worry: when a long transmission from the relay buffer is read, everyone else's replies freeze. You take the memory first, then the order of work.