Lesson 1: Prefill, decode and the decoding loop

Module 11 · Lesson 1 · About 45 minutes · Needs: Module 5 (the model and its causal mask), Module 6 (the base recipe); Module 10's KV cache arithmetic helps but is not required

The situation

Sol 433 at Aurora Station. The link to Earth is down for three sols, so the station can use only its own models, on its own CPUs. Tariq Haddad, the comms officer, wants the log console to draft maintenance entries with base while the link is out. He asks for two numbers, not one: how long before the first character appears, and how long between the characters after that. "Operators forgive a short wait. They do not forgive a reply that stutters more the longer it gets."

The running example is one request from the console: the header Log 4821, sol 434, the night shift. plus a newline, continued by base, greedy, up to 90 new tokens.