Lesson 4: Watching a run, regularising it, resuming it exactly

Module 6 · Lesson 4 · About 50 minutes · Needs: Lesson 1 (the loop, the proxy, the floors); Lesson 2 (AdamW, decay, the step-22 spike)

The situation

Sol 414 at Aurora Station. Maya Lindqvist's log cards (Module 12) will give the lab 1,000 training pairs, not 1.8 million characters. Kofi Mensah asks what happens to the recipe when data is that scarce, and how you would see it before the final number. Hana Aoki adds that life-support jobs can take the CPU servers at any moment: "If a run is cut off, can you pick it up and get the same answer?"

This lesson builds the instruments, the regularisers and the checkpoint. The running example is the 32-document run: Lesson 1's proxy, trained with base's recipe at the proxy's tuned rate, on only the first 32 training documents.