Lab: What Kestrel's training costs, and ADR-004

Module 7 · Lab · About 50 minutes · Needs: lessons 1 to 3 of this module, and the Module 6 lab (or nothing: Step 1 rebuilds Modules 1 to 6)

What you are building

Imran's question is the first of the course's five: "What does one prediction cost, and one training run?" Module 8 answers the first half. This lab answers the second, for the two models Kestrel actually has: v1 from the Module 3 lab and v2 from the Module 6 lab. The lessons measured memory, throughput, checkpoints and prices on stand-ins; the lab puts those tools to work on the real models and writes down the decision Rhys keeps being asked about: does the late-risk model need a GPU?

The lab adds kestrel/scale/report.py, which writes reports/training-throughput.md from measured numbers, and docs/adr/ADR-004-training-hardware.md. It also runs Lesson 2's training script as Kestrel's resumable job: killed half way, relaunched, and checked weight for weight. No cell calls a hosted model.

Open a new Colab notebook, call it module-07-lab.ipynb, and run each cell in order. A CPU runtime is enough.