Lesson 4: Long inputs, experts, and what training costs
30 min read·13 Oct 2026
Module 9 · Lesson 4 · About 50 minutes · Needs: Lessons 2 and 3 (data, tensor and pipeline parallelism, the two-process launcher); Module 7, Lesson 4 (mixture of experts)
The situation
Sol 426 at Aurora Station. Hana Aoki, the Compute Officer, has Lesson 1's scaling fit and the split plans from Lessons 2 and 3. Her question is still open: "What would a model 1,000 times bigger than base cost to train on hired GPUs on Earth? My cap for any single run is $1,500."
The running example is that model, big. This lesson adds the last two ways to split a job, then the systems questions that turn a plan into an invoice.