Module 7: Training at Scale
1 min read·
What you need first
- What a parameter, a batch and an optimiser are (Module 4 lesson 1), a PyTorch training loop (Module 4 lesson 2) and profiling Python (Module 2 lesson 1). The lab's Step 1 rebuilds Modules 1 to 6.
- No GPU and no keys: two CPU processes stand in for two GPUs; a free Colab T4 adds one optional measurement.
- If you can already budget GPU memory, run DDP and resume from checkpoints, go to the lab.