CourseAI / ML Engineer: The Complete Course · Module 7: Training at Scale · part 45 of 93
Part 45 · Module 7: Training at Scale

Lesson 2: Many machines, one model, and surviving a crash

27 min read·
Module 7 · Lesson 2 · About 55 minutes · Needs: lesson 1 (the 16 bytes per parameter and the ring all-reduce) and Module 4's training loop. Runs on a CPU: two processes stand in for two GPUs.

The situation

Week 7 at Kestrel, day three. Rhys has lesson 1's answer: the late-risk network needs no GPU. His next question is the one every platform lead asks: "When a job does need several machines, how do we split it, and what happens when one of them dies at 03:00?"

The running example is the late-risk network trained on 86,014 parcels by one script, scripts/ddp_train.py, in two processes. It is trained, split, measured, crashed after epoch 2 and resumed.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson spreads training across processes and makes it survive a crash (blue: training, experiments and the orchestrator that relaunches it); the rest is grey.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan