CourseAI / ML Engineer: The Complete Course · Module 7: Training at Scale · part 46 of 93
Part 46 · Module 7: Training at Scale

Lesson 3: Where training time goes, and how to win it back

22 min read·
Module 7 · Lesson 3 · About 50 minutes · Needs: profiling Python (Module 2 lesson 1), Module 4's training loop, and lesson 1's roofline. No GPU needed.

The situation

Week 7 at Kestrel, day four. Rhys has a rule for anything that runs every night: "Before anyone buys hardware, show me where the time goes." Lesson 1 found that the late-risk network barely uses a CPU. This lesson finds out what it waits for instead.

The running example is the same network fed three ways: ParcelRows, a Dataset that builds each parcel from its pandas row, the version a first draft would write; tensors read parcel by parcel; and whole batches cut from tensors. Parcel KP0000001, the first training parcel, comes out as the same 15 numbers from all three.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson speeds up training from the inside (blue: training, and the experiments that record its throughput); the rest of Kestrel's system is grey.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan