CourseAI / ML Engineer: The Complete Course · Module 4: Deep Learning · part 26 of 93
Part 26 · Module 4: Deep Learning

Lesson 2: Keeping training stable, and debugging it when not

27 min read·
Module 4 · Lesson 2 · About 50 minutes · Needs: lesson 1 (a training step, AdamW), L2 penalties (Module 3 lesson 1), and the Kestrel data (course-guide/06).

The situation

Week 4 at Kestrel, day three. The lesson 1 network trained in four epochs. Given longer runs, more layers or September's data, networks fail in quieter ways: they memorise, they stall, they blow up on one bad batch. Adaeze's rule for ADR-002: "No network goes in front of the trees until you can tell me why each run behaved as it did."

The running example is a wide network trained on a small sample, 3,000 training parcels, so every failure shows in seconds. KP0104204, the late Dartford parcel from lesson 1, is scored along the way.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson makes training runs stable and explainable (blue): the loop and the checks every later training run reuses; the rest of the system is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan