CourseAI / ML Engineer: The Complete Course · Module 11: Monitoring and Reliability · part 67 of 93
Part 67 · Module 11: Monitoring and Reliability

Lesson 1: What to watch when the model is live

27 min read·
Module 11 · Lesson 1 · About 55 minutes · Needs: Module 3 lesson 3 (PR-AUC, Brier, calibration), Module 9 (the scoring service), Module 3 lesson 2 (isolation forest). The model is trained in one cell here.

The situation

Week 11. Nothing crashed in September, yet the model got worse. Adaeze asks you to replay the month as if monitoring had been running: "When would we have known, and from which signal?" Tomasz admits scanner v4 started sending grams on 7 September.

The running example is Monday 7 September 2026 (simulated), scored by the model that was live then. Doncaster, open since 10 August, and the three island routes, hit by gales on 15 to 17 September, run beside it.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson builds Kestrel's monitoring and the late-label join (blue), reading the scoring service's log; alerting is lesson 2, and the rest is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan