CourseAI / ML Engineer: The Complete Course · Module 3: Classical Machine Learning · part 21 of 93
Part 21 · Module 3: Classical Machine Learning

Lesson 3: Honest models: leaks, baselines, costs and calibration

29 min read·
Module 3 · Lesson 3 · About 55 minutes · Needs: pandas, what a classifier and a train/test split are, and the Kestrel data (course-guide/06). Lessons 1 and 2 teach the models; here they are trained in a few lines.

The situation

Week 3 at Kestrel. Adaeze forwards the spring prototype: "It scored brilliantly. Find out why before anyone trusts it." Morag wants a list, at 06:00, of the parcels worth a warning, and what that list saves in pounds.

One parcel runs through the lesson: KP0104204 (simulated), collected from Dartford on Wednesday 2 September 2026 at 18:00, next-day service on route DAR-R28, 143 stops, a poor address. It arrived a day late, after two attempts.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson builds Kestrel's features and the honest training path (blue); the rest of the system is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan