CourseAI / ML Engineer: The Complete Course · Module 6: Experimentation · part 39 of 93
Part 39 · Module 6: Experimentation

Lesson 1: Tracked experiments and honest comparisons

28 min read·
Module 6 · Lesson 1 · About 55 minutes · Needs: the time split, PR-AUC and search strategies (Module 3 lesson 3), seeds (Module 4 lesson 4), Git basics (Module 2 lesson 1) and the Kestrel data (course-guide/06).

The situation

Week 6 at Kestrel. Adaeze asks: "Can you reproduce last month's model exactly?" Nobody could rerun the spring prototype. Morag will pay for model v2 only if it beats v1 for sure, not on one lucky run.

Two things run through the lesson: a tracked run called v1 (Module 3's untuned LightGBM settings, seed 0), and parcel KP0104204 (simulated), collected at Dartford on 2 September 2026 and delivered late. v1 gives it 0.67; the tuned v2 gives it 0.748.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson adds tracked experiments to Kestrel's training stage (blue); the rest of the system is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan