CourseAI / ML Engineer: The Complete Course · Module 6: Experimentation · part 41 of 93
Part 41 · Module 6: Experimentation

Lab: Model v2, found and tested honestly

103 min read·
Module 6 · Lab · About 60 minutes · Needs: lessons 1 and 2 of this module, and the Module 5 lab (or nothing: Step 1 rebuilds Modules 1 to 5)

What you are building

Model v2, found and tested Every run recorded, one look at the test weeks Active flow 1 Processing step number Models model v1 Module 3 lab model v2 train_v2.py Search Optuna 20 trials v1's recipe paired tests five seeds Records MLflow runs commit, data log.md E-05, E-06 experiments.md generated Test weeks 7 to 21 Sep one look 1 2 3 4
The search (teal) tunes v1's own recipe into v2, the paired tests compare v2 with v1 over five seeds, every run lands in MLflow and the log with its commit (purple), and v2 alone takes the one look at the locked test weeks (gold); circles give the order.

Adaeze's question for this module is the fifth of the course's five: "Can you reproduce last month's model exactly?" Before Kestrel can answer it, every run has to leave a record, and before anyone calls a new model better, the claim has to survive a fair comparison. The lessons built both halves. This lab uses them on the real thing: model v1 from the Module 3 lab (kestrel/train_v1.py, a tuned LightGBM with Platt scaling), and a challenger, model v2, found by a tracked search.

The lab adds kestrel/train_v2.py (the recipe, the search, the model and three helpers) and reports/experiments.md, generated from the MLflow store. It makes the repository a Git repository, so every run records the commit it came from, writes three entries in the experiment log, one of them negative, and spends the locked test weeks on their one look. No cell calls a hosted model.

Open a new Colab notebook, call it module-06-lab.ipynb, and run each cell in order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan