CourseAI / ML Engineer: The Complete Course · Module 6: Experimentation · part 40 of 93
Part 40 · Module 6: Experimentation

Lesson 2: Discipline: one change, one locked test, one honest log

27 min read·
Module 6 · Lesson 2 · About 50 minutes · Needs: lesson 1 (v2, seed spread, paired tests), the time split and the test weeks (Module 3 lesson 3, course-guide/03), the grams fix (Module 5 lesson 3) and the Kestrel data (course-guide/06).

The situation

Week 6, Thursday. v2 is tuned, and two more ideas are on the table: more, slower trees, and dropping the depot inputs. Adaeze sets the rules: one change at a time, every result written down, and the test weeks looked at once, at the end. Morag asks why the islands still get so many surprise late parcels.

The running example is parcel KP0100502 (simulated): route INV-R09, shipped 31 August 2026, promised Wednesday 2 September, delivered late. v2 gave it 0.049. And the experiment log, docs/experiments/log.md.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson adds the locked test weeks, error analysis and the experiment log to Kestrel's training and experiments (blue); the rest is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan