CourseAI / ML Engineer: The Complete Course · Module 16: Capstone · part 93 of 93
Part 93 · Module 16: Capstone

Lab: One model that keeps working without you

227 min read·
Module 16 · Capstone · About 90 minutes · Needs: Modules 1 to 15 (or nothing: Step 1 rebuilds them)

What you are building

The Kestrel system One model in production, end to end Active flow Background or async 1 Processing step number Data pipeline Prefect Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Storage Data lake Parquet, DuckDB Training LightGBM Experiments MLflow runs Training LightGBM, Optuna Features Feast Online store Feast, Redis Features point in time Gates and registry MLflow Gates vs the champion Registry staged versions Serving FastAPI Serving API FastAPI, ONNX Consumers vans, customers Automation Retraining trigger, budget Monitoring Evidently Alerting on call Monitoring drift, Evidently Late labels a day later Shared platform Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images 1 2 3 4 5 6 7 8
The whole Kestrel system the capstone runs: raw scans through the contract into training, five gates and the registry, the ONNX service with shadow and canary flags, monitoring on late labels, and budgeted retraining, all rebuilt by make all from one commit (circles give the order).

In 2017 Google described TFX, the platform behind its continuously retrained models. Its authors wrote that "model validation must be coupled with data validation in order to detect corrupted training data and thus prevent bad (yet, validated) models from reaching production", and that a new model "failing any of these checks is not pushed to serving" (Baylor et al., "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform", KDD, August 2017, sections 2.2 and 5.3). In its Google Play case study, comparing serving logs with training data found features "always missing from the logs, but always present in training"; removing that skew raised installs on the main landing page by 2% (section 7). Uber's version of the same discipline opened Module 10 lesson 2, and Michelangelo opened Module 9 lesson 1.

The capstone builds Kestrel's version end to end and then answers the roadmap's five questions in docs/five-answers.md, every number from this notebook. No cell calls a hosted model.

Which model runs. The Module 10 lab's replay ended with registry version 4, the 21 September retrain, as champion, while the Module 11 lab monitored model v2, the model that served September. This capstone runs v2 end to end: it served September, it has the model card and ADR-010's scope, and it is "last month's model" in question 5. Retrains are registered as challengers that wait for ADR-007's shadow and ADR-010's island check (Step 7).

Open a new Colab notebook, call it module-16-capstone.ipynb, and run each cell in order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan