CourseAI / ML Engineer: The Complete Course · Module 8: Model Optimisation · part 52 of 93
Part 52 · Module 8: Model Optimisation

Lesson 3: Measuring: percentiles, cold starts and cost per thousand

26 min read·
Module 8 · Lesson 3 · About 50 minutes · Needs: lessons 1 and 2 (the network, the trees, ONNX Runtime), percentiles (Module 1 lesson 3), and the Kestrel data (course-guide/06).

The situation

Week 8 at Kestrel, Wednesday 23 December 2026. ADR-005 must choose the inference runtime, and two people want numbers. Rhys: "What is p99, and where does it go?" Imran: "What does one prediction cost?" Lesson 2's timings moved between runs by tens of per cent, so first you need a benchmark that does not mislead.

The running example is KP0104204 (simulated), scored alone as it would be at collection, and inside the 06:00 batch.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
Measuring feeds the promotion gates (p99 in the benchmark) and the serving API (blue); the rest of Kestrel's system is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan