CourseAI / ML Engineer: The Complete Course · Module 9: Serving and Inference · part 56 of 93
Part 56 · Module 9: Serving and Inference

Lesson 1: Batch, online and streaming: three ways to score

26 min read·
Module 9 · Lesson 1 · About 55 minutes · Needs: Module 3 lesson 3 (the warning threshold), Module 8 lesson 3 (latency percentiles), and what an HTTP request is. The model is trained in one cell here, so the lesson runs on its own.

The situation

Week 9 at Kestrel. The model exists; nothing calls it yet. Morag needs every parcel due today scored before the 06:00 warnings. Rhys needs a score within his 50 ms p99 when a driver scans a parcel at collection. Tomasz asks whether the scan feed could be scored as it streams.

One parcel runs through the lesson: KP0104204 (simulated), collected from Dartford on Wednesday 2 September 2026 at 18:00, promised Thursday 3 September. Kestrel scores it online at collection and again in the 06:00 batch.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson builds Kestrel's scoring service and the paths to its consumers (blue); the rest of the system is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan