CourseAI / ML Engineer: The Complete Course · Module 5: Data Engineering for ML · part 32 of 93
Part 32 · Module 5: Data Engineering for ML

Lesson 1: Pipelines that can run twice: batch, stream and backfill

26 min read·
Module 5 · Lesson 1 · About 50 minutes · Needs: pandas, the Kestrel data (course-guide/06), and Module 3 lesson 3 for why columns known only after delivery stay out of training.

The situation

Week 5 at Kestrel. Every night each depot sends Tomasz a scan file: the parcels it collected that day. The spring prototype read one CSV that someone exported by hand. Adaeze wants a data path that runs every night with nobody touching it. Rhys asks the question that decides the design: "Some nights the Inverness link drops. When the job runs again, what happens to the data?"

One depot-day runs through the lesson: Dartford on Monday 7 September 2026, 435 parcels (simulated), the day scanner software v4 arrived. Lesson 3 catches what v4 did to weight_kg; this lesson makes sure the day lands once, and only once.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson builds the nightly ingestion from depot scans and the orchestrator that runs it (blue); the rest of Kestrel is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan