CourseAI / ML Engineer: The Complete Course · Module 5: Data Engineering for ML · part 33 of 93
Part 33 · Module 5: Data Engineering for ML

Lesson 2: Where the data lives: columns, partitions and tables

24 min read·
Module 5 · Lesson 2 · About 50 minutes · Needs: lesson 1 (the nightly files), pandas, and a little SQL (SELECT, WHERE, GROUP BY).

The situation

Week 5, Wednesday. Lesson 1's flow lands a file per depot per night. Now Adaeze wants every parcel since 6 July in one place that training, monitoring and Morag's questions can all read. Morag's question this morning: "How many parcels did Dartford collect in the week of 7 September, and how many were late?" Tomasz warns that whatever you build has to survive a day being rewritten, because lesson 3 will correct September's weights.

That question runs through the lesson: Dartford, week of 7 September 2026 (simulated): 2,212 parcels, late rate 0.111. The point is not the answer but how little data you can read to get it.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson builds the data lake that the nightly ingestion writes into (blue); everything downstream reads from it, and the rest of Kestrel is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan