Lesson 5: Serving a model: batches, speed and memory
16 min read·13 Oct 2026
Module 6 · Lesson 5 · About 40 minutes · Needs: Lesson 1 (attention and the KV cache), Lesson 3 (precision and quantisation)
The situation
Sam, the security lead, does not want ticket text leaving the building, so Dev proposes running an open model on a rented GPU. Priya sends sixteen tickets at once and asks how long they will take. Dev asks: "Will the model fit on a 16 GB card, and how many tickets a second will it answer?" Marta adds: "One server or four?"
Inference is running a trained model to get its output. These questions are about how requests are grouped, how long each waits, and how much memory the model and its cache take. distilgpt2 on a CPU stands in for the model Dev wants.