Lesson 3: Judging a classifier with the right numbers
17 min read·13 Oct 2026
Module 3 · Lesson 3 · About 40 minutes · Needs: Lesson 1 (train, test, cross-validation) and Lesson 2 (the ticket classifier)
The situation
Priya wants warranty tickets sent straight to the repairs team. A customer whose kettle has stopped heating should not queue behind "do you sell gift cards". Dev points the Lesson 2 classifier at one yes/no question, is this a warranty ticket, and reports: "85% accurate on tickets it never saw."
Priya asks one thing. "Out of ten warranty tickets, how many does it catch?" Dev does not know, and nobody has asked what a model that does nothing would score. This lesson answers both.