Lesson 5: Scores that lie, and how to catch them
19 min read·13 Oct 2026
Module 5 · Lesson 5 · About 40 minutes · Needs: Module 3 Lesson 3 (precision and recall) and Module 5 Lesson 2 (sentence embeddings)
The situation
Priya has three months of real tickets, and the mix is not what the forty labelled tickets suggested: warranty questions are one in eleven, not one in four. Dev reruns the Lesson 3 classifier on a sample with that mix and reports 76 percent accuracy. Priya replies that the repairs team has not received a single ticket from it all week.
She also wants a one-line summary of each help-centre article, and asks how you will know the summaries are good. Both questions are about a score that looks fine and is not.