Lesson 4: A model as judge, checked against people
26 min read·13 Oct 2026
Module 10 · Lesson 4 · About 50 minutes · Needs: Module 10 Lesson 3 (rubric v2, Lena's and Ravi's ratings, kappa); Module 6 Lesson 1 helps (structured output)
The situation
It is Friday 14 August 2026, week 11. Lena and Ravi can double-rate 20 drafts a week. The assistant writes about 1,800.
Maya asks: "Can a model read the drafts we never see, and how would we know it reads them as we do?" Joel asks what it will cost. You will turn rubric v2 into a judge and measure it against the 20 drafts Lena and Ravi rated. The running example is again the T-1001 draft for Ana Duarte's duplicate charge of 288 USD, which Maya labelled "edit" under rule R4. Drafts and labels are simulated, as in Lesson 3.