Lesson 3: Human evaluation: two raters, one rubric, one number
18 min read·13 Oct 2026
Module 10 · Lesson 3 · About 45 minutes · Needs: Module 4 Lesson 2 (labels from written definitions); Module 10 Lesson 1 helps (deterministic checks)
The situation
It is Thursday 13 August 2026, week 11. Two days ago Ravi approved a German draft that promised Brandt Logistik a full refund it was not due. Joel stopped the money at the approval step, but the promise had been sent.
Maya asks: "If one of us can miss that, how far can we trust any of our ratings?" In week 10 Lena and Ravi both rated the same 20 drafts, blind. You will measure their agreement, settle their disagreements and write a labelled set. The running example is the draft for T-1001, Ana Duarte's duplicate charge of 288 USD: Lena rated it "edit", Ravi "send". All ratings and drafts here are added in this lesson and simulated.