Lesson 4: Model judges and fair experiments

Module 11 · Lesson 4 · About 55 minutes · Needs: Module 9 (the answer contract), Module 8 (the reranker), and hit rate from Lesson 1 of this module

The situation

Grace's pilot on Monday 2 November depends on the Module 11 gate. Two claims land on Friday 9 October. Kwame reports that switching off the reranker and lowering the hybrid weight "made search faster and no worse". Hannah wants every answer graded before the pilot, and there are too many for people to read.

This lesson builds what both need: a model that grades answers, checked against Grace's labels, and experiments that change one thing at a time and know their noise. The running example is Ellie's "Is it covered?" and the fact behind it: X200 motors are covered for 5 years. The Hollin data is simulated.