Lesson 1: Measuring retrieval before you measure answers

Module 11 · Lesson 1 · About 50 minutes · Needs: Module 4 Lesson 4 (recall@k, MRR, the scorer in evals/retrieval.py) and Module 6 Lesson 2 (the hybrid search)

The situation

On Friday 9 October 2026 Grace asks the question every module has been building towards: is Hollin Answers good enough for her agents' pilot on 2 November? The Module 6 lab shipped hybrid search as "not worse, likely better" and left the proof to this module.

Take q05, "how long until I get my appliance back from repair"; the prototype's log has it as "repair turnaround time". A full answer needs 10 working days, from the repairs policy, and 3 extra days for repairs booked 1 to 16 October, from the repair centre notice as updated on 8 October (this lesson searches the 2 October export, where it still says 2; the module lab grades each answer against the corpus its run searched). This lesson measures retrieval on its own, then learns to say which stage broke a bad answer. The Hollin data is simulated.