Lesson 3: Classical models against LLMs, measured fairly

Module 10 · Lesson 3 · About 55 minutes · Needs: Module 5 Lessons 1 and 2 (macro F1, bootstrap intervals, abstaining); Lesson 1 of this module helps, but this lesson builds its own LLM classifier

The situation

It is Monday 9 November 2026, project week 10. Rosa has read about large language models and asks the obvious question: "Why not send every message to one and stop maintaining the classifier?" Amara adds her condition: only redacted text leaves the building. Ines wants to know whether the answer would change her queues.

One message runs through the lesson: HT29 from the hand-written test set (simulated). "We are a logistics company offering cheap shipping from overseas factories. Minimum order 500 units. Message us for prices." Ines labels it spam. The rule router sends it to order_status; Module 5's logistic regression sends it to returns_refund, with a confidence of only 0.23.