Lesson 2: Attention, and words that change with their neighbours

Module 7 · Lesson 2 · About 55 minutes · Needs: basic Python and NumPy; Module 6 (word vectors, cosine similarity) helps; Lesson 1 of this module helps but is not required

The situation

It is Monday 19 October 2026, project week 7. Rosa asks whether a transformer would route the inbox better than Module 4's TF-IDF model. Before fine-tuning one in Lesson 4, you need to know what is inside it.

One message runs through the lesson: HT35 from the hand-written test set (simulated), "Charged twice. £45 and £45. Fix it.", labelled returns_refund. Beside it sits a battery twin, added in this lesson: "My battery charged to 80% then died." To a word-count model, both contain the same word.