Lab: A tokenisation and normalisation report for Rosa

Module 2 · Lab · About 60 minutes · Needs: lessons 1 to 3 of this module, and the Module 1 lab (or its cells, pasted here)

What you are building

Rosa's question this week is short: does any of this tokenising and normalising help the inbox, or is it busywork? The lessons gave three answers in three notebooks, each on its own slice of data. This lab puts the three lessons' code into one file, inbox/tokens.py, with its tests, and then writes one report on the same messages: how big the vocabulary is under each option, how many words of the 80 hand-written HARD_TEST messages are unknown, and what each option does to Module 1's keyword router.

Open a new Colab notebook, call it module-02-lab.ipynb, and run each cell in order.