CourseNatural Language Processing: The Complete Course · Module 2: Tokenisation and Word Normalisation · part 18 of 73
Part 18 · Module 2: Tokenisation and Word Normalisation

Incident: Tokens the model never learned

43 min read·14 Oct 2026
Module 2 · Incident file · About 30 minutes

What happened

THE LIKELY CAUSE (RUMBELOW AND WATKINS, LESSWRONG, FEBRUARY 2023) 1. tokeniser built on text A 2. string common in A: own token 3. model trained on text B 4. rare in B: row barely learned 5. user types the string 6. model cannot repeat it
The tokeniser learned the string from its text (blue); the model, trained on other text (amber), barely saw it, so it failed on it (red).

In February 2023 Jessica Rumbelow and Matthew Watkins published "SolidGoldMagikarp (plus, prompt generation)" on LessWrong. They had found odd entries in the vocabulary of the tokeniser used by GPT-2 and GPT-3: whole-word tokens such as " SolidGoldMagikarp", which is a Reddit username. Asked to repeat these strings back, the models could not, and they answered strangely about them.

The likely cause, as the write-up explained it, sits between two sets of text. The tokeniser was built on text in which these strings were common, so each earned a token of its own. The model was then trained on text in which they were rare, so it learned almost nothing about those tokens. The tokeniser handed the model pieces it had barely seen.

Lesson 2 of this module trained a subword tokeniser and warned that a tokeniser ships with its model. This case is the other half of that warning: a tokeniser and a model must also be built from the same text.

The rest of this part opens with an account

Signing up is free and takes one email. Nothing to pay, and nothing renews.

Start reading free