CourseNatural Language Processing: The Complete Course · Module 4: Text as Vectors and Classification · part 31 of 73
Part 31 · Module 4: Text as Vectors and Classification

Incident: A word that says who wrote it

42 min read·14 Oct 2026
Module 4 · Incident file · About 30 minutes

What happened

past CVs mostly men's "women's" marked down terms ignored project dropped Amazon's recruiting tool, as Reuters reported in October 2018 about ten years of past CVs as the training data
A model trained on a skewed history learned to mark down a word (red); editing out the words (amber) was not enough, and the project was dropped (grey).

In October 2018 Jeffrey Dastin of Reuters reported that Amazon had scrapped a secret recruiting tool that showed bias against women. According to Reuters, the tool scored job applicants' CVs, and it had been trained on about ten years of CVs submitted to the company, most of them from men.

The model learned that pattern. Reuters reported that it penalised CVs containing the word "women's", as in "women's chess club captain", and marked down graduates of two all-women's colleges. Amazon edited the tool to ignore those terms, but could not be sure it would not find other ways to sort candidates that were just as unfair, and dropped the project. Amazon told Reuters the tool was never used by its recruiters to evaluate candidates.

Lesson 4 of this module trained the same kind of model: words in, weights learned from past examples, a score out. This file finds the same failure in Tidewell's intent classifier.

The rest of this part opens with an account

Signing up is free and takes one email. Nothing to pay, and nothing renews.

Start reading free