Incident: A suggested word guessed the gender

Module 7 · Incident file · About 30 minutes

What happened

In November 2018 Paresh Dave reported for Reuters, in "Fearful of bias, Google blocks gender-based pronouns from new AI tool", on Smart Compose, the Gmail feature that suggests how to finish a sentence as you type.

A Google research scientist typed "I am meeting an investor next week", and Smart Compose suggested "Do you want to meet him?". Nothing in the sentence said the investor was a man. The model had guessed.

Google's answer was not a better guess. It decided that Smart Compose would not suggest gendered pronouns at all, because a wrong guess could offend and could not be ruled out.

Tidewell's inbox writes to customers too, so the question for this module is direct: would a suggestion built on the DistilBERT this module fine-tuned make the same guess about Tidewell's customers?