Part 58 · Module 8: Extraction, Question Answering and Summarisation

Incident: The answer finder answered the wrong question

32 min read·14 Oct 2026

Module 8 · Incident file · About 30 minutes

What happened

The reported case in order: a televised match (blue), a US-cities category, an answer of the wrong type (red), a small wager and a win (amber).

In February 2011 IBM's question-answering system, Watson, played Jeopardy! on television against two of the show's champions.

In Final Jeopardy the category was "U.S. Cities". Watson answered "What is Toronto?????". Toronto is not a US city. Its confidence in that answer was low, and its wager was small.

Watson still won the match. Afterwards IBM's team explained that the category name had been given little weight as evidence. The system had ranked first a candidate of the wrong type.

The wager kept the cost small. The lesson is general: a question-answering system can rank first an answer that is not even the kind of thing the question asks for. Tidewell's answer finder from Lesson 2 does exactly this.

What happened

The rest of this part opens with an account