Incident: A chatbot that learned from its attackers

Module 13 · Incident file · About 30 minutes

What happened

On Wednesday 23 March 2016 Microsoft put a chatbot called Tay on Twitter. It was built for 18 to 24 year olds in the US, and, as TechCrunch's Sarah Perez reported the next day, it "learns" from the people it talks to (TechCrunch, 24 March 2016). Users taught it offensive content quickly, and it often repeated racist tweets back with its own commentary. Microsoft silenced it later that Wednesday, after 16 hours of chats.

A Microsoft spokesperson told TechCrunch: "within the first 24 hours of coming online, we became aware of a coordinated effort by some users to abuse Tay's commenting skills", and said Tay had been taken offline for adjustments.

On 25 March Peter Lee, a Microsoft corporate vice president, published "Learning from Tay's introduction" (Official Microsoft Blog). He apologised for "the unintended offensive and hurtful tweets from Tay". Before launch the team had added filtering, run user studies and stress-tested Tay. Even so, "a coordinated attack by a subset of people exploited a vulnerability in Tay". Microsoft took "full responsibility for not seeing this possibility ahead of time" and said Tay would return "only when we are confident we can better anticipate malicious intent." Microsoft did not publish the technical detail of the vulnerability, so this file does not guess at it.