Incident: A chatbot changed after it was tested

Module 11 · Incident file · About 30 minutes

What happened

The National Eating Disorders Association (NEDA), a US charity, offered a chatbot called Tessa. A team led by Ellen Fitzsimmons-Craft built it with NEDA funding, and the version they tested was rule-based: "all of the responses were pre-programmed", she told NPR's Kate Wells (Michigan Public via KUNC, 8 June 2023). Cass, a mental health chatbot company, ran Tessa as a free service.

According to Cass's founder, Michiel Rauws, changes were made in 2022 as part of a "systems upgrade", including an "enhanced question and answer feature" that uses generative AI and was, he said, part of NEDA's contract. NEDA's chief executive, Liz Thompson, said: "NEDA was never advised of these changes and did not and would not have approved them." In October 2022 the director of another eating disorders charity shared screenshots of Tessa telling her to avoid "unhealthy" foods.

In May 2023 NEDA was closing its human helpline. On a Monday, Sharon Maxwell, a consultant in the eating disorder field, posted that Tessa had given her weight-loss tips: losing 1 to 2 pounds a week and a daily calorie limit. NEDA took Tessa down "until further notice" on 30 May, less than 24 hours after she sent screenshots. Thompson told Global News the harmful advice reached "a small percentage" of the 2,500 people who had used the bot since its launch in February 2022, and blamed "bad actors" trying to trick it (Global News, 1 June 2023). The two parties' accounts of the change differ; this file relies only on what each published.