Incident: Accurate in every test, reading the wrong thing

Module 14 · Incident file · About 35 minutes

What happened

On 31 May 2021 Alex J. DeGrave, Joseph D. Janizek and Su-In Lee published "AI for radiographic COVID-19 detection selects shortcuts over signal" in Nature Machine Intelligence (volume 3, pages 610 to 619; received 25 September 2020; nature.com).

Researchers had reported AI systems that detect COVID-19 in chest radiographs accurately. The authors rebuilt such systems and found that they "rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals."

They trained models on one of two datasets and tested on held-out images from the same sources (internal) and on images from a new hospital system (external). The models scored well internally, but "half of the models' predictive performance is lost when testing on dataset II". The gap held for all nine further architectures they tried, including ones built for this task.

The instruments then showed why. Saliency maps "frequently highlight laterality markers", the letters placed on a radiograph during acquisition, whose style differed between the COVID-19-positive and negative sources. The authors' conclusion: "explainable AI should be seen as a prerequisite to clinical deployment".