Incident: 62 models, none fit for clinical use

Module 6 · Incident file · About 35 minutes · Needs: Lesson 2 (the locked test set, test-set leakage, error analysis) and Module 3 lesson 3 (the time-based split)

What happened

In 2020 researchers rushed to build machine learning models that detect or predict COVID-19 from chest X-rays and CT scans. A team led by the University of Cambridge reviewed them. Its summary, "Machine learning models for diagnosing COVID-19 are not yet suitable for clinical use", appeared on 15 March 2021 (University of Cambridge research news, 15 March 2021); the paper is Roberts et al., Nature Machine Intelligence 3 (2021), DOI 10.1038/s42256-021-00307-0 (the paper page was not read for this file; what follows is from the university's summary).

The team searched papers published between 1 January and 3 October 2020. "Their search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in the systematic review." The finding: "None of the 62 models was of potential clinical use". Joint senior author Dr James Rudd: none of the literature reached "the threshold of robustness and reproducibility essential to support use in clinical practice".