CourseAI / ML Engineer: The Complete Course · Module 3: Classical Machine Learning · part 23 of 93
Part 23 · Module 3: Classical Machine Learning

Incident: The model learned that asthma lowers risk

15 min read·
Module 3 · Incident file · About 35 minutes · Needs: Lesson 1 (logistic regression and its weights) and Lesson 3 (leakage and the prediction point)

What happened

WHAT HAPPENED From a 1990s study to a 2015 paper Mid 1990s pneumonia risk project Neural nets AUC 0.86, most accurate A rule found asthma lowers risk Decision logistic regression used Aug 2015 Caruana et al., KDD
The case in order: the most accurate model (amber) was set aside after a readable model exposed the asthma rule (red), and a model people could read was used instead (green).

In the mid 1990s a large multi-institutional project, funded by Cost-Effective HealthCare, built models to predict the probability of death for patients with pneumonia. The aim was to send high-risk patients to hospital and treat low-risk patients as outpatients. Rich Caruana and colleagues retell what happened in "Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission" (KDD 2015, Sydney, 10 to 13 August 2015).

The most accurate models were multitask neural nets. On one dataset "the neural net had AUC=0.86 compared to 0.77 for logistic regression". Yet "after careful consideration they were considered too risky for use on real patients", and logistic regression was used instead.

WHAT HAPPENED A true pattern, a dangerous rule Asthma and pneumonia Straight to ICU aggressive care Lower death rate in the data Model: asthma lowers risk send asthmatics home?
The data was right: asthmatic patients went to intensive care and died less often (green); the model turned that into 'asthma lowers risk' (red), which would send the wrong patients home.

The reason was a rule. A rule-based model, less accurate but readable, learned "HasAsthama(x) ⇒ LowerRisk(x)" (spelled so in the paper): pneumonia patients with a history of asthma had a lower risk of dying than the general population. It "reflected a true pattern in the training data". Asthmatic patients with pneumonia "usually were admitted not only to the hospital but directly to the ICU", and that care "was so effective that it lowered their risk".

So "models trained on the data incorrectly learn that asthma lowers risk, when in fact asthmatics have much higher risk (if not hospitalized)". Used to decide who goes to hospital, such a model would send home the patients who most need care. The authors reason that if the rules had learned it, "certainly the neural nets had learned it, too", and nobody could see what else the nets had learned.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan