Incident: A classifier that learned the wrong word

Module 4 · Incident file · About 35 minutes

What happened

In 2019 Timothy Niven and Hung-Yu Kao published "Probing Neural Network Comprehension of Natural Language Arguments" at ACL. The task was the Argument Reasoning Comprehension Task (ARCT): given an argument, pick which of two "warrants" makes it hold. It needs world knowledge, and in the 2018 SemEval shared task "all but one participating system... could not exceed 60% accuracy". Then BERT, the model Module 6 opens with, scored 77 percent on its best run, "only three points below the average (untrained) human baseline" of 79.8 percent.

The authors did not celebrate: "it does not seem reasonable to expect it to perform so well." Looking at the data points BERT found easiest, they found it "exploits the presence of cue words in the warrant, especially 'not.'" Across the dataset, "if you pick the warrant with 'not' you will be right 61% of the time, which covers 64% of all data points." Trained on the warrants alone, with no argument to read, BERT still reached 71 percent. They then built an adversarial set in which every claim also appears negated with the label flipped, so the cue is equally present on both labels. "On this adversarial dataset all models perform randomly, with BERT achieving a maximum test set accuracy of 53%." Their conclusion: "BERT has learned nothing about argument comprehension."