Machines cleared SQuAD's human exact-match score without learning to read

The claim that machines out-read people

By late January 2018, the leaderboard for SQuAD, the Stanford Question Answering Dataset, showed several reading systems sitting above the row labeled Human Performance.¹ An ensemble from Microsoft scored 82.650 and one from Alibaba 82.440, against a human 82.304.¹ The coverage did not hedge: Microsoft and Alibaba, one headline ran, had beaten humans in a Stanford reading-comprehension test for the first time.²

The systems really did clear that number. What the headlines made of it was another matter. SQuAD reports two scores, and the January systems had passed the human line on only one of them. And the human line was itself a measured number, produced in a way worth knowing before the comparison means anything.

Every answer is a span from the passage

SQuAD 1.0, released in 2016 by Pranav Rajpurkar and colleagues at Stanford, is a set of 107,785 questions written by crowdworkers about 536 Wikipedia articles, drawn from 23,215 paragraphs.³ Its defining constraint is where the answers live. Every answer is a span: a run of consecutive words copied directly from the passage, which a crowdworker highlighted.³ Ask "Which team won?" and the answer is not the word "won" rephrased but the exact stretch of text, say the Denver Broncos, sitting in the paragraph.

A system's job, then, is not to write an answer but to point at one: given a passage and a question, select the start and end of the span that answers it. That single design choice, answers as copied spans, is what makes the two SQuAD scores easy to compute and, later, easy to game.

Two scores for one copied span

Because both the model's answer and the reference answer are short strings, SQuAD scores them by comparing strings, and it does this two ways.³ Before either comparison, each string is normalized: lowercased, stripped of punctuation, and stripped of the words a, an, and the, so that "the Broncos" and "Broncos" count as the same answer.⁴

The first score is exact match, written EM: the percentage of questions where the model's normalized answer is identical, character for character, to one of the crowd's reference answers.³ It is all or nothing. "Denver Broncos" against a reference of "the Denver Broncos" earns a full point, because normalization removed the the; "Denver Broncos defense" earns zero, because one extra word makes the strings differ.

Exact match is strict, and most right-but-not-identical answers fail it, so SQuAD reports a second, more forgiving score: F1, which measures overlap in words rather than demanding an identical string.³ Treat each answer as a bag of words. Precision is the share of the model's words that appear in the reference. Recall is the share of the reference's words the model recovered. F1 is the harmonic mean of the two, a single number that is high only when both precision and recall are high.⁴

Take the model answer "Denver Broncos defense" against the reference "the Denver Broncos." After cleanup the reference is two words, denver and broncos; the guess is three, denver broncos defense. Two words are shared. The model's precision is two of its three words; its recall is two of the two reference words. Their harmonic mean is 0.80. The same answer that scored zero on exact match scores 0.80 on F1.

For a question with several reference answers, SQuAD takes the best score against any one of them, then averages over all questions.³ A model's reported EM and F1 are those two averages across the test set. In the original 2016 paper a plain logistic-regression baseline reached 40.4 EM and 51.0 F1; the humans, measured the same way, reached 77.0 and 86.8.³

Which score the systems actually crossed

The human line deserves the same scrutiny as the model scores, because it was produced by the same procedure, not by watching people read. For each question in the evaluation set, the SQuAD team collected at least two extra crowdworker answers beyond the original, giving each question several.³ They then took a single one of those answers, the second worker's, treated it as if it were a model's prediction, and scored it against the remaining answers.³ The human number is therefore one crowdworker's single answer, graded the same way a system is graded. It is not a panel, not a vote, and not an expert. On the 2016 test set that procedure gave 77.0 EM and 86.8 F1.³

The public leaderboard ran on a lightly revised release, SQuAD 1.1, where the same one-answer procedure put the human line at 82.304 EM and 91.221 F1.¹ This is the line the January 2018 systems crossed. Alibaba reached the top of the leaderboard on January 11 and Microsoft a day later.⁵ Their exact-match scores, 82.650 for Microsoft and 82.440 for Alibaba, sat just above the human 82.304.¹ Their F1 scores did not. Microsoft's F1 was 88.493 and Alibaba's 88.607, and the best F1 anywhere on the board, 89.281, still trailed the human 91.221 by nearly two points.¹

So the crossing was on exact match only. Stanford's own announcement said as much, calling the Alibaba result the first to exceed human-level performance on the exact-match metric and noting that on F1 humans still led by about two and a half points.² Alibaba's own paper claimed only the top leaderboard position for single and ensemble models, not a win over humans.⁶ The qualification existed. Most of the coverage dropped it, and "beat humans at exact match" became "beat humans at reading."

Even the exact-match line was soft. The SQuAD 2.0 authors later wrote that the original human figure "evaluated a single human's performance" and so likely underestimated what people can do.⁷ A machine edging past a low, one-person bar by a few tenths of a point is a thinner result than a headline about beating humans suggests.

One extra sentence collapsed the scores

There was already direct evidence that the top systems were not reading in the sense the headlines implied. In 2017 Robin Jia and Percy Liang tested whether SQuAD models understood their passages by appending a single distracting sentence to each one.⁸ The added sentence was built to look relevant to the question while leaving the real answer unchanged, and a human reader was not fooled by it.⁸ Across sixteen published systems, average F1 fell from 75.4 to 36.4 once that one sentence was present.⁸ In 96.6 percent of the cases where a model failed, it had moved its answer into the inserted sentence.⁸

That is the tell. A system that comprehended the paragraph would ignore an irrelevant add-on. A system matching surface patterns near the question's words would be pulled toward any new text that shared those words, and the models did the second thing. They had learned to find the span that looked like an answer, which on a clean SQuAD passage is usually the right span, and which stops working the moment a decoy is present. It is the reading equivalent of scoring well by pointing at the likeliest-looking phrase without following the sentence it sits in.

The effect was not limitless, and Jia and Liang said so. Human F1 also dropped on the attack, by about 13 points in its harshest form, though only about 3 points on a milder version that did not tune the sentence against a specific model.⁸ The gap between human and machine robustness was large, not infinite.

The deeper problem was structural. On SQuAD 1.0 every question had an answer somewhere in the passage, so a model that always emitted its best-guess span was never punished for failing to say "no answer here." In 2018 the same team released SQuAD 2.0, adding 53,775 unanswerable questions written to look answerable, each with a plausible wrong span sitting in the paragraph.⁷ Now a model had to know when to abstain, and picking the closest span scored zero.⁷ The gap reopened at once: the best system scored 66.3 F1 against a human 89.5, a 23.2-point difference, where the same kind of model had trailed humans by about five points on 1.0.⁷

None of this means the SQuAD number was empty. Within the year, BERT reached 93.2 F1 on SQuAD 1.1, clearing the 91.2 human F1 line for real, and it lifted scores on the harder 2.0 as well.⁹ The benchmark did track genuine progress in reading systems. What failed in January 2018 was narrower and datable. A few-tenths lead on exact match, measured over a one-person baseline on questions that always had an answer, was read as the arrival of machine reading comprehension.