The ‘human-level’ language score is an average of nine English tests

When the number passed the humans

On 6 June 2019, a Microsoft model called MT-DNN scored 87.6 on GLUE, a public leaderboard for English, and passed the 87.1 that a group of human annotators had scored on the same tests.¹ Nineteen months later, on 6 January 2021, a model called DeBERTa scored 89.9 on GLUE’s harder successor, SuperGLUE, and passed the human mark of 89.8 there too.² In the coverage that followed, one line kept recurring: machines had reached human-level understanding of language. Pretty good, if true.

A single headline score can hide how it was produced, the way an MMLU score hides how it was measured. GLUE hides something more specific: what got averaged. The number that crossed each human line is not one measurement. It is an average of nine.

What the score averages

GLUE, introduced by Wang and colleagues in 2018, bundles nine separate English tasks under one leaderboard.³ They do not test the same skill, and they are not even scored in the same units.

Some tasks are scored in plain accuracy, one (CoLA) in a correlation coefficient, one (STS-B) in two more, and three pair accuracy with F1. To reach the headline number, each task is scored in its own unit, the paired-metric tasks are averaged internally first, and the nine results are then averaged together with equal weight.³ A task with 634 training examples counts exactly as much as one with 393,000.

The human baseline is a measurement, not a fixed ceiling. Nangia and Bowman built it in 2019 by hiring non-expert crowd workers, giving each brief instructions and about twenty example items, and scoring their majority answers on the same test sets.⁴ Their average was 87.1, and it beat the best models of the moment on six of the nine tasks.⁴ The number labeled “human” on the leaderboard is that: a quick estimate from untrained annotators, published in late May 2019 and passed by MT-DNN about two weeks later.¹

When models saturated GLUE, its authors did not declare language solved. They read the shrinking headroom as a reason to raise the bar, and in 2019 they released SuperGLUE: eight harder tasks, the same averaging recipe, and a fresh human baseline of 89.8.⁵ The per-task human scores folded into that 89.8 ranged from 80.0 on one task to a perfect 100.0 on another.⁵

The task that sat at chance

Return to that first crossing. MT-DNN reached 87.6, just above the human 87.1. Averaged across nine tasks, the model looked essentially human. Did it read English like one?

On one of the nine tasks, WNLI, the answer is no. WNLI has two possible answers, and always guessing the more common one scores 65.1. Before MT-DNN, the strongest models, BERT included, could not beat that majority-vote floor. Human annotators scored 95.9.¹ The task was a wall. MT-DNN cleared the human line overall only after a special method lifted its WNLI score to 89.0.¹ Fix the one task the model was failing, and the nine-task average steps over the human mark.

This is the first thing the aggregate hides. A model can sit at human level on the average while performing no better than guessing on one of the nine tasks inside it. The single number reports the middle and says nothing about the spread.

Answers that leak into the tasks

Even where a model beats the floor, a high score can come from a shortcut rather than comprehension. Take natural language inference, the format behind several GLUE tasks: the model is shown two sentences, a premise and a hypothesis, and must say whether the first implies the second, contradicts it, or neither, one label out of three.⁶ In 2018, Gururangan and colleagues ran a control. They hid the premise and showed a classifier only the hypothesis. Guessing blindly among three labels scores about 33%. The hypothesis-only model instead scored about 67% on one standard inference set and about 53% on MultiNLI, the set GLUE folds in as MNLI.⁶ The people who wrote the data had left tells. Negation words tracked contradiction. Vague hypotheses tracked entailment. A model can learn those tells and never weigh the premise at all.

A shortcut in the data does not mean every high scorer used it. On COPA, a SuperGLUE task where the model picks the more plausible cause or effect, a baseline reading only surface cues in the answer options scores 59.6 against 50 for chance.⁷ The cues are real. But whether a model leans on them varies by the model.

BERT had leaned on the cues, and lost ground when they went away. RoBERTa had not, and if anything did better on the cleaned task.⁷ A high GLUE or SuperGLUE score is consistent with shortcut exploitation, and does not by itself rule it out.

Who called it understanding

So where did “machines understand language” come from? Not from the people who built the benchmarks or topped them. GLUE’s authors treated saturation as a reason to build SuperGLUE.⁵ When DeBERTa passed SuperGLUE’s human baseline, the team behind it said plainly what the score did not show.

The overreach lived in the shorthand between the leaderboard and the headline. “Surpassed the human baseline” became “reached human-level understanding,” and the two are not the same claim. Bender and Koller made the underlying point in 2020: a system trained only on the form of language has, in principle, no path from form to meaning, so clearing a form-based benchmark is not evidence of comprehension.⁸ They named the reading directly, warning against hype in which these models “are being described as ‘understanding’ language.”⁸ Saturation is not comprehension.

The careful readers kept the claim narrow. A trade outlet reporting DeBERTa’s result repeated the same scores, 89.9 for the single model and 90.3 for an ensemble against the 89.8 human mark, and called it a macro-average on this benchmark rather than human comprehension.⁹ The number was accurate. The story stacked on top of it was the part that broke.