An ensemble beat SuperGLUE’s human score before DeBERTa, the model credited with doing it first

The number passed the humans again

SuperGLUE ranks language models with one number: the equal-weighted average of their scores on eight separate English-language tasks, sitting on a public leaderboard beside a row labeled Human.¹ Some of those tasks are graded by plain accuracy, others by F1 or exact match, and the leaderboard folds all of it into one decimal.¹ The Human row is not a fixed ceiling. It is what a set of hired annotators scored on the same eight tasks, under a protocol SuperGLUE’s own authors published alongside the number.¹

On 6 January 2021, a single DeBERTa model scored 89.9 against that Human row’s 89.8, and its authors called this the first time a single model had surpassed human performance on SuperGLUE, in terms of macro-average score, their term for the same equal-weighted average described above.² Microsoft, which built the model, announced the result the same day.³ A headline that ran that afternoon put it more plainly: AI models had already surpassed human performance on the benchmark.⁴ By the week’s end, the framing had simplified further, into a new model having surpassed human baseline performance on language understanding, full stop.⁵

How eight tests become one number

Start with what SuperGLUE actually bundles: eight tasks, each testing something different, scored in three different kinds of measurement. Five are graded by plain accuracy. The other three carry two metrics apiece, an F1 score alongside either exact match or a second accuracy figure, averaged together before that task even enters the eight-task average.¹ SuperGLUE’s own paper states the rule for combining them as plainly as it sounds: “weighing each task equally, and for tasks with multiple metrics, first averaging those metrics to get a task score.”¹

To see why that choice matters, invent two tasks of your own, honestly invented and not from SuperGLUE itself. Task A is scored by plain accuracy: a model gets 90. Task B carries two metrics, and the same model scores 70 on one and 50 on the other. Average Task B’s two numbers first, the way SuperGLUE’s rule does, and it earns a task score of 60. Average that against Task A’s 90, and the two-task leaderboard reads 75. Skip that first averaging step and combine all three raw numbers directly instead, 90, 70, and 50, and the same model’s average comes out to 70. Two invented tasks, one model, and a five-point swing that traces entirely to which averaging rule got picked, before either task result changed at all.

SuperGLUE did not invent this rule. Several of its authors had also built GLUE two years earlier and carried over the same equal-weight recipe.⁶ They built SuperGLUE because GLUE had run out of room to measure progress: models had already passed non-expert human performance on it, leaving little headroom for further research.¹

The human row is a measurement too

The 89.8 in the Human row came from Amazon Mechanical Turk. For each of the eight tasks, SuperGLUE’s authors gave hired crowd workers instructions and up to 30 practice items from the task’s own development set, then collected five workers’ answers per item and took the majority vote as human performance.¹ The average pay across every task and both stages was $23.75 an hour.¹

One label, 89.8, sits over that whole spread. It is real data, but it is a hired, briefly trained crowd’s data, not a fixed fact about what people can do on these eight tasks.

A high score on any one of these tasks also does not rule out a shortcut in the data doing the work instead of the model. The earlier lesson on GLUE, this benchmark’s predecessor, shows exactly how that can happen: a task score can rise without the model ever weighing the sentence the task was built to test. SuperGLUE’s eight tasks carry the same risk, and a single averaged number cannot show whether any one of them is being lifted that way.

The line DeBERTa crossed had already been crossed

By late 2020, the leaderboard’s top model was Google’s T5, an 11-billion-parameter model that scored 89.3, half a point under the human row.⁷ The gap SuperGLUE had opened with at launch, a nearly 20-point spread between the best model and human performance, had by then closed to less than one point.⁴

On 6 January 2021, Microsoft announced that a single DeBERTa model, 1.5 billion parameters against T5’s 11 billion, had reached 89.9.³ A DeBERTa ensemble, several DeBERTa models combined, scored 90.3 and topped the leaderboard outright.²³

The single-model qualifier in DeBERTa’s own claim is not decoration. DeBERTa’s own results table lists a Google entry, an ensemble called T5 + Meena, at 90.2, already above the human row by the time that table was published.² A report filed the same day named both: DeBERTa from Microsoft and, separately, T5 + Meena from Google, as having surpassed the human baselines.⁴ DeBERTa was the first single model over the line. It was not the first system to cross it, and its own authors never claimed otherwise.

None of this comes from watching the leaderboard change in real time. The page is a client-side application, and an automated fetch of it returns only an HTML shell, not the rendered rankings.⁸ The scores above are pinned instead to DeBERTa’s own results table, Microsoft’s reproduction of the leaderboard dated 6 January 2021, and two reports filed the same day.²³⁴⁵

What the builders said the same week

The people who produced the 89.9 said themselves what it did not mean. The paper that reported the result capped its own claim in the same breath.

Microsoft’s own announcement of the result carried the same line.³ Sam Bowman, a co-author of both GLUE and SuperGLUE, went further the same week.

The report carrying Bowman’s caution ran the same day as one that did not: a second outlet repeated the crossing, “for the first time, a new model surpassed human baseline performance,” with no caveat in its body at all.⁵ The claim that circulated without qualification did not come from the people who built DeBERTa or the people who built SuperGLUE. Both of those groups capped their own claim in writing, the same week they made it. It came from the step after: coverage that dropped the qualifier the builders had kept in.