HellaSwag's right answer is just the sentence that came next

What a right answer looks like

HellaSwag is a multiple-choice test of sentence completion. Each item gives a short context and four ways to continue it. One is the real continuation and the other three are decoys, and a model earns a point for picking the real one.¹ Here is the first item in the public validation set, exactly as it ships.

The keyed answer, option 3, is not the wittiest ending or the one that reads best on its own. It is the sentence that actually came next in the source, a set of captions describing a video of a man removing roof shingles.¹ Telling it apart from three machine-written decoys is the whole task. A right answer on HellaSwag is recovering the real continuation, not showing a chain of reasoning about the world.

The released dataset is 59,950 of these items, split into 39,905 for training, 10,042 for a validation set, and 10,003 for a held-out test whose answer key is withheld.¹ The paper that introduced HellaSwag advertised a rounder "70k problems."² The released count is the one that can be checked against the shipped data.

How the four answers are built

The three decoys are not written by hand. HellaSwag manufactures them by a procedure its predecessor, a 2018 dataset called SWAG, named adversarial filtering. The idea is to over-generate wrong endings with a language model, then keep only the decoys a second machine cannot tell apart from the real ending.³ HellaSwag ran that procedure with two of the strongest language models of its moment: OpenAI's GPT to write the wrong endings, and BERT-Large, a system trained to classify text, as the filter.²

1. Start from something real

   Take a genuine context and its true next sentence from one of two text sources: captions describing videos in a dataset called ActivityNet, and how-to steps from WikiHow.²

2. Over-generate wrong endings

   Have the language model write many endings for that context. In SWAG, the authors produced 1,023 candidate wrong endings per item before any filtering.³

3. Keep only the hard ones

   Train the filter to separate the machine endings from the real one, and discard the wrong endings it catches easily. Repeat until only decoys it cannot separate survive; in SWAG, nine per item.³

4. Check that people still pass

   Show the survivors to human annotators, and keep the item only if they still pick the real ending. A decoy that fools people too is thrown out.²

On the finished test, people scored 95.6%, while the best model of 2019, the same BERT-Large used as the filter, managed 47.3%.² As a way to manufacture a hard test cheaply, adversarial filtering worked: it turned free captions and how-to text into a benchmark no 2019 model could touch, without anyone writing a single trick question by hand. The authors pitched the paper as a correction. An earlier benchmark had looked solved, and HellaSwag showed the models clearing it were exploiting patterns in the data rather than reasoning.²

The gap that took four years to close

At launch the gap was enormous: 47.3% for the best model against 95.6% for people, close to fifty points.² It did not last. By 2023, GPT-4 scored 95.3% on HellaSwag at ten-shot, meaning it was shown ten solved examples before each question.⁴ The leaderboard the benchmark's own author maintains lists that same GPT-4 entry against a human baseline it describes as over 95%.⁵

No checked model clearly passes 95.6. GPT-4's 95.3 is the strongest verified figure, and it sits three-tenths of a point under the human mark.⁴ What the top scores show is that the machines are level with people: bunched within about a point of the human line and of one another. At that range, the difference between two scores is noise.

When scores pile up at the top of a test this way, the test stops doing its job. It can no longer tell a stronger model from a weaker one, a state the field calls saturation. In 2024, the operators of the most-cited open leaderboard retired HellaSwag for exactly this reason, writing that models "are now reaching baseline human performance on HellaSwag, MMLU, and ARC, a phenomenon called saturation."⁶ Their verb is "reaching," not "surpassing," which is the honest word for what the figures show. This is the same arc a language benchmark called GLUE traced a few years earlier.

The typo in the answer key

A score is only as good as the key it is graded against, and two independent audits have found HellaSwag's key in poor shape. A 2025 validity study reports that almost 40% of the validation questions have ungrammatical prompts, and among the items drawn from ActivityNet captions the figure is 95.7%.⁷ In 21.1% of questions at least one decoy is as good as the keyed answer, and the keyed answer is the correct one in 96.3% of cases, so roughly one item in twenty-seven is simply mislabeled.⁷

The same study ran a check the reader can judge directly. It replaced each context with meaningless "Lorem ipsum" placeholder text and re-scored the models. More than 65% of their predictions did not change.⁷ A right answer is supposed to be the real continuation of the described scene. If the scene is blanked out and the model still picks the same ending most of the time, the endings alone are giving the answer away, and the situation the item describes is doing little of the work.

A separate hand review by Surge AI read 300 random validation rows and found 107 of them, 36%, contained an error: bad grammar, a typo, or a keyed answer no better than a decoy.⁸ And the flaw is not confined to the wrong answers. Here is one item whose keyed-correct ending is itself broken.

The prompt reads "People is around the field," and the ending marked correct reads "side to side of the ield," carrying the source's own typo for "field."⁸ A model is scored right for matching that. These are not stray bad items at the margin. Nearly all of the ActivityNet portion of the test is ungrammatical, and that is a large share of the questions any HellaSwag score is averaged over.⁷

Why 95% still reads as commonsense

GPT-4's technical report, published in 2023, lists its 95.3% on HellaSwag in a table of academic benchmarks, under a one-line description that calls the benchmark "commonsense reasoning around everyday events."⁴ Explainers repeat the reading. One describes closed-source models as "now performing on par with humans, with GPT-4 scoring 95.3%."⁹ By 2023, though, the test was already saturated and, as the audits show, substantially broken.

So a 95.3% on HellaSwag is consistent with three things at once that the bare number hides. The model is picking the real continuation, which the Lorem ipsum check suggests it often does without reading the situation.⁷ It is doing so on a test where much of the answer key is unreliable. And it is scoring at a ceiling where the number no longer separates one model from another.

The cost shows up when two close scores are read as a ranking. GPT-4 sits at 95.3, another model at 94 or at 95.6, and the higher number looks like the one with more commonsense. On a saturated test with an error-laden key, that gap is noise, not a real difference. Two such scores do not order two models by commonsense. They mostly report which one better matched a key that is wrong about one item in twenty-seven and ungrammatical on nearly 40%.⁷

None of this was the benchmark's failing when it was built. In 2019, HellaSwag did the job it was designed for. It separated the models of its day cleanly and punctured a claim that machine commonsense was solved.² Both the 47.3 from that year and the 95.3 from 2023 came off the same 59,950-item test. The first score sorted the models taking it. The second, at the ceiling, sorts nothing.