On the same WinoGrande questions, more training lifts the score from 50% to 79%

A pronoun that turns on one word

The trophy does not fit in the suitcase because it is too big. Change one word, big to small, and the pronoun points the other way: now the suitcase is too small, and it is the suitcase. A person resolves the reference without noticing the work. A program has to know that a big object will not go inside a small container, and then work out which object the sentence just made too big.¹

That pair is a Winograd schema: two sentences differing in one word, with a pronoun whose correct referent flips when the word changes. Hector Levesque, Ernest Davis, and Leora Morgenstern proposed them in 2012 as a commonsense test a machine could not pass by counting words.¹ They called this being Google-proof: you cannot look the answer up in a table of word co-occurrences, because the wrong reading and the right one rest on words about equally common in ordinary text.¹

WinoGrande is the benchmark that took that idea and built 43,972 such problems, up from the original collection of a little more than a hundred.² Its score now rides on the report cards of systems from GPT-3 to Llama 2, printed as a commonsense-reasoning number.³⁴ Its difficulty was engineered, question by question.

The old test got solved without the reasoning

The original challenge stayed small. Levesque, Davis, and Morgenstern published a collection of a little more than a hundred schemas and proposed scoring a system on tests of fifty questions.¹ A later standardized set, WSC273, fixed 273 of these schemas as a shared test, and that 273-question set is what most papers mean when they cite the Winograd Schema Challenge.⁵ The 2012 paper itself reports no machine scores. It is a design, not a result.¹

The design held a while, then stopped. The one time the challenge ran as a formal competition, at the IJCAI conference in 2016, six systems entered and the best answered 58% of a 60-question set, barely above the 50% a coin flip earns.⁵ Within three years that had inverted. By 2019 transformer language models were scoring above 90% on WSC273.⁵ The scores looked like commonsense reasoning had arrived.

It had not. Some of the questions were answerable by exactly the word statistics the schema was meant to defeat. When Trichelair and colleagues went through WSC273 by hand, they found that 13.5% of the items could be settled by simple association: the correct answer word just occurs more often near the context word than the wrong one does, no reasoning required.⁶ A model that has read enough text picks those up for free. Reviewing the record later, two of the original challenge's authors, Davis and Morgenstern, judged that the high scores did not reflect the ability the test was built to find.

Filtering out the questions a model can guess

WinoGrande, published in 2019 by Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi at the Allen Institute for AI, set out to fix both faults at once: the small size and the leftover shortcuts.² Size was the easy part. Crowd workers wrote fresh schemas, each primed with a random anchor word pulled from WikiHow so the sentences would not all sound alike, and three separate workers checked every one. That produced about 53,000 validated problems.²

The shortcuts were the hard part. The failure has a name: an annotation artifact, a pattern that leaks into a dataset while people are building it and lets a model score well by matching the pattern instead of doing the task.² In a commonsense set the artifact can be as plain as the correct answer word tending to be the more common one, the same association Trichelair found in the old test.

To strip them out, the team used adversarial filtering: rather than trust people to spot the shortcuts, let a model find them and then delete the questions it finds. Their version, AFLite, works in rounds. It turns every question into a vector from RoBERTa, a standard language model, then trains simple linear classifiers on random slices of the data over and over and watches which questions they keep getting right. A question that a plain linear model predicts correctly again and again carries an exploitable pattern, so AFLite throws it out.²

What comes out the far end is smaller and harder. The full pool holds 43,972 problems. The filtered set the hard scores are reported on, the one the paper calls debiased, is 12,282.² You can watch the shortcut leave. The team summarized how strongly the answer words alone gave the answer away, as a single divergence number, and it fell from 2.53 to 0.12.² The filter is not specific to pronouns. Run on a natural-language inference set instead, it drops BERT's accuracy from 92.0% to 63.5% while people keep answering the questions.⁷

The score moves with the training budget

On the filtered set, three crowd workers voting together answer 94.0% of the questions correctly. That is the human figure WinoGrande reports.² The best machine at publication, RoBERTa fine-tuned on the whole training set, reached 79.1%.² A fifteen-point gap between people and the strongest model, on a test built to demand reasoning, is what made WinoGrande look like a real measure of it.

That 79.1% is not a fixed property of the questions, though. It is a property of the questions together with how much training the model was given. The team ran the same RoBERTa on the same filtered test after training it on larger and larger slices of the data, and the score climbs the whole way. Trained on 160 examples it scores 50.4%, no better than guessing. At 640 examples, 58.6%. At about 2,500, 67.6%. At about 10,000, 74.7%. On the full 40,938, 79.1%.²

The questions never change across that curve. Only the training budget does. So how hard WinoGrande is has no one answer: it is 50% hard or 79% hard depending on a number you get to set. The difficulty is a fact about the construction, the model the filter was tuned against, and the training allowed, not a fixed reading of how much reasoning the task takes.

The near-human scores came back

WinoGrande exists because near-human scores on the old test turned out to be hollow. Within about a year of its release the same shape returned. By 2021 a transfer-trained system called UNICORN reported 86.6% on WinoGrande, roughly seven points under the human 94.0%.⁸ GPT-3, in 2020, reached 77.7% with a few examples in its prompt and no fine-tuning at all.³ Llama 2's 70-billion-parameter base model scored 80.2%.⁴ The fifteen-point gap that made WinoGrande interesting was closing.

Two things complicate reading the closing gap as reasoning gained. The first sits in the model reports themselves. Both the GPT-3 paper and the Llama 2 paper flag possible contamination of their training data with the test set, in their own words, and publish the WinoGrande number anyway.³⁴ A score rises when a model has already seen the answers, and these labs say plainly they could not rule that out.

The second is the filter. AFLite defines hard relative to one model's view of the data, RoBERTa's vectors. Its own authors say so, and add a warning: deleting the items a model finds easy also deletes some items that are easy for an honest reason, not only items with shortcuts.⁷ So a low machine score partly reflects what one model family could not represent, which is not by itself reasoning difficulty. Davis and Morgenstern press harder. Going through WinoGrande's released questions, they argue that many stay solvable by word association and do not come close to meeting the original schema rules, with examples where the supposedly irrelevant words still lean toward one answer.⁵

None of this makes WinoGrande a bad benchmark. It is a real advance on the test it replaced: bigger, validated, and cleared of the specific word-association shortcuts that broke the original.² It still is not a clean readout of reasoning. The paper even overreaches its own math here: it states that reaching the human 94% would take over 118,000 training examples, though the fit it prints does not reach 94% by that size.²

So what does a WinoGrande number on a model card license in 2026? That the model, trained on however much data it was trained on, answers this one filtered set of pronoun questions at some rate. It does not license the jump to a settled quantity of commonsense reasoning. The score moves with the training budget, the filter's sense of hard is tied to a single model, and the labs reporting the highest numbers cannot promise the answers never leaked.