The Instruments

Change which SWE-bench tasks are counted and GPT-4o's score doubles

SWE-bench grades a model by running a real repository's own tests in a sealed container, and OpenAI reports the same GPT-4o clearing that bar on 16% of the original 2,294 tasks but 33.2% of the 500 its annotators kept.

10 min read2026-07-20

Why this matters

Every few weeks a lab announces that its coding agent resolves some share of real software bugs, and the share is almost always a SWE-bench score. The claim lands like a plain fact about a model, the way "passes 90% of the driving test" sounds like a fact about a driver. A SWE-bench score is built one bug at a time inside a container that runs a real project's tests, and knowing how it is built is the difference between reading it as a property of the model and reading it as one measurement among several. By the end you will know the three things a bare percentage hides: which set of tasks it was scored on, what software wrapped around the model to produce it, and whether the model had already seen the fix. You will also know why the same GPT-4o can post 16% and 33.2% on what both announcements call SWE-bench.

Background optional reading

01: The lab that ships an MMLU score rarely says how it was measured — the sibling case of a fixed test set, where contamination and format do the same quiet work to a multiple-choice number.
02: Chatbot Arena's Elo is a preference vote that pays out for style — the other kind of instrument, a live human vote rather than an executed test, for contrast with this one.

What the container actually checks

When a lab reports that its agent resolves some percentage of real GitHub issues, the number is almost always a score on SWE-bench, an evaluation built by Carlos Jimenez and colleagues at Princeton and the University of Chicago and presented at ICLR 2024.¹ SWE-bench is 2,294 problems drawn from real issues and the pull requests that fixed them, across 12 widely used Python projects, among them Django, scikit-learn, Flask, matplotlib, and sympy.¹ Each problem, called a task instance, bundles three things that already sit in a repository's history: the text of a bug report or feature request, the change a human merged to resolve it, and the project's own test suite.

The grade is a program that runs. To score one instance, a harness checks out the repository at the commit just before the merged fix, installs the project, and gives the model the issue text and the code around it. The model returns a patch. The harness applies that patch, then runs a named set of tests inside a sealed container and reads what passes.¹ One set, the FAIL_TO_PASS tests, comes with the human fix and fails on the unpatched code; it must pass after the model's patch, confirming the bug was addressed. A second set, the PASS_TO_PASS tests, passes before and must still pass after, confirming the patch broke nothing that already worked. The model resolves the instance when every test in both sets passes.¹ The tests themselves are supplied by the harness, not the model; the model only ever sees the issue and the code it can retrieve.¹

Hold onto that definition, because it is narrower than the word suggests. To resolve an instance is to make a fixed list of tests go green in a container, which is not the same as a maintainer reading the patch and judging it the right change. Grading by execution is what makes the number reproducible and hard to fudge, and it is also why "resolved" certifies only that a particular set of tests passed, not that the fix was the one the project wanted. The containerized harness the maintainers ship lets anyone rerun the grade and get the same answer.² When the original paper ran in 2023 the strongest model resolved under 2% of the set.¹ The trouble is not the grade but what a lone percentage leaves out.

One name, four different sets of problems

The first thing the word SWE-bench hides is which SWE-bench. The name now covers at least four sets of problems, and a percentage means little until you know which one it names. They differ in size, language, and how hard they are by construction.¹

The four sets that all travel under the name "SWE-bench," by their public test splits. A number on one is not a number on another.
Set	Test instances	What is in it
Full	2,294	12 Python repositories, the original set¹
Lite	300	a filtered, easier subset of the full set³
Verified	500	instances human-checked as fair and solvable⁴⁵
Multimodal	517	17 JavaScript libraries, each task carrying an image⁶

These are not four difficulties of one exam. Multimodal is a different language, JavaScript, and each of its tasks carries at least one image in the problem or the tests, so it measures something the Python sets never touch.⁶ Lite was pared down to instances with short, single-file fixes, which is why its own authors recommend it only when compute is tight, not as the headline number.³ Verified is where the trap is easiest to walk into: it looks like the same benchmark reporting a higher score, and a launch slide will quote it beside a full-set number as though the two were rungs on one ladder.

Why 500 of the tasks were thrown out

Verified exists because the original set graded some models unfairly low. An OpenAI-led team, working with the SWE-bench authors, had 93 professional Python developers review 1,699 random instances, three reviewers each.⁴ They found 38.3% of instances had underspecified problem statements and 61.1% had tests that could reject a valid solution; in all, 68.3% of the sampled instances were filtered out, leaving the 500 that make up Verified.⁴ One kept example from the blog shows the failure concretely. In the scikit-learn instance numbered 14520, a hidden test demands the solution raise a specific deprecation warning with an exact wording that was only settled in the pull request's discussion, text the model never sees; OpenAI's own verdict is that it "would be nearly impossible for an agent to solve this sample."⁴ The tests are the grade. When they check for a message the issue never states, a correct fix still fails.

Remove those instances and the same model resolves more of what is left. OpenAI reports that GPT-4o, on its best scaffold, reaches 33.2% on SWE-bench Verified, "more than doubling its score of 16% on the original SWE-bench."⁴ Nothing about the model changed between those two numbers. The set of tasks it was graded against did. The team argues, fairly, that the rise is honest: scores go up within each difficulty band once impossible instances are removed, so they took out the unsolvable, not the hard.⁴ The overall mix shifted easier too, and both can be true at once. Either way, the lesson for quoting holds. A 16 and a 33.2 that describe one model are both correct SWE-bench numbers, and setting them side by side as progress is a mistake about which set each was measured on.

The tasks a model may have read before

Verified corrects a number that ran too low. Contamination pushes the number the other way. Every instance is scraped from a public GitHub repository, and most predate a model's training cutoff, so the fix and its discussion may already sit in the data the model learned from. OpenAI states the risk plainly: large models trained on internet text "are likely to be contaminated on the tasks."⁴ A resolved instance can therefore mean the tests passed on a fix the model had seen, which is not the skill the score is taken to measure.

An independent study puts a size on the suspicion. Researchers at Purdue and Microsoft, none of them authors of SWE-bench or scoring their own agent on it, gave models only an issue's text, with no access to the repository, and asked them to name the buggy file. The strongest models identified the correct file up to 76% of the time on SWE-bench Verified, but only up to 53% on comparable tasks from repositories not in SWE-bench.⁷ A model that can point to the file to edit from the issue alone, having never been shown the code, has most likely seen the fix before. The same study also found far more verbatim overlap between model output and the real patches on SWE-bench than on tasks elsewhere, and concluded that its findings "raise concerns about the validity of existing results."⁷ This is a diagnostic, not a resolve rate, and it proves no single headline score inflated. It does mean a number scored on tasks older than the model carries a question a number on fresh tasks does not.

The software around the model, and the board that still sorts

A score can also be measured on a slice of the set and quoted as the whole. When Cognition launched the Devin agent in March 2024 it reported 13.86% on SWE-bench. That figure was 79 of 570 instances, a randomly chosen quarter of the 2,294, set beside the paper's full-set baseline of 4.80%.⁸ Cognition disclosed the quarter in the technical report; the distortion enters when "13.86% on SWE-bench" is repeated without it, against a baseline measured on the full set.⁸

The agent wrapped around the model moves the number as much as the model does. On SWE-bench Lite, one GPT-4 scored 2.7% with an early retrieval-based scaffold and 28.3% with a stronger one called CodeR, more than a tenfold spread with the weights unchanged.⁴ The original paper had shown a gentler form of the same effect: Claude 2 resolved 1.96% of the set when a keyword search chose its context and 4.80% when it was simply handed the files the human fix had touched.¹ The official leaderboard controls for part of this by giving every model the same system prompt, which, as Simon Willison notes, is fair for comparison but means the quality of a lab's own harness and prompt is not what the board measures.⁹

None of this needs anyone to lie. Overclaiming, as Arvind Narayanan and Sayash Kapoor argue, is the predictable response to what a leaderboard rewards, and a bare accuracy figure quoted without the compute or the scaffold behind it drops the column needed to price it.¹⁰ The instrument still sorts real systems when one party runs it in a single piece. An official run in February 2026, not self-reported by the labs, placed Claude 4.5 Opus at 76.8% on Verified, with Gemini 3 Flash and MiniMax M2.5 just behind at 75.8%, an order that matches how engineers who use these agents would rank them.⁹

The takeaway

A SWE-bench score is a genuine measurement, and it answers a narrow question: on this set of tasks, with this software feeding the model, how often did a repository's own tests go from failing to passing. Read that way, "65% on SWE-bench" is a question rather than an answer. Sixty-five percent of which set: the 2,294 originals, the 500 that annotators judged solvable, the 300 in Lite, the JavaScript of Multimodal? Produced by which scaffold, when the same GPT-4 swung from 2.7% to 28.3% on that choice alone? And scored on tasks the model was trained after, or on ones whose fixes it may already have read, the way the strongest models can name the file to edit from an issue's text alone far more often on tasks they have seen? The GPT-4o that resolves 16% and 33.2% is one model both times; only the set of tasks changed. Keep in view what "resolved" certifies, that a fixed list of tests passed, and the percentage becomes what it is, a checkable estimate of one thing, too often quoted as though it settled another.

Go deeper optional reading

01: Simon Willison, "SWE-bench February 2026 leaderboard update" — a plain reading of the current official standings and the harness caveats behind them.
02: Liang, Garg and Zilouchian Moghaddam, "The SWE-Bench Illusion" — the contamination diagnostics in full, including the buggy-file test on tasks the models never saw.

What the container actually checks

One name, four different sets of problems

Why 500 of the tasks were thrown out

The tasks a model may have read before

The software around the model, and the board that still sorts

Sources