OpenAI's state-of-the-art GAIA score came from the set whose answers are public

OpenAI reported a score that looked like it was catching up to people

On 2 February 2025, OpenAI announced its Deep Research agent and said it had reached a new state of the art on GAIA.¹ The figure was 67.36 percent.² GAIA is a benchmark for general AI assistants: real-world questions a person can answer but a machine mostly cannot. The humans who wrote and checked those questions answered 92 percent of them.³ A machine at 67 where people sit at 92 reads like a system most of the way to human performance on real tasks.

Two things complicate that reading. The 92 percent is not a casual human with no help. It is the benchmark's own annotators, working with web access and tools and taking six to seventeen minutes a question.³ And the 67.36 percent was measured on GAIA's public set, whose questions and answers are posted online, while the leaderboard that actually ranks assistants runs on a separate set of questions whose answers are withheld.⁴

GAIA is not the only agent number the course has met. SWE-bench scores an agent on fixing real repository bugs, τ-bench on holding to a single customer-service script through a tool-using conversation, and METR's task-time-horizon on how long a task an agent can finish before it fails. GAIA's claim is broader than any of them: general assistance, any everyday task with a checkable answer. Because the tasks are so varied, a high GAIA score is easy to hear as competence at real work in general.

A GAIA score starts with 466 questions and one reference answer

The benchmark was built by a team led by Grégoire Mialon, with co-authors including Clémentine Fourrier, Thomas Wolf, and Yann LeCun, and published in 2023.³ They wrote 466 questions, each with a single correct answer they had verified.³ Hugging Face hosts the data and runs the public leaderboard.⁴

Every question is built to be easy for a person and hard for a machine. Not a trivia lookup, but a chain of steps that may require reading a document, calling a tool, or handling an image or spreadsheet along the way. The questions come in three difficulty levels, set by how many steps and tools each one needs.

Grading is mechanical. A model's answer is compared to the one reference answer by quasi-exact match: the two strings must agree after a light normalization that depends on the answer's type, so that 8 and eight, or a differently spaced list, can still count as equal.³ For that to work the benchmark has to tell the model exactly how to shape its reply.

At release the gap looked like this. The annotators scored 92 percent aggregated across all three levels. GPT-4 equipped with plugins scored 15 percent overall, and on the Level 3 chains it solved none.³

The public questions and the withheld questions are not the same test

Of the 466 questions, 166 were released with their answers as a developer set, and the answers to the other 300 were withheld to power the leaderboard.³ The public 166 are the validation set. The withheld 300 are the test set.

Why keep the answers secret? Because a public answer stops testing anything once a model can read it. Hugging Face gates the data and makes every user agree not to reshare it outside a private repository, to keep the questions out of the text models train on.⁵ The validation set has already lost that protection. One explainer notes that its questions and answers are widely available online, so a model "might have 'memorized' them during their training rather than deriving solutions from genuine reasoning."⁶ That is contamination, the same failure an earlier lesson traced through LiveCodeBench, where the fix was to score models only on problems released after they were trained.

That is why the leaderboard changed. The public validation leaderboard was closed, and the ranking Hugging Face maintains now scores the withheld test set only, reporting each system's percentage of correct answers on that split.⁴ A validation score and a test score are two different measurements, and only one of them is the ranked result.

The 67 percent got read as a leaderboard win it never was

Return to OpenAI's claim. Deep Research's 67.36 percent was a validation-set figure. Hugging Face's own engineering team, which includes GAIA co-author Clémentine Fourrier, reported it as near 67 percent correct on average and 47.6 percent on the hardest Level 3 questions, all on the validation set.² Framed the way OpenAI framed it, as "a new state of the art on GAIA,"¹ the number traveled as though it were a rank on the leaderboard. It was not. The leaderboard runs on the test set, and this figure was never on it.

The cost showed up when the number got compared. On 17 March 2025, H2O.ai reported its own agent at 75 percent, claimed the top spot, and placed itself "ahead of OpenAI's Deep Research."⁷ H2O.ai said plainly what the comparison rested on. Its 75 percent came from the test set, "a more rigorous and extensive evaluation involving nearly double the number of questions with no known data leaks." Deep Research and the newcomer Manus, it noted, "were evaluated on validation data."⁷ A 75 on the test set and a 67 on the validation set do not rank one system above the other. They are scores from two different exams.

How far apart are those exams? One report shows it on a single system. JD's JoyAgent-JDGenie scored 75.2 on the validation set and 67.1 on the test set: the same system, eight points lower once the questions shift from public to withheld.⁸ A headline that stacks one lab's validation figure against another's test figure is reading across that gap without seeing it.

None of this means GAIA measures nothing. The public set's exposure is a reason to distrust a validation number read as a ranking, not a reason to distrust the benchmark. A recent independent survey of agent evaluations, the Holistic Agent Leaderboard, uses GAIA as a routine web-search-and-reasoning benchmark and raises no contamination flag against it.⁹ The problem is narrow and repeatable: comparing a number from the leaky public set with a number from the clean withheld one.

A high GAIA score licenses less than the headline says

So what does a GAIA number actually support? Take it on the test set, where the answers are clean. A high score says the system can carry a multi-step task with a single checkable answer through to completion, using tools and the web, and land the exact string the grader wants, more often than the systems below it on the same split.⁴ That is a real and difficult thing to do.

It does not support the rest of what the headlines say. It does not show the system beats people: the human 92 percent is an annotator with tools and time, and the highest verified test-set score in this record is H2O.ai's 75 percent, well under it.⁷ The validation figures that feed "assistants beat humans" sit on the set a model can memorize.⁶ And a GAIA score says nothing about tasks with no single checkable answer, which is most of what a person actually hands an assistant.