o1 beat GPQA's recruited PhDs, not its own expert ceiling

The day a model outscored the recruited PhDs

On 12 September 2024, OpenAI released o1 and reported that it answered 78.0 percent of the questions on GPQA-Diamond correctly using a consensus of samples, and 77.3 percent on a single pass. In the same post OpenAI described recruiting people with PhDs, giving them the same 198 questions, and recording their score: 69.7 percent. o1 was, by OpenAI's account, the first released model to sit above that human number on this test.¹

The reading came fast. Nature's news desk told readers that o1 had "become the first large language model to beat PhD-level scholars" on the Diamond set,² and the launch-day trade press had already stamped the same phrase, with VentureBeat announcing a model "claiming PhD-level performance."³ A model had passed PhDs on graduate-level science questions, and the result traveled as a machine that could out-reason working scientists. Two words in the benchmark's own title, "graduate-level" and "Google-proof," did much of the compressing. They sound like a guarantee that a high score reflects the kind of thinking a scientist does.

They are not that. They are claims about how GPQA's questions were written and checked. Reading the o1 result correctly means knowing what those two words were built to mean, which subset the 78 percent lives on, and what the 69.7 percent baseline was. Each of those is checkable, and together they narrow "a machine passed the PhDs" into something accurate.

What the two adjectives were built to mean

GPQA is a set of 448 multiple-choice science questions, each with four options, written by people holding or pursuing PhDs in biology, physics, and chemistry.⁴ That authorship is the whole of the "graduate-level" claim. It says domain experts working at the frontier of their subjects composed the questions, not that they were scraped from textbooks or exams. It does not, by itself, say what solving them requires.

The paper measures how hard the questions are for people. Experts in the matching field scored 65 percent, rising to 74 percent once you discount questions where the expert later agreed they had made a clear mistake.⁴

"Google-proof" is the second measurement. The authors recruited skilled non-experts, people with technical training but not in the question's field, and let them answer with unrestricted access to the web. They averaged more than thirty minutes per question and still scored 34 percent.⁴ A question you cannot look up in half an hour is Google-proof in the paper's sense. Chance on four options is 25 percent, so the non-experts landed barely above guessing with the whole internet open.

A score counts correct answers. It does not record how they were reached. The 34 percent tells you the non-experts mostly failed to find the answer online. It does not tell you the experts' 65 percent came from reasoning rather than recall, and it cannot tell you how a model reached whatever it reached. Melanie Mitchell, who studies how machines are evaluated, states the general version plainly: AI systems have repeatedly been found to exploit unintended associations in a benchmark to give "the right answer for the wrong reason."⁵ A correct answer and a creditable process are two separate findings. Nothing about a percentage keeps them together.

The two human numbers Diamond ships with

GPQA is not one test. The authors ship three nested sets: an Extended set of 546 questions, a Main set of 448, and a high-confidence core of 198 called GPQA-Diamond.⁴ Almost every headline model score, o1's included, is reported on Diamond, so it is the one worth understanding.

Diamond is the hardest, most-discriminating slice. A question enters it only when both experts who validated it answered correctly and most non-experts got it wrong.⁴ That construction sets a trap for anyone who reads an expert number off this subset. The paper reports 81.3 percent expert accuracy on Diamond in its Table 2 and marks the figure with an asterisk: it is "skewed by selection effects."⁴ Because Diamond keeps only questions the validating experts already answered correctly, measuring their accuracy on that filtered set inflates it by construction. The 81.3 percent is the highest human figure GPQA records here, but it measures the selection rule as much as the experts.

The 69.7 percent human baseline OpenAI reported was not those validators. It was a fresh crew of recruited PhDs who answered the questions without having built them.⁶ So the launch measured the model against the weaker of two human numbers the benchmark carries: a recruited baseline near 70 percent, not the 81.3 percent the paper records for the validating experts on the same set. "Above the human line" was true. It was true of the lower human line.

Three steps between the score and "beats PhDs"

Three specific moves separate o1's 78 percent from the claim that a machine out-reasons scientists. Each one is checkable.

The first is the baseline, already named. The human number o1 passed was a recruited group, not the experts whose agreement defines Diamond.⁶ Clearing the 69.7 percent recruited baseline is not clearing the 81.3 percent the validating experts reach on the same questions.

The second is leakage. GPQA is public and downloadable, which means its questions can end up in the text a model is trained on. The authors saw this coming. The dataset ships with a canary string and an explicit request not to reproduce the questions in plain text online, "to reduce the risk of leakage into foundation model training corpora."⁷ A model that has seen a question during training can return the answer from memory, which looks identical to solving it on a score sheet. Arvind Narayanan and Sayash Kapoor showed how concrete this gets: GPT-4 could regularly solve coding problems published before its training cutoff and almost none published after, a dated pattern that points to memorization rather than skill.⁸ A score cannot tell recall and reasoning apart.

The third is the format. Diamond is four-option multiple choice. Picking the right option from four is a task large language models are unusually good at: it rewards retrieving and ranking known facts, not producing a derivation someone could check. A high multiple-choice score is strong evidence of knowledge and weak evidence about process.

None of this means OpenAI overclaimed. Its launch post carried a careful caveat.

The lab hedged. The coverage did not. "More proficient on some problems a PhD would be expected to solve" became "PhD-level" in Nature's news desk and in the launch-day trade press,²³ and the three steps above dropped out.

A Diamond score without the diploma

The honest ledger has two sides. First, what a high GPQA-Diamond score cannot tell you. It cannot tell you a model reasoned rather than recalled, because the format and the leakage risk both leave that question open. It cannot tell you a model matches a working scientist, because it is 198 multiple-choice items in three fields, not the open-ended work of research. And it cannot certify "graduate-level" thinking, a phrase that described who wrote the questions, not how a machine would answer them.

Second, what it still can tell you. The numbers themselves are trustworthy. Epoch AI, an independent evaluation group, re-ran the top models' Diamond scores and found the labs' self-reported figures line up with independent measurement, with no sign of systematic inflation. The largest gap among leading models was statistically insignificant.¹¹ The benchmark is not spent either. Reviewing Diamond's hardest questions by hand, Epoch estimated at least 90 percent of them are still valid, well-formed problems.¹² A high Diamond score is real evidence that a model holds and can deploy a large store of specialized science knowledge. That is worth measuring. It is simply not a diploma.

What it cost to blur the two is visible in the climb. Claude 3 Opus scored 50.4 percent on Diamond in March 2024; o1 reached 78 percent that September; by 2026 frontier models sit around 94 percent.⁹¹⁰ A benchmark that domain experts and paid validators built, designed so thirty minutes of web search would not crack a single question, went from roughly half to near its ceiling in about two years. Read as a measure of knowledge under a hard answer key, that curve is impressive and largely honest. Read as a scoreboard of machines overtaking scientists, it sold a story the number was never built to support, and the people who bought it now have to unlearn it.