The same o3 scored 75.7% and 87.5% on ARC-AGI, and only the compute changed

One model, one task set, two very different numbers

In December 2024, OpenAI's o3-preview scored 87.5% on ARC-AGI-1, and a wave of coverage read it as the arrival of artificial general intelligence. The same model, on the same 100 tasks, also scored 75.7%.¹ Nothing about the model or the puzzles changed between the two figures. What changed was the test-time compute each run was allowed to spend, meaning the amount of computation the model used while answering, set here by how many separate candidate solutions it sampled per task before settling on its final answers.

The 75.7% run sampled six candidate solutions per task, at about $26 a task. The 87.5% run sampled 1,024 per task, at about $4,560 a task, roughly 172 times the compute for the identical set of grids.¹ The eval set was the same in both: the ARC-AGI-1 semi-private set of 100 held-out tasks.¹ A percent on this test is not yet a measurement until its compute cost and its eval set are named beside it.

What ARC-AGI-1 actually asks

ARC-AGI-1 is a set of grid-reasoning puzzles. Each task shows a few example pairs, usually about three, where an input grid of colored cells is transformed into an output grid, and the solver has to infer the rule and apply it to a new input.² The design intends that a system deduce the rule on the spot rather than recognize it from training, so the tasks are novel and the rules are not published.²

The corpus is split four ways. There are 400 public training tasks to build against, 400 public evaluation tasks to test a finished system, 100 semi-private tasks held back to test closed models like o3, and 100 fully private tasks reserved for the ARC Prize competition.² The semi-private and private sets are withheld so a model cannot have trained on the answers. When you read an ARC-AGI score, which of these four sets it was measured on is half of what the number means, because a public-set figure and a held-out figure are answering different questions about the same model.

Grading is mechanical and strict. The rule is pass@2: for each task the system submits two candidate output grids, and if either one is an exact match for the withheld answer the task counts as solved, otherwise it scores zero.³ Exact match means pixel-perfect. Every cell's color and position has to be identical to the keyed solution; a single wrong cell fails the task.³ The final score is the fraction of tasks solved this way.

The dial that set the number

Because a task is solved if either of two submitted grids is exact, a system can raise its score by generating many candidate solutions, ranking them, and submitting its best two. More samples means more chances to have produced the right grid somewhere in the pile. Samples cost compute, and compute costs money, so the score has a dial attached to it, and the dial is spending.

One name for each run, kept for the rest of the lesson: the low-compute run scored 75.7% at six samples a task, and the high-compute run scored 87.5% at 1,024 samples a task. ARC Prize's own report labels them the other way around, calling the cheap run "high-efficiency" and the costly run "low-efficiency," because efficiency there means score per dollar.¹ Same two runs, opposite-sounding labels. This lesson names them by how much they spend.

The spending decides which number is even eligible for the public board. ARC Prize reports that the low-compute 75.7% "is within the budget rules of ARC-AGI-Pub (costs <$10k) and therefore qualifies as 1st place on the public leaderboard."¹ The high-compute 87.5% run is reported separately, on its own, and is not the leaderboard entry.¹ The figure that qualified as a public record and the figure that ran the headlines were not the same figure.

The test and the thing it is named after

The confusion in "o3 reached AGI" lives in a slide between two nouns. ARC-AGI is the name of a test. AGI, artificial general intelligence, is the ability people heard the test had certified. The person who built both keeps them apart. François Chollet designed ARC in his 2019 paper "On the Measure of Intelligence," and his argument there is that a score is exactly the thing you should not read as the ability. Skill at a task, he writes, "is heavily modulated by prior knowledge and experience," so "unlimited priors or unlimited training data allow experimenters to 'buy' arbitrary levels of skills for a system, in a way that masks the system's own generalization power."⁴ The compute dial is a second way to buy the same skill. More samples purchase a higher pass@2 score without changing what the model can do on its own.

Chollet said as much about o3 directly, in the same post that reported the 87.5%.

Independent readers landed in the same place, and on the same mechanism. Rolf Pfister and Hansueli Jud argued in early 2025 that the 87.5% was reached "through the extensive use of computing power," by massive trialling of predefined operations, which "cannot be a basis for AGI."⁵ Melanie Mitchell, writing days after the announcement, priced the same run from the outside: "several thousands of dollars per task, and over $1M total for the whole evaluation," and asked whether o3 was solving the tasks with the kind of abstraction the benchmark was built to measure at all.⁶ A model passed some held-out grids at a chosen compute budget under a pass@2 rule. That is what happened. General intelligence arriving is a different claim, and it was not what the number showed.

What the next test did to the reading

If a high ARC-AGI-1 score had meant the test was solved, the systems that do well on it would keep doing well on harder problems of the same kind. In 2025 ARC Prize released ARC-AGI-2, which keeps the input-output grid format and adds tasks meant to probe reasoning the first version no longer separated.⁷ The frontier collapsed. ARC Prize's own measurement of the released o3 shows the drop cleanly.

The pattern is general, not a quirk of one model. ARC Prize reports that no leading model has surpassed a 5% success rate on ARC-AGI-2, while comparable models score between 20% and 50% on ARC-AGI-1.⁷ For contrast, every ARC-AGI-2 task was solved pass@2 by at least two people during its human study, so the tasks are not beyond ordinary reasoning.⁹ A test read as evidence that a general ability had arrived was replaced by a successor of the same design, on which the same class of systems scored near zero. The "solved" reading did not survive the next test.

One honest wrinkle points the same way. The o3 that shipped to the public in April 2025 scored 53% on the ARC-AGI-1 semi-private set, far below the December preview's 75.7% to 87.5%, because OpenAI says the released model is a different, lower-compute system.⁸ The name "o3" stayed fixed while the compute budget dropped, and the number dropped with it. That is the compute dial again, seen from the product side.