OpenAI carved out MATH-500 because 4,500 of the 5,000 test problems were in its training data

What the grader actually reads

A MATH problem is a competition question paired with a full worked solution, and the model is asked to place its final answer inside a LaTeX \boxed{} command. The grader reads only what sits between those braces. It takes that string, compares it to the answer key, and marks the problem right when the two match exactly. The reported score is the fraction of the test set that matched. It says nothing about the steps that produced the answer.¹

MATH is 12,500 competition problems, split into 7,500 for training and 5,000 for testing, drawn from contests such as the AMC and the AIME and sorted into five difficulty levels.¹ When the dataset was released in 2021, the best model its authors tested got 6.9% of the test set right, and a few-shot GPT-3 with 175 billion parameters got 5.2%. Their conclusion was that scaling model size alone would not solve it.¹

A three-time IMO medalist scored 90% on the same test

The score did not stay at 6.9%. In 2022, Google's Minerva model reached 33.6% on the full test set on a single attempt, and 50.3% when it answered each problem sixty-four times and kept the most common answer, a method called majority voting.² By 2024 a math-specialized open model, Qwen2.5-Math, reported 85.9%.³ In 2025 DeepSeek-R1 reported 97.3%.⁴

Some of that climb is real. When the MATH authors gave the test to people, a computer-science PhD student got about 40%, and a three-time International Math Olympiad gold medalist got 90%, missing only on small arithmetic slips.¹ A score in the nineties is not past a strong human reasoner, so a model reaching one is not by itself proof that the test is broken.

The same answer, graded right and graded wrong

Exact string-matching is strict, and it errs in both directions. In one direction it fails correct answers. The value 1/√3 and the value √3/3 are the same number written two ways, but as strings they do not match, so a model that writes its answer in the wrong form loses a point it earned. Minerva's team addressed this by grading with SymPy, a symbolic-math checker that decides whether two expressions are equal rather than whether two strings are identical. That change recovered about one percent.²

In the other direction, string-matching passes wrong work. A model can reach the correct final answer through invalid steps: a sign error that cancels another sign error, or a guess that happens to land. The grader never reads the steps, so it counts the problem solved. When Minerva's team checked its own marked-correct solutions by hand, about 8% of them rested on flawed reasoning, and at Level 5, the hardest problems, that share rose to about 30%.²

So the score is not a count of problems a model solved. It is a count of final answers that matched the key, which runs a little low on formatting and a little high on the hardest problems, where wrong reasoning most often reaches a right number.

Why "MATH" and "MATH-500" are not one number

In 2023, OpenAI researchers ran into a problem of their own making. Building a system that grades reasoning step by step, they had put 4,500 of the 5,000 MATH test problems into its training data, a set they call PRM800K.⁵ A model that has seen the test during training can recite answers instead of working them out, which is called contamination, and it made an honest MATH score impossible for them to report. Their fix was to evaluate on the 500 test problems they had not trained on, chosen at random, and they showed those 500 match the full test set's spread of subjects and difficulty. They named the subset MATH-500.⁵

MATH-500 is now the number the leading labs report. DeepSeek-R1's 97.3%, and the 96.4% it lists for OpenAI's o1, are both MATH-500 on a single attempt, a metric written Pass@1.⁴ The original 6.9% and Minerva's 33.6% are the full 5,000. Qwen2.5-Math's 85.9% is labeled only "MATH," measured with four worked examples placed in the prompt, and the page does not say whether it ran on the 500 or the 5,000.³ The name on all of them is the same.

The same model on the same 5,000 problems scores 33.6% or 50.3% depending only on how many attempts it gets. The 85.9% does not say which set it ran on. The 97.3% ran on a tenth of the problems the 6.9% did. Lined up as a single "MATH" column, these numbers read as one rising measurement. They are not.

That the MATH problems leak at all is not a surprise: they are copyrighted competition questions, mirrored across the web for years, which is exactly the text models train on. The most-used copy of the dataset, on Hugging Face, now returns a single line where the data used to be. It was pulled under a copyright takedown.⁶

Sometimes the source of a MATH number vanishes altogether. A GPT-4 MATH score is quoted often, but OpenAI's GPT-4 technical report lists a GSM8K score and no MATH score at all, so any "GPT-4 on MATH" figure in circulation comes from some other evaluation, not from that report.⁷

The one thing the boxed answer never shows

Every one of these numbers comes down to one string between two braces. That is the score's strength: it is cheap, automatic, and repeatable across thousands of problems, which is why it became the standard math number in the first place. It is also the whole of what the score can see. A competition grader reading a paper by hand gives partial credit for a sound method with an arithmetic slip, and no credit for a right answer with no valid work behind it. The MATH score does the reverse of each: it fails the first on formatting and passes the second, because the only thing it reads is the final answer.²

Epoch AI, which tracks how hard benchmarks stay for frontier models, now treats even MATH's hardest level as saturated and has moved its frontier testing to harder, freshly written problems.⁸ A score that began at 6.9% and sits near 97% has little room left to measure anything. What it measured all along was whether a final boxed answer matched a key, never whether the reasoning that reached it was sound.