Gemini 2.5 Pro (Experimental 03-25) fell from 92% to 87% when the exam turned over

A three-hour exam built for teenagers, borrowed by labs

The American Invitational Mathematics Examination is a real exam. The Mathematical Association of America writes and administers it every year for high schoolers who already qualified through an earlier round, the AMC.¹ It gives students three hours to solve 15 problems, and it runs twice a year, AIME I and AIME II, so a student can sit up to 30 problems total across both.¹ Every answer is a single integer from 000 to 999. One point per correct answer, no partial credit, no penalty for a wrong guess.¹

A lab turns that fixed problem set into a benchmark score by feeding the same 15 or 30 problems to a model instead of a student, generating one or more completions per problem, and checking each completion's final integer against the answer key. What differs from a human sitting is what happens next: a lab can run the model on a problem once and grade that single try, or run it dozens of times and average the results. OpenAI's announcement of o1 reports all three approaches on the same problem set and the same model. Graded on a single try, pass@1, it scores 74%. Graded by taking the model's most common answer across 64 tries, consensus@64, it scores 83%. Graded by generating 1,000 samples and re-ranking them with a separate scoring step, it scores 93%.² Three numbers, one model, one exam. (The mechanics of sampling many completions and scoring them are taught in full in this course's HumanEval lesson, linked above; this lesson assumes them.) A separate line of research backs the practice of averaging. Grading a model on one random sample bakes in whatever randomness that sample happened to carry. Repeated sampling is how a lab estimates a score the single-run number cannot give on its own.³

One problem is worth 6.7 points of the score

Fifteen problems make a score out of fifteen possible points, so answering one more or one fewer moves the percentage by 1/15, or 6.7 points.¹ Solve all but one and the score reads 93.3%, not 100%. Miss two and it reads 86.7%. Pool both exams into one 30-question score and each problem is worth half as much, 3.3 points, still a coarse grid for a number a system card often reports to a decimal place.

Put that grid next to a real comparison. DeepSeek's R1 paper reports 79.8% pass@1 on AIME 2024.⁴ OpenAI's o1 reports 74% pass@1 on the same exam, already cited above. The gap between them, 5.8 points, is smaller than the value of a single problem. A ranking built on a gap that size is one problem's difference dressed up as a margin. Before comparing two labs' AIME numbers at all, the relevant question is whether each used the same convention: pass@1 against pass@1, consensus@64 against consensus@64. Many system cards state a single percentage without naming which.

The 15-problem count is an extreme case of a general problem in small evaluation sets: how a test is formatted and scored changes the measured spread even before contamination or sample size enters the picture. Reformatting MMLU's multiple-choice questions into a different answer format, with the underlying model and questions unchanged, moved the measured run-to-run spread for one 7-billion-parameter model from 0.57 down to 0.22.⁵ Nothing about the model's actual ability changed. What changed was how the test counted an answer. AIME's per-problem arithmetic is just the most visible version of a fact that holds across most small benchmarks. A methodology choice a headline number does not disclose can move the score by more than the difference the headline is built to report.

The same 15 problems, circulating online for years

AIME problems do not stay secret. Solutions and discussion threads go up within days of each sitting, and sites built to archive competition math, including the one that documents the exam's own rules, host the full problem sets from prior years.¹ A model trained afterward on a general web crawl can absorb the answer key along with everything else. A benchmark run against the same fixed problems year after year risks testing what a model memorized, not what it can work out.

MathArena tested for exactly that on AIME 2024. The researchers built their expected-score baseline from how actual human contestants performed across AIME's own two sittings. A model's AIME 2025 score sets a quantile, and that quantile predicts what the same model should score on AIME 2024 if only the difficulty had changed.⁶ Most leading models scored 10 to 20 points above that expectation on the 2024 exam. One, QwQ-Preview-32B, scored nearly 60 points above it, a gap the researchers called extreme contamination.⁶ The same paper ran the identical check on HMMT, a competition far less prominent online, and found a much smaller gap there. The authors attribute the difference to exposure: HMMT is less likely to have made it into training data at all.⁶ That 60-point figure is not a raw AIME percentage. It is a gap against an expected score, a different measurement answering a different question than "how many of 15 problems did the model solve." Both readings are true of the same result: a model can score a modest raw percentage and still be flagged as extremely contaminated, if that percentage is far above what its general skill predicts.

A separate check, VAR-MATH, rewrites AIME problems with new numbers plugged into the same structure and the same required method, then re-scores the same models.⁷ Accuracy fell by an average of 58.8% on the 2024 set and 72.9% on the 2025 set once the specific numbers changed, with DeepSeek-R1 alone dropping from 81.5% to 61.3%, a 24.8% decline, on the reworked 2025 problems.⁷ The models really did answer that share of the original problems correctly. What VAR-MATH exposes is how much of that answering held onto the exact problem as written rather than the method behind it, on years both stale and fresh. Some of what reads as "solved AIME" is closer to matching a memorized problem than to running its method on a new one. Benchmark leakage is not unique to this exam. Scraped training data has been shown to raise scores on other fixed test sets the same way, and models that solve older competitive-programming problems easily often struggle on problems written after their training data was collected.⁸

The score that fell when the exam turned over

Google's Gemini 2.5 Pro Preview Model Card, published April 16, 2025, carries the fix. It lists 92.0% on AIME 2024 and 86.7% on AIME 2025 for one checkpoint, Gemini 2.5 Pro (Experimental 03-25), both rows labeled "single attempt (pass@1)."⁹ Same document, same checkpoint, same label, both years. The card's own methodology note adds a caveat: Google averaged over multiple trials for smaller benchmarks even under "single attempt," the same noise-reducing practice the earlier section described. Neither number is one raw try.⁹ What is stated plainly is the change: a 5.3-point drop, on the same checkpoint, when the problems it faced changed.

Google's own next update to the card shows why naming the checkpoint matters. Updated June 27, 2025, that document carries no AIME 2024 row at all. Its only AIME 2025 figure, for the shipped model, unqualified "Gemini 2.5 Pro," is 88.0%, different from the preview checkpoint's 86.7% on the same exam months earlier.¹⁰ A citation that just says "Gemini 2.5 Pro scored 86.7% on AIME 2025" could point to either document, and only one carries an AIME 2024 figure to compare against.

Size the drop against the arithmetic already built. A 5.3-point fall is smaller than the 6.7 points a single problem is worth on a 15-question sitting. Taken alone, a move that size does not prove contamination. It is consistent with a checkpoint that got unlucky on roughly one extra problem's worth of difficulty, nothing more, and the drop by itself does not say which explanation is true.

What makes the 2024 number suspect is the independent evidence from the previous section, not its own later fall. MathArena found leading models scoring 10 to 60 points above their expected performance specifically on the 2024 problems. VAR-MATH found accuracy on both years collapsing once the same problems were reworded with new numbers. Neither result depends on the 2025 score existing at all, and together they point the same direction. At least some of the near-saturated 2024 claims that circulated through 2024 reflected exposure to problems that had been public for months. They did not reflect a math ability that transferred cleanly to a test the model had never seen. That is what the fresh exam cost the earlier claims. None of these results run a controlled test of what any one model memorized, so this is not a disproof. A year and a differently worded problem set were still enough to undercut the earlier headline.