A passing contamination check on GSM8K would not catch a renamed problem

The 8,500 problems and the one rule for scoring them

GSM8K is a set of 8,500 grade-school math word problems. Karl Cobbe and co-authors at OpenAI built it in 2021 to test whether a language model can work through arithmetic reasoning rather than recite it.¹ The team split the set into 7,500 training problems and 1,000 held out for testing. Every problem takes between two and eight steps to solve, aimed at a bar a bright middle-school student could clear.¹ Workers hired through Upwork and Surge AI wrote the problems by hand, rather than scraping a source whose solutions might already be indexed online.¹ The authors found that 1.7 percent of problems still produced contractor disagreement, rounded in the paper to "less than 2 percent."¹

Grading one of these problems is exact match: a model writes a full solution in plain language, and the check is whether its final number matches the one keyed for that problem.¹ The paper's headline result compares two ways of running that check. A finetuned model samples one low-temperature solution and is scored right or wrong on the spot.¹ A "verifier" instead samples many higher-temperature solutions, scores each with a separately trained grader, and keeps whichever solution that grader ranks highest, based only on whether the final answer was correct.¹ On the full training set, a 6-billion-parameter model using the verifier method slightly outperformed a finetuned 175-billion-parameter model, a boost the paper likens to a 30-fold increase in model size.¹ Neither model's percentage is printed in the paper's text, only unlabeled curves in its figures. Even the comparison that made GSM8K famous has no stated numbers in the paper that built it.

One ablation shows why the written solution matters: finetuning the 6-billion-parameter model to output only the final answer dropped its test accuracy from 20.6 percent to 5.2 percent.¹ Exact match checks only the last number a model writes, but the score depends heavily on whether the model was trained to show its work first.

The 92 percent that shipped with a footnote

GSM8K's most quoted number is probably GPT-4's. OpenAI's 2023 report lists GPT-4 at 92.0 percent on GSM8K, five-shot chain-of-thought, against 57.1 percent for GPT-3.5.² An asterisk on that 92.0 points to a footnote: "For GSM-8K, we include part of the training set in GPT-4's pre-training mix."² The appendix explains the choice: OpenAI mixed data from GSM8K's own training set, and from a related benchmark called MATH, into the data GPT-4 learned from before it was ever tested.² The same appendix reports a clean leak check, then adds an instruction most coverage leaves out: "We recommend interpreting the performance results reported for GPT-4 GSM-8K in Table 2 as something in-between true few-shot transfer and full benchmark-specific tuning."²

Meta made the opposite choice, and said so, citing OpenAI's report directly. Llama 3's developers write that they "do not include any training sets from commonly used benchmarks" in the model's late-stage training data, aiming to measure genuine few-shot learning rather than benchmark-tuned performance.³ The same report shows how much that policy choice can move a score on its own. A small amount of non-benchmark math data lifted a pretrained Llama 3 8B model's GSM8K score by 24.0 points. The gain nearly vanished at 405 billion parameters.³ Two frontier labs handled that relationship in opposite, disclosed ways. A bare "GSM8K: X percent" beside another lab's "GSM8K: Y percent" is not one procedure run twice. It can be two different procedures wearing the same label.

Anthropic's Claude 3 model card adds a cross-check, reprinting GPT-4's 92.0 and GPT-3.5's 57.1 next to its own three GSM8K scores, shown below.⁴ The reprinted figures match exactly, so headline numbers travel intact across documents. What does not travel, unless a reader opens an appendix, is each lab's training-data policy.

Rename the numbers, and the score still moves

In 2024, researchers at Apple built a second instrument to test what a GSM8K percentage measures. Their method, GSM-Symbolic, takes each problem's template and generates new versions by swapping only the names and numbers, leaving every required reasoning step exactly as it was.⁵ They generated 50 such versions per template and ran roughly 25 models against all of them.⁵ If a GSM8K percentage measured a stable capacity to do the reasoning a problem asks for, a model should score about the same across all 50 relabeled versions of the same problem.

It did not. Accuracy typically varied by about 15 percentage points across the 50 versions of a single template, from changing nothing but names and numbers.⁵ One documented case: Mathstral-7b-v0.1 scored 80.0 percent on the original GSM8K questions and 74.0 percent, give or take 3.5 points, on the relabeled versions of those same questions.⁵ The paper's own authors offer one explanation for the swing. On 21 of the 25 models tested, the pattern was consistent with some of GSM8K's original test problems having leaked into training, inflating scores on the exact wording those models had already seen.⁵

A second, sharper version added one clause to each relabeled problem, one that reads as relevant but changes nothing about how to solve it, calling the result GSM-NoOp.⁵ In one example, a problem asks how many kiwis someone picked over three days, then adds that five of them "were a bit smaller than average." Several models, including o1-mini and Llama3-8B, subtracted those five kiwis from the total, even though the question never asks about kiwi size.⁵ Across all tested models, GSM-NoOp produced what the researchers call a catastrophic decline: over 65 percent for the smallest model tested, Phi-3-mini, with "even stronger models such as o1-preview" also showing significant drops.⁵ One technology reporter covering the release restated the same 65-percent figure for readers who would never open the arXiv page.⁶

A twin test built where none of the models could have looked

A separate 2024 project attacked the same question differently: Scale AI built GSM1k, a fresh set of grade-school word problems matched to GSM8K on difficulty and on how often human test-takers solved each one.⁷ Every problem was written new, so no model could have seen it during training.⁷ Comparing GSM8K and GSM1k scores isolates something renaming cannot: whether a model has actually seen this problem's wording before, not just problems shaped like it.

The gap runs as high as 8 percentage points, the largest in the paper's own table, for a model called Yi-6B-Chat.⁷ Other named drops include 7.2 points for a fine-tuned Mistral variant and 6.3 for Phi-2.⁷ The paper also reports a modest positive relationship between memorization and gap size. It found a Spearman correlation of 0.36 between how likely a model was to generate GSM8K's own test text word for word, and how large its GSM8K-to-GSM1k gap turned out to be.⁷ Models that showed more sign of memorizing GSM8K tended to lose more ground on the unseen twin.

The paper builds in two guardrails against over-reading its own result. Phi-2's score fell 6.3 points moving to GSM1k, but it still solved more than half of GSM1k's problems, which it could not have memorized: a real drop coexists with real reasoning ability.⁷ The gap is not purely contamination either. Mixtral-8x22b and its instruction-tuned variant showed similar overfitting despite very different likelihoods of having generated GSM8K's exact text. That points to other causes, such as picking a checkpoint because it already scored well on GSM8K.⁷ One caveat cuts the other way: GSM1k is not perfectly matched to GSM8K's difficulty, and some models, especially Anthropic's Claude family, score better on the newer set, consistent with GSM1k running slightly easier.⁷ A negative gap is not, on its own, proof of zero overfitting.

The contamination check most labs actually run

GPT-4's report says OpenAI "conducted contamination checking to verify the test set for GSM-8K is not included in the training set."² That kind of check is standard, usually run by matching strings: searching training data for close overlap with a test problem's exact wording.⁸ A 2023 paper from UC Berkeley and LMSYS tested how well that standard method holds up, using GSM8K as one of three benchmarks checked alongside MMLU and HumanEval.⁸ Their finding: rewording or translating a test problem is enough to slip past a string-matching check. Once that happens, a much smaller model can overfit to the reworded test data and score close to a model many times its size.⁸

None of this proves any particular lab's own contamination check failed. It shows that the standard method a check like OpenAI's typically runs would not catch a GSM8K test problem sitting in training data with its names and numbers changed. That is exactly the kind of change GSM-Symbolic showed still moves scores.⁵⁸ A check that passes shows the exact wording was absent, not that a renamed twin was too.