Adding six wrong answers to each MMLU question fixed its guessing problem

The number that took MMLU's place

Meta's model card for Llama 4, published in 2025, lists a benchmark called MMLU-Pro near the top of its benchmark tables.¹ The switch was not cosmetic. By 2024 the strongest models were clearing the high 80s on MMLU, the 2020 test of general knowledge across 57 school and professional subjects, and a test almost everyone passes has stopped sorting anyone.² Hugging Face reached the same conclusion and dropped MMLU from its Open LLM Leaderboard in favor of MMLU-Pro, calling the older test "noisy" and "too easy."³

MMLU-Pro was released in 2024 by TIGER-Lab, a research group led by Wenhu Chen at the University of Waterloo, and it was built to repair three specific things MMLU got wrong.² Whether it repaired them is the question worth asking of any score, and MMLU-Pro turns out to have three different answers to it.

How each question gets built

MMLU-Pro is not written from scratch. TIGER-Lab assembled it in five passes over questions that mostly already existed.

1. Start from MMLU

   The build began with 13,937 questions drawn from MMLU.⁴

2. Drop the easy ones

   Eight open-source models answered every question, and any that more than four of them got right was cut as too easy. That removed 5,886 questions.⁴

3. Add harder sources

   New questions came in from STEM websites, TheoremQA, and SciBench, with GPT-4-Turbo pulling a short answer out of each worked solution.⁴

4. Expand four options to ten

   GPT-4-Turbo wrote six extra plausible wrong answers for each question, taking the four original choices up to ten.⁴

5. Review every option twice

   Human experts checked the answer keys, then Gemini-1.5-Pro re-examined every option to catch anything wrongly marked wrong, followed by a second human pass.⁴

What comes out is 12,032 questions across 14 disciplines.⁴ Of those, 6,810 came from MMLU. The other 43% are new, and they change what the test covers: Math is now the single largest discipline at 1,351 questions, ahead of Physics and Chemistry.⁴ So MMLU-Pro is not MMLU made harder. It measures a different, STEM-weighted subject mix, and a model's MMLU-Pro score and its MMLU score are grading partly different knowledge.

Ten options, and a reward for thinking out loud

Two of those five passes carry almost the whole design, and they aim at different weaknesses. The first is the jump from four options to ten. The guess rate is the score a model earns by picking answers at random, and on MMLU's four options it is 25%, one in four.⁵ Add six more wrong answers and that floor falls to roughly one in ten. TIGER-Lab states the purpose plainly: "by increasing the distractor numbers, we significantly reduce the probability of correct guess by chance."⁶

The second change is subtler. By filtering out questions the models already solved and tilting toward ones that need working through, MMLU-Pro rewards reasoning. Chain-of-thought is a prompt that tells a model to work through its steps before committing to an answer, rather than naming an option straight away. On MMLU that habit barely helped: GPT-4o scored 1.5 points higher with it than without. On MMLU-Pro the same model gained 19.1 points from it.⁴ The wider gap is evidence that the questions reward reasoning, though not proof that the models reason: a follow-up test, MMLU-Pro+, found strong models still often lock onto a first-picked option instead of reworking the problem.⁷

Those are the two levers TIGER-Lab pulled. Whether they repaired MMLU's known flaws has three separate answers.

Three flaws, three different grades

The guessing floor is the one repair that worked by arithmetic. Ten options instead of four cannot be talked around: a model guessing blindly now clears about 10% rather than 25%, and no choice of prompt or grader moves that number. The single qualification is small. Because 17% of questions carry fewer than ten options, averaging 9.47, the real floor sits just above 10% rather than exactly on it.⁴ Graded on its own terms, this repair is finished.

Label noise is harder to grade. It is the share of questions whose marked-correct answer is actually wrong, or that have no single correct answer, and MMLU had a real amount of it. In 2024 a team led by Aryo Gema re-annotated MMLU and estimated that 6.49% of its questions contain such errors.⁸ In one subject, Virology, 57% of the questions they examined had a problem, a third of them a flatly wrong answer key.⁸ The errors changed results: once the broken Virology questions were removed, Llama 3.1 405B moved from 16th place to 1st.⁸ MMLU-Pro's two-phase expert review is aimed squarely at this defect and surely reduces it. But no one has since measured MMLU-Pro's own error rate the way Gema measured MMLU's, so "reduced" is the most the evidence supports, not "removed."

Prompt sensitivity is how far a model's score moves when the questions stay fixed and only the wording of the surrounding instructions changes. Here TIGER-Lab measured its own repair: across 24 prompt styles, a model's MMLU score swung 4 to 5 points, while its MMLU-Pro score swung about 2.² That improvement is real and measured. It also has a hard edge that the measurement does not reach. Prompt wording is only one format a benchmark can be sensitive to, and a separate 2024 study shuffled the order of the answer options on MMLU and watched accuracy fall for every model tested, by 6 points for one and 27 for another.⁹ That study never ran on MMLU-Pro, and TIGER-Lab's robustness test never touched answer order. MMLU-Pro is steadier against prompt wording and simply untested against answer order.

What a gap between two scores is worth

Put those grades together and a rule falls out for the thing readers actually do with the number: compare two models by it. A small gap is the first place the rule bites, because a small gap can come from the measurement rather than the models. Meta's own Llama 4 card reports the same Maverick model at 62.9 and at 80.5, an 18-point spread, because one run is a pretrained model measured with five worked examples and the other an instruction-tuned model measured with none.¹ If the setup can move a single model 18 points, a 2-point gap between two models from different labs says nothing until you know both were run the same way.

Saturation is the second place. Two years on, the top of the MMLU-Pro board is crowding the way MMLU's did before it. As of August 2026 the leading models sit near 88%, the top four within a point of each other.⁶ That is roughly where MMLU stood when it stopped telling models apart. Near the ceiling, the point separating first from fourth is closer to noise than to a real difference in ability.

The number is worth trusting where its construction is strongest: a large gap between two models run the same way, on a mix that is now weighted toward STEM. A ten-point MMLU-Pro gap between two like-for-like runs is a real signal. A two-point gap, or any gap between numbers measured under different settings, is not something to rank on. And because the questions lean on STEM, a high MMLU-Pro score increasingly certifies a model's STEM multiple-choice more than the broad general knowledge the name still implies.