o1 scored 74% on the 2024 AIME on one try and 83% on a vote over sixty-four

How o1 reported two scores for one exam

When OpenAI introduced its o1 reasoning model in September 2024, it reported two scores for the 2024 American Invitational Mathematics Examination, a fifteen-problem high-school competition. Given one attempt per problem, o1 solved 74%. Given sixty-four attempts and graded on the answer that came up most often, it solved 83%.¹ The model did not get better between the first attempt and the sixty-fourth. The scoring rule changed.

That second rule has a label, consensus@64, and several cousins. Self-consistency is the underlying method; cons@k, consensus@k, and maj@k are the numbers it reports, where k is how many attempts were sampled. Wherever the label appears, the score is a vote among a model's own tries rather than a single try graded once. The vote almost always scores higher than the single try.

How a majority vote becomes a score

A reasoning model does not jump to an answer. It writes out a chain of thought first, a run of intermediate steps in ordinary text, and gives its final answer at the end. It also does not write the same chain every time. At each word the model holds a probability distribution over what comes next, and a setting called the sampling temperature controls how much it gambles. At temperature zero it always takes its single most likely next word, so one prompt yields one fixed chain; this is greedy decoding. Above zero it samples, and the same prompt can produce a different chain, and a different final answer, on every run.²

A majority-vote score is built on that variation, in three steps.

1. Sample k chains

   Run the same problem k times at a nonzero temperature, collecting k independent chains of thought, each ending in its own final answer.²

2. Tally the final answers

   Strip the reasoning from each chain, keep only its final answer, and count how many chains landed on each distinct answer.

3. Report the plurality

   Score the answer that appears most often, the plurality or majority vote, and report whether it is correct as cons@k or maj@k for that k.

This is the self-consistency method Wang and coauthors introduced in 2022: replace greedy decoding with a diverse set of sampled chains, then keep the answer that most of them agree on. In their experiments they sampled forty chains per problem for a model like PaLM-540B, at a temperature of 0.7.²

Why the vote beats a single try

The smallest version of the effect fits in ten tries. Suppose one competition problem has the answer 24, and a model samples ten chains of thought for it: six reason their way to 24, three make the same wrong turn and reach 18, and one reaches 36. Keep a single chain and what you report is a matter of luck. Six times in ten you would report 24, three times 18, once 36. Take the most common answer across all ten and you report 24 every time. The vote lands on the right answer for one specific reason: the model reaches 24 more often than it repeats any single wrong answer.

The same effect holds at scale. On GSM8K, a set of grade-school math word problems, Wang and coauthors ran PaLM-540B two ways. Kept to a single greedy answer it scored 56.5%. Scored on a majority vote over forty samples it scored 74.4%, a gain of 17.9 points from nothing but sampling and counting.²

A fair objection: should not one good, directed attempt beat a scattered vote? When o1's numbers appeared, one analyst argued that its single-attempt score ought to at least equal its consensus@64 score, on the reasoning that a model searching for the right answer should beat one that merely samples and tallies.³ The measured numbers run the other way in every case here, and the ten tries say why. A vote is not a search for the best chain. It is a way to cancel out the chains that wander off, and it works whenever the wanderers disagree with each other while the correct chains agree.

That is also the boundary of what a vote can do. Turn the counts around: if seven of the ten chains reach 18 and only three reach 24, the plurality is 18 and the vote is confidently wrong. Sampling more chains does not help, because the model is not scattering around the right answer, it is converging on the wrong one. A majority vote lifts a score on problems the model half-knows. It cannot rescue a problem the model is sure about and wrong about.

The vote climbs to a ceiling

Feeding a model more tries does raise a different number without any ceiling in sight. Brown and coauthors measured coverage: the fraction of problems for which at least one sampled chain is correct. On a software-engineering benchmark, coverage for one model rose from 15.9% at a single sample to 56% at 250 samples, and kept climbing as they sampled further.⁴ Counting whether any of k tries is correct is the rule behind pass@k, a scoring convention used for code, and it is not what a vote measures.

Coverage asks whether the right answer is anywhere in the pile. The vote asks whether the right answer is the most common thing in the pile. The two come apart as the pile grows. Brown and coauthors found that where no automatic checker can confirm an answer, majority voting and similar selection methods plateau after several hundred samples and stop tracking the sample budget, even while coverage keeps rising.⁴ The correct chain can be sitting in the pile, and if the wrong chains outnumber it, the vote never picks it out.

Reading a vote as if it were one try

Every extra try in a vote is real computation. A consensus@64 score is the accuracy of a system that generates sixty-four full chains of thought for each problem and then votes; a single-try score is the accuracy of a system that generates one.¹ The vote can cost sixty-four times the inference of the single try. Both can be honest numbers. They are not the same measurement, and the trouble starts when they are read as though they were.

Inside the report that produces them, the two usually sit side by side. Take Minerva, Google's 2022 math model, on MATH, a set of 12,500 competition-mathematics problems.⁵ Its Table 2 lists the 540-billion-parameter model at 33.6% for a single greedy answer and 50.3% for a majority vote over sixty-four samples, both labeled, one beside the other.⁶ Reported that way, the vote is honest.

The same table carries the trap. Farther down the same MATH column sits the prior published state of the art: 6.9%, a single-try number.⁶ A reader scanning the column sees 6.9% and 50.3% in one frame and reads a leap of more than sevenfold in skill. Part of that leap is real and part of it is the gap between scoring one answer and scoring the vote of sixty-four. The like-for-like comparison to the old 6.9% is Minerva's own single-try 33.6%, still a large jump, and it is the number that tends not to travel.

The mismatch does its real damage across models rather than within one table. DeepSeek reported its R1-Zero model at 86.7% on the 2024 AIME by majority vote, and said the score matched an OpenAI o1 version.⁷ That comparison is fair, because DeepSeek set its cons@64 against o1's cons@64, one vote against another.⁷ The 86.7% stops being fair the moment it is quoted next to a single-try score, or quoted with the sixty-four dropped, so that a vote among sixty-four tries reads as one model's one-shot skill.

The layer of the field that compares models against each other has moved to close that gap. Artificial Analysis, which ranks models on a public leaderboard, scores every benchmark at pass@1, one attempt per problem, and states plainly that reporting the best of k attempts would inflate the numbers.⁸ Where systems are compared head to head, the single-try number is the standard now, because a voted number is not comparable across systems that paid different amounts of compute to produce it.