Chatbot Arena's Elo is a preference vote that pays out for style

The label says chess, the estimator says regression

Chatbot Arena, run by LMArena and begun as the LMSYS project, pits two unnamed models against each other: a user types a prompt, reads two anonymous answers side by side, and votes for the winner or a tie. The platform reported over 240,000 such votes by the time its authors wrote the method up,¹ and from them it prints a leaderboard headed by a column it labels Elo, after competitive chess.

The label is a holdover. The founding paper does not run chess Elo, which nudges a rating after each game by a fixed step. It fits a Bradley-Terry model: the probability that model m beats model m' is a logistic function of the gap between their ratings, and the ratings are estimated by maximum likelihood over all the pairwise votes at once, ordinary logistic regression.¹ The authors are explicit: earlier versions "reported different ranking scores, such as the Elo score," and they moved to Bradley-Terry coefficients because those "are better for the purpose of statistical estimation."¹

Grant the instrument its due first: the paper reports that its crowdsourced questions are diverse and discriminating and that the crowd's votes agree with those of expert raters.¹ A rank that falls out of this is a real quantity, a summary of which answers a population of voters preferred, fit by a sound estimator. The trouble is what that quantity quietly folds together.

The interval the board prints, and the crowd that fills it

Here the paper and the deployed board diverge. The paper works out two ways to put error bars on a coefficient, the pivot bootstrap and the sandwich robust standard error, and prefers the sandwich intervals for coming out smaller in large samples.¹ The live board has long reported plus-or-minus intervals of the bootstrap kind, so the shorthand that "the Arena uses bootstrap confidence intervals" describes the board, not the estimator the paper recommends.

Either way, the intervals are wide enough to matter. In its own defense, LMArena notes that scores like 1069 (±27) and 1054 (±18/22) have overlapping intervals, so adjacent ranks can be statistically tied.² A single place can be noise. Nor are votes spread evenly: the platform samples pairs actively, steering battles toward the least certain comparisons, so which models met, and how often, is itself a choice.¹

Then there is who votes, and on what. There is no fixed test set: the prompts are whatever a self-selected, largely technical crowd types, and both the crowd and the field drift over time.¹ A rating over one population is a claim about that population's tastes, which is worth knowing and not the same as which model is more capable.

What a vote quietly buys

If the rating tracks preference, the next question is what preference tracks, and the operators have measured part of the answer. LMArena runs the same logistic regression a second time with additive terms for style: an answer's length, and its use of markdown headers, bold text, and lists.³ Length dominates, at 0.249 when length and markdown are controlled together and 0.267 for length alone; the markdown elements are small, 0.019 to 0.056.³ Longer, better-formatted answers win votes at a measurable rate that has nothing to do with being correct. Subtract that premium and the board reorders, two numbers from the same instrument that disagree.

Grok-2-mini falls twelve places once length and formatting are held fixed; Claude 3 Opus climbs six.³ LMArena's own summary: "it's not just what you say, but how you say it."³

The vote pays out for a subtler thing too, agreement. Studying sycophancy, Anthropic researchers found that "both humans and preference models prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time."⁴ A voter is a human preference model, so part of what moves a vote is a confident, agreeable tone rather than a right answer.

The anonymity you can break

Anonymity is what turns a battle into a preference instead of a brand survey, and it does not hold as firmly as it looks. Researchers studying vote manipulation analyzed roughly 1.7 million historical Arena votes and built a de-anonymizing classifier that identifies the hidden model from the wording of its response.⁵ Once it can be picked out, votes can be aimed at a chosen target, and the same work shows that rigging only hundreds of fresh votes moves a ranking.⁵ The defense against a determined provider is the cost of the effort, not a property of the estimator.

Stuffing the ballot is not even required. Because length, formatting, and agreeableness reliably win battles, a provider can simply train toward them, a legitimate optimization against what the Arena measures and a model built to win preference under these incentives rather than to be more useful.

The 1417 that shipped as a different model

The clearest case came in spring 2025. Meta's Llama 4 launch post advertised "an experimental chat version scoring ELO of 1417 on LMArena,"⁶ a variant that ranked second at submission, behind Google's Gemini 2.5 Pro.⁷ The weight sits in two words: experimental chat version. The 1417 belonged to a model built for the battle, not the weights anyone could download.

The experimental version produced verbose answers with frequent emoji; the released Maverick gave concise, emoji-free replies, and once run through the same battles the open-weight model was reported around 32nd.⁷ That rank comes from the public battle record, not a figure Meta or LMArena published, so it stands as reported. The exact place is soft, the size of the gap is not: the launch post and the download sat about thirty places apart.

LMArena released more than two thousand battle results for review and said Meta "should have made it clearer that Llama-4-Maverick-03-26-Experimental was a customized model to optimize for human preference," and that "Meta's interpretation of our policy did not match what we expect from model providers."⁸ The statement ran on a platform that blocks automated readers; an unaffiliated engineer quoted it in full the same day.⁹ This is a disclosure failure, not proven cheating: Meta's post did use the word experimental, its policy then permitted variants, and a spokesperson said the company routinely tests custom ones.⁷ What the episode demonstrates is narrower and more useful: the metric rewards style strongly enough that a variant tuned for the vote earns a headline rank the shipped weights do not.

A broader critique tried to make that pattern systemic. "The Leaderboard Illusion," by researchers at Cohere Labs, Princeton, MIT and elsewhere, charged that a few providers get privileged access: 27 private Llama variants tested before the Llama 4 release, and an estimated 19.2% of test prompts going to Google and 20.4% to OpenAI against 8.9% for 41 open-source models combined. It argued that lifting a model's share of Arena data from 0% to 70% more than doubled its win rate on a static benchmark, a 112% relative gain, and that 205 of 243 public models had been silently retired.¹⁰ LMArena rejected the core point by point: its own figures put open models at 40.9% of data, not 8.9%, and it called the overfitting chart a simulation drawn from Gaussians with "an arbitrarily chosen variance" and the 112% figure an artifact measured on a static set rather than the live board.² It did concede governance ground: it would state that providers may test multiple variants, clarify how models are retired, and mark some scores provisional until 2,000 fresh votes arrive.² The asymmetry is contested; the concessions are real.

Reading a rank you did not compute

A rank on Chatbot Arena summarizes one crowd's votes under one set of incentives, and reading it means recovering them. Two are visible on the board: whether the figure has had length and formatting subtracted, which moved models as much as twelve places under the operators' own control,³ and whether its interval swallows the model just above, which LMArena's figures show it often does.² The rest sit behind the number. It was earned by some particular variant that may not be the model on offer, the way the 1417 belonged to a Llama 4 chat version that never shipped;⁶ and it rests on however many votes a self-selected crowd chose to cast, on prompts no one fixed.¹

Preference measured this way is a genuine quantity, narrower than the leaderboard's fame suggests: which answers this crowd liked, formatting and confidence included. A launch slide reports a place; the paper behind it reports an estimator, an interval, and a style term the place omits. Carrying all three turns "Model X is first on the Arena" from a slogan into a claim with a known method, a known population, and a margin.