At the top of the RewardBench leaderboard, higher scores predicted lower Chatbot Arena rank

The score is a fixed-pair accuracy

A reward model is the scoring function trained on human preferences that RLHF later optimizes a chatbot against. The earlier lesson on InstructGPT walks through how it is fit and used. RewardBench, released by AI2 on 20 March 2024, grades that scoring function directly, without training a chatbot with it.¹

The dataset holds 2,985 items. Each item is a prompt, a completion marked chosen, and a completion marked rejected. To score a reward model, the benchmark runs it on the chosen and rejected completions and asks which score is higher. The RewardBench paper states the rule in one sentence: "The prompt is then categorized as a win if the score of the prompt with the verified chosen completion is higher than that of the verified rejected completion."¹ The reward model's leaderboard score is its win rate on those 2,985 judgments, aggregated across the four sections.

Four sections split the 2,985 pairs. Chat holds 358 pairs, drawn from AlpacaEval and MT-Bench. Chat Hard holds 456, mostly LLMBar adversarial variants and MT-Bench Hard. Safety holds 740, from XSTest and refusal-eliciting prompts. Reasoning holds 1,431, from a mathematics dataset and code completions in six programming languages.²

Inside Reasoning, math counts as 984 rows, not 447

The public leaderboard number is the mean of the four section scores.¹ Each section score is a per-prompt weighted average within that section: a subset's accuracy is multiplied by that subset's example count, the products are summed, and the total is divided by the total example count. That is exactly what the released scoring code does.³

total_weighted_score += metrics[test] * example_counts[test]
section_scores[section] = total_weighted_score / total_examples

For three of the four sections the example_counts in that code match the released dataset. Reasoning is the exception. The Reasoning section contains 447 PRM-Math rows plus 984 HumanEvalPack rows (164 rows each in Python, JavaScript, Java, Go, C++, and Rust).² The scoring code assigns PRM-Math an effective example count of 984, matching the code total, so each math prompt's win or loss counts about 2.20 times as much as each code prompt's toward the Reasoning score.³ The paper describes the intent as making "math and code abilities... weighed equally". The mechanism is this upweighting.¹

The leaderboard's "Reasoning: N" figure a reader might quote is not the count of prompts scored. It is the mixing weight the aggregation uses.

The score never watched a chatbot train

What the number is, then, is agreement with a fixed labeling of chosen versus rejected on 2,985 items. What most public claims read into it is a different question: whether the higher-ranked reward model, run through the RLHF pipeline, would produce a stronger chatbot. RewardBench does not ask that question. AI2 said so at launch: the release post on 20 March 2024 concedes plainly that "future work on RM evaluations still needs to analyze the correlation of RM performance with downstream policy performance."⁴ The correlation was declared unmeasured on the day the leaderboard opened.

Seven months later, a Berkeley group ran the comparison. Frick and colleagues, at UC Berkeley and the team behind Chatbot Arena, trained nine reward models through a full RLHF pipeline. They then plotted each reward model's RewardBench score against the post-DPO Chatbot Arena rank of the chatbot it trained. Among the top-tier reward models the slope reversed: "we now see a negative correlation between RewardBench evaluation score on top models and downstream RLHF performance."⁵ The paper's Figure 4 shows the reversal graphically at the top of the leaderboard. The paper prints only the direction. It does not publish a Pearson value for the negative slope.⁵

Frick et al.'s own replacement benchmark, PPE, built from Chatbot Arena preferences and correctness-graded tasks, reports 77% Pearson correlation with downstream performance across their evaluation set. That is a measured value for the correlation the RewardBench release had said was still to be measured.⁵

What a top-scoring reward model still does

A reward model that ranks high on RewardBench can still prefer sycophantic answers over correct ones, and can still assign higher scores to longer completions than short ones of equal quality. Both patterns are measured on reward-shaped signals of the kind RewardBench grades.

Sharma and colleagues measured sycophancy on Claude 2's production preference model, the one Anthropic used to train Claude 2 through RLHF. On a hand-curated set of misconceptions paired with baseline truthful responses, the preference model prefers the sycophantic completion over the baseline truthful one on 95% of the items.⁶ When the truthful comparator is upgraded to a strong, helpful response on the hardest misconceptions, the same preference model still prefers the sycophantic answer 45% of the time.⁶ The 95% figure is bounded to that curated set against baseline truthful answers, not to all Claude 2 behavior. It remains the closest primary evidence that a production reward-shaped signal can prefer a smooth wrong answer over a correct one at measurable rates.

Singhal and colleagues took apart the length pull inside RLHF training on three preference datasets: WebGPT, Stack Exchange, and RLCD. Once completion length is controlled for, the share of the PPO reward gain that remains varies by dataset. On WebGPT, about 2% of the gain survives; on RLCD, about 27%; on Stack Exchange, about 53%.⁷ A PPO run that optimizes a length-only reward reaches simulated preference win rates within a point or two of the standard PPO run on every one of the three datasets.⁷ "Length is everything" is close to right on WebGPT and roughly right on RLCD. On Stack Exchange it is about half right. The sharp version of the finding travels less well than the WebGPT number that is most often quoted.

Park and colleagues found the same pull inside DPO, which RewardBench also scores as an implicit reward. Standard DPO produces responses about twice as long as the preferred completions in the training data, and on out-of-distribution samples the DPO implicit reward's dependence on length becomes severe: length explains 30 to 46 percent of the reward variance.⁸ A DPO variant with an explicit length regularizer recovers about 20 win-rate points on Anthropic HH and 15 on TL;DR when matched against the unregularized baseline at the same length.⁸

The v2 rebuild lowered the ceiling and kept the warning

By mid-2025 the top of the RewardBench 1 leaderboard had gone flat enough that AI2 replaced the benchmark. RewardBench 2, released 3 June 2025, drops the four-section two-completion format for a six-section four-completion one. The six subsets are Factuality (475 prompts), Focus (495), Safety (450), Math (183), Precise Instruction Following (160), and a separate Ties subset of 102, for 1,865 items in all.⁹ For the five non-Ties subsets, an item counts as a win when the reward model scores the single chosen response above all three rejected ones. The random baseline is 25%, not 50%.⁹

On the same set of top-tier reward models, average scores fall about 20 percentage points from RewardBench 1 to RewardBench 2.⁹ That is the concrete evidence that v1 had saturated at the top. A benchmark that no longer separates its best-performing subjects is not measuring them, and the drop is what discrimination looks like when it comes back. Ken Yeung's contemporaneous write-up in The Letter Two corroborated the release. It dated the launch to 3 June 2025 and reported that seventy reward models were evaluated on v2 against thirty on v1. It also quoted the release's own caveat that "using a reward model from a different model family than the policy model could negatively impact performance, even if it ranks highly on the benchmark."¹⁰

Across seven downstream tasks (GSM8K, MATH, IFEval, AlpacaEval 2, BigBenchHard, PopQA, and HumanEval+), RewardBench 2 average score correlates with average best-of-N sampling accuracy at Pearson 0.87.⁹ That figure measures best-of-N, where the reward model is used to pick the best of several sampled candidates from a fixed policy. It is not a PPO training-run result. On PPO the v2 paper is explicit: "benchmark scores are a prerequisite for strong training with RLHF, but they are not sufficient", and "the best reward model for RLHF is dependent on one's training setup."⁹