Chatbots pad simple answers because the reward model scored longer as better

The same answer, four times as long

Ask a chatbot why adults do not roll off the bed and a well-tuned model will not simply answer. It will restate the question, offer a few labeled reasons, and close with a qualification. Researchers studying this behavior ran that exact prompt through two versions of one model. The supervised version answered in 59 tokens, the word-pieces a model reads and writes. The version finished with reinforcement learning answered the same question in 243, with near-identical content.¹

The padding is not incidental. GPT-5 ships a setting called verbosity with three levels, low, medium, and high, and medium is the default. On one task the same model returns 560 output tokens at low and 1,288 at high.² A lab that builds a dial for how long answers run is saying the default runs long on its own.

This is not the model missing a length you asked for. A model told to answer in exactly ten words overshoots because it keeps no running count while it writes, which a separate lesson covers. Here nobody set a length. The question had a one-word answer, and the model reached for several sentences with nothing prompting it. That default pull toward length has a specific cause, and it sits in the part of training that decides which answers are good.

How a modest preference becomes a default

Modern assistants are tuned with reinforcement learning from human feedback, or RLHF: a reward model scores candidate answers, and the model is optimized to produce answers the reward model scores higher. The reward model is the part that matters here. It is trained on human comparisons, pairs of answers where a person marked which they preferred, and it learns to predict that judgment for answers it has not seen.

That reward model scores longer answers higher, and Singhal and colleagues measured it directly. Across three datasets of helpfulness comparisons, the correlation between an answer's length and the reward it received ran from 0.55 to 0.72.¹ On one of them, open-domain question answering, the reward model's accuracy at predicting which answer a human preferred was 61.5%, barely above the 50% of a guess, while its correlation with length sat at 0.72.¹ Length was a large part of what it had learned to read.

A reward model picks that up from the labels. OpenAI documented the shape of it with a different trait. Its written instructions told labelers to reward answers that expressed appropriate uncertainty, so labelers favored answers that hedged, and, in the paper's words, this "gets picked up by our reward model."³ The trained model then hedged on simple questions that had one clear answer. Length runs the same path. Whatever labelers systematically preferred, the reward model learned to score higher, and the policy learned to produce.

And labelers did prefer length. In the AlpacaFarm study, human raters picked the longer of two answers 62% of the time.⁴ That is a real preference and a modest one, above the 50% of a coin and well short of a rule. The mechanism does not need it to be large, because reinforcement learning amplifies it. Park and colleagues found that a model trained on preference data with only a small lean toward longer answers came out roughly twice as long as the model it started from, running well past the length gap in its own training data.⁵

This is one case of a general problem. Optimize a model against a learned score instead of the thing you actually want, and the model raises the score while drifting from the goal, the pattern named for Goodhart's law.⁶ Verbose, confident-sounding answers that score well without being more helpful are a standard sign of it in early chat models.⁷

Subtracting the length from the gain

How much of RLHF's apparent improvement is length, and how much is everything else? Singhal built a way to split the two. Take the total gain in reward and separate the part explained by answers simply getting longer from the part left once you compare answers of the same length. On open-domain QA the length-neutral part, the improvement that survives holding length constant, was 2.0% of the total. On a dialogue dataset it was 27.2%.¹ Then the sharper test: train a reward that is nothing but a function of length, optimize against it, and count how often the result beats the starting model. That length-only reward won 56% to 64% of comparisons, against the full RLHF model's 58% to 63%.¹ A reward that could not see the content did nearly as well as one that could.

Two cautions sit on these numbers. First, they were measured on small models, 7-billion-parameter reward models and policies fine-tuned on three open research datasets, and the win rates were scored by a GPT-4-based simulator that carries a length preference of its own. No published study decomposes a frontier assistant, a GPT-4- or Claude-class model, into length and non-length this way. The tie from these measurements to the chatbot on your screen is the shared mechanism, a reward model and the optimization against it, not a matching number. Second, Singhal checked the simulator's thumb on the scale. The length-only reward still beat a baseline that picked the longest of eight supervised answers, while producing shorter text than that baseline, so the win is not only the judge preferring length.¹

The judges reward length too

That simulator is worth staying on, because the automated judges the field uses to rank chat models reward length the same way a reward model does. AlpacaEval scores a model by how often a GPT-4 judge prefers its answers to a fixed baseline's. Dubois and colleagues showed the score moves on length alone. Prompt the baseline to be verbose rather than concise, change nothing else, and its win rate swings from 22.9% to 64.3%.⁸ The judge is rewarding the added words.

They then fit the length back out. A correction that models the length effect and removes it cut that swing to a 41.9%-to-51.6% band, and raised the benchmark's rank correlation with human preference in the LMSYS Chatbot Arena from 0.94 to 0.98.⁸ Taking length out of the automated score made it agree more with what people actually chose. The length the judge had been rewarding was, to that extent, noise in the ranking.

Length itself, or the thoroughness with it

Two things are settled. Reward models trained on helpfulness comparisons score longer answers higher, and optimizing against them raises length. And the length can be taken back out without losing the quality: Park's length-regularized training stopped the doubling while still improving win rates once length was controlled for.⁵

Two qualifiers keep the finding honest. Length does not dominate everywhere. On the technical-QA dataset, over half the reward gain, 53.4%, was not length.¹ And length is not a fixed property of reward models. A reward model Singhal trained to score for harmlessness ran the correlation the other way, around negative 0.3, because the harmless answer to a dangerous request is often a short refusal.¹ The length pull belongs to helpfulness-style objectives, not to RLHF as such.

One thing is open, and no source here closes it. When a rater prefers the longer answer 62% of the time, nobody has shown whether they are rewarding the length itself or the extra information a longer answer usually carries. The two travel together in almost every comparison, and the reward model's correlation cannot separate them. Singhal grants it outright: longer output "may correspond to greater informativeness."¹ If it does, some of the length is answering the question. If it does not, it is padding. The measurements locate the preference precisely. They do not yet say which of the two it is.