A model can't count the words it's writing

Ask for ten words, get a hundred

In an informal test run in 2023, one practitioner sent a model 150 separate requests for a specific word count on a specific topic, five target lengths across three topics, ten runs each. Short requests came back badly overshot: a ten-word request produced roughly a hundred words. Long requests came back undershot, output settling around 600 to 700 words even when the prompt asked for 1,000 or 2,000.¹ The model wasn't confused about the topic. Every response was fluent and on-topic. It just never landed on the number.

That's strange, because hitting a number sounds like the easiest instruction there is. A person asked for 100 words counts as they go, or writes past it and cuts back until the count is right. A model that can define a term, translate a paragraph, or draft working code can't do what a word processor has done automatically since the 1980s. The reason isn't that counting is hard arithmetic. It's that nothing in how the model writes ever performs the count.

Nothing is tallying while the model writes

A model does not draft an answer and then check it against a limit. It writes forward, one token at a time. At each step the entire model does one job: given everything written so far, produce a distribution over what could come next, and append whichever piece is chosen.² The model's own training objective states this directly. The probability of the whole output is the product of each next piece's probability, given only the pieces before it.³ Nothing in that equation tracks how long the output has gotten, or how much room is left.

The published lesson on how generation runs shows the other half of the same fact. The moment a token is chosen and appended, it becomes ordinary input for the next step, exactly like every word that came before it. There is no separate scratch space holding a draft that could still be edited. That's why a model can't do what a person does under a word limit: get to the end, notice it ran long, and trim. By the time "long" would be noticeable, the words that made it long are already part of what every later word was conditioned on. Cutting them now would mean generating a different answer from scratch, not editing this one.

Even a counter would count the wrong thing

Suppose the architecture did carry a running tally. It would still be counting the wrong unit. The vocabulary a model generates from isn't built out of words. It comes from byte-pair encoding, a method that merges frequently occurring character sequences into reusable pieces regardless of where a word boundary falls. The method was chosen because a fixed word-level vocabulary can't cover every name, compound, and rare word form a language actually produces.⁴ The GPT-2 paper gives a small, telling example of what that produces: "dog," "dog.," "dog!," and "dog?" each occupy separate vocabulary slots. The merging procedure treats trailing punctuation as part of the pattern, not as noise to strip before counting.³

A companion lesson works through what this means for a single word. The model never sees "strawberry" as eight letters; it sees a short sequence of integer IDs. Counting those IDs is a different operation from counting the letters, or the words, a person would count by eye. A word-count instruction asks for the reader's unit. A model tracking its own output, even with a hypothetical tally, would be counting in the writer's unit instead. That unit sometimes takes several tokens to cover a single word, and sometimes folds a word and its punctuation into one token. A tally kept in tokens doesn't convert cleanly into a word count without doing the counting again, in the unit that was actually asked for.

Trained to run long before it ever sees a limit

Even without a counting problem, a model would still lean toward one particular kind of miss: too long. A 2024 study of the reinforcement-learning stage of RLHF training found that a learned reward model scores longer answers higher largely independent of whether the extra length says anything useful.⁵ The researchers isolated the effect directly. They swapped the normal, content-aware reward model for one that scored nothing but output length. Reinforcement learning against that content-blind reward matched or nearly matched a policy trained against the real reward model, on two of three benchmarks tested: 56% against 58% win rate on one question-answering set, 64% against 63% on a dialogue set.⁵ A policy optimized this way hadn't gotten better at answering. It had gotten longer, and length alone bought most of the apparent improvement.

A separate lesson traces this same reward-shaping pattern to the habit of writing in bullet points and headers whether or not the answer calls for them. It's the same mechanism, one level up: post-training rewards structure and length as shallow signals of quality, regardless of the instruction that produced them.⁵ One caveat matters before going further: this finding was measured with no length instruction anywhere in the prompt. It describes how a model writes by default, not how it behaves once told to hit a number.

How big the miss gets, and what still isn't measured

Researchers test this directly with what one 2023 benchmark calls a "verifiable instruction": a rule simple enough for a program to check automatically, rather than judged by a human or another model. Length rules are one category that benchmark tests, alongside things like which language a response uses or whether a keyword appears.⁶

A 2024 benchmark built specifically to isolate length gives the clearest number for how bad the miss gets. Researchers took AlpacaEval's ordinary instruction-following prompts and prepended a hard word ceiling to each one, sized to be tight but achievable. They then checked automatically whether the response stayed under it: no judge, no partial credit, just a word count against a limit.⁷ On 802 such prompts, GPT-4 Turbo went over its own ceiling 49.3% of the time.⁷ A second, larger 2025 benchmark tested 26 models under an even stricter "hit the number exactly" rule and found 23 of them scoring below 60 out of 100 on a length-accuracy metric built for the test.⁸

The same 2024 benchmark tested nine other models on the identical task, and the spread is itself informative:

Instruction tuning clearly helps: the untuned base model at the bottom of the table misses nearly nine times as often as the best tuned model at the top. It doesn't come close to solving the problem. Every model in the full study, tuned or not, missed at least some of the time. The gap between the best and worst performers is itself an open question: why some models track length so much better than others, when none of them has an architectural counter to lean on. No source in this record fully answers it.

None of the studies above measures the whole chain end to end. The reward-model study measured default writing with no length instruction present. The benchmarks that measure explicit misses don't test whether the same models were also biased toward writing long by default. No experiment in this record starts with a length instruction, checks how the same model writes without one, and attributes part of the miss to that inherited habit. Reading the two kinds of finding together still points at a plausible compounding. A model already biased toward writing long starts an explicit word-limit answer further from the target. The architecture then gives it no way to catch that drift as it writes, because nothing is tallying and nothing already written can be taken back. That reading is an inference from what the studies separately show, not a result any one of them reports.

The same weakness surfaces anywhere a model has to hit an exact count while it writes, not only a word limit. A character limit instead of a word limit is the identical problem in a smaller unit. A list capped at exactly five items, a table with a fixed number of rows, a summary that must land in precisely two sentences: each asks for a tally kept in the right unit and checked before the text is called final. Generation offers none of that, in any of these forms.