A model can recognize strawberry's letters and still miscount them

The wrong answer is not a spelling mistake

Treat the miss as a fact to be explained rather than a lapse to be scolded. It is not random. Fu and colleagues tested the behavior directly and found it systematic: models could recognize the letters in a word yet fail to count them, and they used strawberry as the motivating case.¹ A random slip would come and go. This one has a structure, and the structure points at how the word reaches the model in the first place.

So start where the word enters and work down. Each step names one real part of the system and asks what that part can and cannot do. The descent ends at a step where nothing below it would change the answer. That is the ground the lesson is after.

What the model receives instead of letters

Run strawberry through the tokenizer that GPT-3.5 and GPT-4 actually use, and the word comes back as three pieces: str, aw, berry. Each piece is then swapped for a fixed number from a lookup table. The three numbers are 496, 675, and 15717.² That triple is the whole of what the model gets. The ten letters never arrive.

You do not have to take the split on faith. Willison, walking through the same tokenizer, shows that any word can be dropped into a tool that prints its pieces and their numbers, and he points at the small surprise that teaches the most: many tokens carry a leading space, so the same word spelled with a space in front becomes a different, often shorter, sequence.³ strawberry is a sharp case of that. Written with a leading space,  strawberry stops being three pieces and becomes a single number, 73700, in that same tokenizer.² One common word, one opaque integer, no seam where the letters were.

The method behind the split has a name and an origin. Sennrich, Haddow, and Birch introduced byte-pair encoding for language models: rather than keep a word for every word, encode text as a sequence of subword units built by merging the most frequent adjacent pairs.⁴ GPT-2's authors put a version of it at the mouth of the model and described what it does to a word. Frequent strings survive as one token; infrequent ones get broken toward smaller pieces, so the scheme sits between whole words and bare characters.⁵ An earlier lesson walks through how that table is built. What matters here is the output: the model reads a row of integers, each an index into a vocabulary, and begins its work from there.

Counting the r's with only the token in hand

Now do the model's job yourself, with only what the model has. You are handed 15717 and asked how many r's it contains. The number does not contain letters. It is an index, an arbitrary address in a table, no more spelled than a page number is. To answer at all you must already know that 15717 stands for the piece berry, and then that berry is written b-e-r-r-y. Both are facts about the number, not contents you can read off it.

A part number in a warehouse catalog works the same way. You can learn, from handling orders long enough, that part 15717 is the strawberry-jam refill and even that its label carries a double r, but the digits 15717 hold none of that. The knowledge rides alongside the number, gathered from every time you saw the part and its label together. Set the catalog down before it misleads: the point is only that the spelling is a learned association attached to the token, never a string the token exposes.

The model does hold that association, and better than you might guess. Kaushal and Mahowald probed token representations and found they reliably encode which characters a token contains, with larger models encoding it more accurately, learned from the times training text happened to spell things out or vary how a word was cut.⁶ So the model is not blind to letters. It is statistically sighted: it can usually recover that berry has two r's. This is why a clean story of the failure has to be careful. The model often can tell you the spelling.

Which is exactly what Fu and colleagues found. Their models could recognize the letters in a word and still miscount them.¹ Recovering the spelling is one job. Walking the recovered letters in order and keeping a running tally is a second job, and nothing in the system runs it by default. A person counts r's with a loop: eyes move left to right, a finger holds the place, a count ticks up. The model has no such loop over characters. It has learned associations, produced in one pass, and a count that lives inside a piece rather than across it. Two r's sit together inside berry and one more inside str; none of the three tokens is a bare r.² The miss is what a system without a counting loop does when asked to count.

The same cut runs through the digits

If the boundary is real it should show up away from spelling, and it does. Singh and Strouse looked at how models tokenize numbers and how that changes arithmetic. GPT-3.5 and GPT-4 assign separate tokens to one-, two-, and three-digit chunks, so a long number arrives already grouped, while some other models split every digit on its own. Forcing the digits to be grouped from the right, the way people add, measurably improved addition accuracy.⁷ The same cut that hides letters inside a piece hides digits inside a chunk, and moving the cut moves the answer. The failure is a property of the boundary, not a quirk of one word.

It is tempting to push that further and say the miss must be worse for rare words, since those seem more alien to the model. The evidence does not support it. Fu and colleagues measured it and reported that the frequency of the word, and of its tokens, had no significant effect on the errors.¹ The tokenizer's own logic cuts the other way: a rare word falls toward smaller pieces, closer to bare characters, which exposes more of its letters, not fewer.⁵ The defensible claim is the structural one. The failure is systematic because it comes from the boundary itself, and it does not need the word to be rare.

Where the explanation stops being settled

Two things in this lesson are settled engineering. Text is turned into tokens before the model sees it, and the model operates on those tokens, not on characters. Everything above rests on those two facts, and they are not in dispute.

One thing is not settled: why newer models miss less often. The easy guess is that the tokenizer was fixed. For this word it was not. GPT-4o uses a newer tokenizer than GPT-2, and both split strawberry the same way, into st, raw, berry; only the specific numbers differ.² The letters are buried in the same places they always were. So the improvement, where it is real, has to come from somewhere else, and here the honest answer is a list of candidates rather than a cause. More training text that spells words out would strengthen the association the model already has. And a model can be prompted or trained to spell the word out first, one piece at a time, before it counts, which is a form of the intermediate-steps method Wei and colleagues showed lifts multi-step and arithmetic performance.⁸ Spelling the word out builds, in the open, the ordered list the model could not walk internally.

Even that is not a clean fix. Liberman tested reasoning models on a harder counting task and watched them fail, which is the honest brake on any claim that the newest systems have put this behind them; his advice is to distrust any current system on a task that comes down to counting. That models learn to spell at all, from text that never taught them letters directly, is striking enough that the field has a half-joking name for it, the spelling miracle.⁹ The miracle is that the association exists. The miss is what happens when you ask it to do a job an association cannot do.