Tokenize the same sentence by letters instead of words, and its perplexity falls 66 percent

A score with no task attached

A language model's whole job, at each point in a piece of text, is to output a probability for every token that could come next. Perplexity is the name Jelinek, Mercer, Bahl and Baker gave that quantity in 1977: how many equally likely choices a model faced at each step.¹ It is the exponential of the average of a model's negative log-probabilities, written PPL(X) = exp{-1/t ∑ log pθ(x₀ | x<₀)}, the same average negative-log-probability-per-symbol Shannon had already derived for entropy three decades earlier.²³ It is also equal to two raised to the model's cross-entropy on the text, the same entropy Shannon defined as H = -∑p log p.²⁴ A perplexity of 20 behaves like a model choosing uniformly among 20 options at every position.

That is a different kind of number from every other measurement this desk has covered. A benchmark like MMLU or GSM8K hands a model an invented problem and grades whether the answer is right. Perplexity asks nothing and grades nothing. It runs a model over a held-out test corpus, text nobody trained it on, and records the probability it assigned to what actually happened next.⁵ WikiText-103, one of the standard corpora built for this job, is 103 million tokens of Wikipedia-quality prose kept fixed as a common benchmark.⁵ GPT-2's paper, for instance, cites its scores on WikiText-103 and the Penn Treebank.⁶

The arithmetic, one sentence at a time

Take a toy model, tested on the four-word sentence "the cat sat down." At each word the model does not check whether it is right. It puts out a probability for every word it knows, and the test reads off whatever probability it assigned to the word that actually came next.

Five steps turn those four numbers into one perplexity, run strictly in order. First, read off the probability the model assigned to what actually happened at each position: 0.20, 0.15, 0.40, 0.25. Second, take the base-2 logarithm of each one: −2.32, −2.74, −1.32, −2.00. Third, negate each logarithm, so a confident guess produces a small number and a surprised one produces a large number: 2.32, 2.74, 1.32, 2.00. Fourth, average those four negated logarithms: (2.32 + 2.74 + 1.32 + 2.00) ÷ 4 = 2.095 bits per word. Fifth, raise 2 to that average: 2^2.095 ≈ 4.27. That is the sentence's perplexity: on this sentence, the model behaved as if it were choosing among about 4.27 equally likely words at every step.

The test set decides what gets averaged

The four probabilities above came from running the model over a specific text and recording what it thought would happen next at every position. Whoever reports a perplexity picks that text and publishes it alongside the number. GPT-2's paper names WikiText-103 and the Penn Treebank as the two benchmarks its scores refer to.⁶

Watch the number move when the test set does. Extend the sentence to "the cat sat down slowly," and suppose the model assigns "slowly" a probability of only 0.05, a fifth negated logarithm of 4.32 bits. Average five negated logarithms instead of four: (2.32 + 2.74 + 1.32 + 2.00 + 4.32) ÷ 5 = 2.54 bits per word, and 2^2.54 ≈ 5.82. The perplexity rose from 4.27 to 5.82, and the model got no worse at anything. The test set now includes a word the model finds harder to predict, and perplexity averages over whatever the test set contains.

The same logic makes perplexity dataset-dependent in a way a task score usually is not. A corpus of routine news writing produces a lower perplexity, on the identical model, than one full of rare words and unusual syntax, because rare words are harder to predict.⁷ Two labs both reporting "a perplexity of 15" have made the same claim only if they measured it on the same corpus.

Even holding the model, the tokenizer, and the test set fixed, the scoring procedure still moves the number. One walkthrough scores GPT-2 at a perplexity of 19.44 on WikiText-2 in 1,024-token strides, and at 16.44 in strides of 512 instead.³ The shorter stride gives the model more context, more often. All of this is fair, as long as the test set and procedure stay the same. The condition almost nobody checks is the third: whether both scores came from the same tokenizer.

The same sentence, cut two ways

Perplexity's formula divides total surprise by however many steps the test was scored over. A "step" is whatever the tokenizer decided a token is. Two tokenizers can cut the identical sentence into a different number of tokens. Because the formula divides by that count, changing the count changes the reported perplexity, even when the model's grip on the sentence has not moved at all.

Take "the cat sat down" under its four-word tokenization: 8.38 bits of surprise total, the sum of the four negated logarithms above, divided across 4 tokens, for a perplexity of 4.27. Now score the identical sentence character by character. "The cat sat down" is 16 characters, including spaces. Give the character-level model exactly as much information as the word-level model had: the same 8.38 bits of total surprise, spread across 16 steps instead of 4. The average falls to 0.524 bits per step, and 2^0.524 ≈ 1.44.

Both rows agree, exactly, on how likely "the cat sat down" was: 8.38 bits, however it gets divided up. Slicing the sentence into sixteen characters spreads that unchanged total across four times as many steps, so each step looks less surprising and the average comes out lower. A perplexity of 1.44 and a perplexity of 4.27 here describe identical knowledge. Ranking one model above another on raw perplexity, without checking the tokenizer, can be exactly this comparison wearing a scoreboard.

The same effect shows up in production models. An independent benchmark ran nineteen of them through the same test text, correcting each one for its own token count.⁸

None of those shifts said anything about which model predicts text better, only how many pieces its tokenizer cut the same text into.

The fix is to stop dividing by tokens: score bits per byte, the same total surprise divided by the text's raw byte count, a number every tokenizer agrees on. The Pile, an 800-gigabyte training benchmark, reports bits-per-byte for exactly this reason.⁹ The idea predates any system that needs it: a word-trigram model tested on 5.96 million characters of the Brown Corpus in 1992 put written English's entropy at 1.75 bits per character.¹⁰ That figure holds regardless of tokenizer, because it counts characters, not tokens.

When it actually misled someone

The clearest real case of this trap is not a thought experiment. In 2025, researchers testing Llama 3 on Japanese ran it against psychological predicates, words like "kanashii" (sad) that Japanese grammar requires to carry an evidential marker when used about someone else's feelings. On certain character combinations in these sentences, Llama 3's tokenizer fell back to raw UTF-8 bytes instead of its normal subword pieces. That fallback split those exact spots into far more tokens than the same words received elsewhere in the sentence.¹¹

That inconsistency did not land evenly. Fixing the tokenization dropped the reported perplexity of the grammatical sentences by a factor of roughly 28. The ungrammatical sentences, which triggered the byte fallback less often, dropped by only about 1.8 times under the identical fix.¹¹ That gap was large enough to flip which sentence type scored better. On the raw, inconsistent numbers, Llama 3's perplexity favored the ungrammatical sentence. Once tokenization was made consistent, the model's actual preference for the grammatical sentence came through.¹¹

Llama 3 had learned this piece of Japanese grammar. Its raw perplexity score said the opposite, and the error traced to an uneven tokenizer, not to the model's own knowledge. A team judging the model's Japanese grammar from the raw number would have reached the wrong conclusion with full confidence, because nothing about a perplexity score flags a broken tokenization.