The token a model can't repeat back

The token that comes back as something else

In early 2023, two researchers asked a GPT-3 model to do the simplest thing a language model can do. They typed, in effect, "Please repeat back the string SolidGoldMagikarp to me." The model answered distribute.¹ Not a refusal and not an error message. A different word, returned with full confidence.

The string was not alone. Working at a research program over two months, Jessica Rumbelow and Matthew Watkins ran a clustering algorithm over one model's token embeddings. It pulled out a cluster of about 141 strings that sat closest to the center of the whole vocabulary. Asked to repeat any of them, GPT-3 and ChatGPT failed in the same odd family of ways: substituting another word, dodging the question, hallucinating, claiming they could not, and in some cases insulting the user. Rumbelow and Watkins called the strings anomalous, weird, and unspeakable.¹ Writers outside that forum soon reported the same behavior, citing back to the original posts.²

These are the tokens the popular version of the story calls glitch tokens. The obvious first thought is that they are exotic strings the model simply never saw. That thought is close to right, and the rest is a matter of making it exact. We can work backward from the wrong word to the one decision that produced it, one step at a time.

The vocabulary is printed before the model trains

Start with what SolidGoldMagikarp is to the model. It is not a word and not seventeen letters. It is a single token, one entry in a fixed list of 50,257 that GPT-2 and the models built on it read text through.³ The model never sees characters. It sees token numbers, and that list of tokens is decided in advance.

The list is built by frequency. The method, byte-pair encoding, starts from single characters and repeatedly merges whichever adjacent pair occurs most often across a big pile of text. Common runs like the and and collapse into one token, while rare runs stay broken into pieces.³ Feed that process enough text in which some string keeps appearing, and it earns its own token.

A vocabulary built this way is like a phrasebook printed before a trip. The phrases are chosen from one body of text, the book is bound, and nothing that happens once you land can change what is on the pages. In the model this is literal. The tokenizer loads two files from disk, the token list and the ordered merge rules, and uses them unchanged no matter which trained weights sit behind it.⁴ The vocabulary is a finished artifact before the model has learned anything. That much is settled engineering: a token can be in the book while the model behind it is a total stranger to that token.

Where SolidGoldMagikarp came from

So where did the strangers come from? Watkins traced them one by one. SolidGoldMagikarp, TheNitromeFan, and RandomRedditorWithNo are usernames from r/counting, a subreddit whose members count upward together and so post enormous volumes of near-identical short messages. The most prolific counters' handles showed up often enough in the tokenizer's source text that each was merged into a single token. Other glitch tokens came from software rather than people: PsyNetMessage from the game Rocket League, externalToEVA and InstoreAndOnline from e-commerce backends.⁵ The leading space in SolidGoldMagikarp is part of the token, not a typo; the tokenizer treats a word and the same word with a space in front as two different entries.

The text that built the tokenizer is not the text the model trained on. GPT-2 trained on WebText, a scrape of every outbound link from Reddit that had at least three karma.³ The username SolidGoldMagikarp was common enough in the tokenizer's source to win a token, but the counting posts that carried it were scarce or absent from the pages WebText actually collected. The token existed. The model almost never met it.

When a model trains, each token's embedding, its learned vector, gets nudged a little every time the token appears in the text. A token that never appears is never nudged. Its embedding stays close to the random values it was handed at the start, which is roughly where all the untouched vectors sit together, near the center of the space.⁶ That is exactly where Rumbelow and Watkins found the glitch tokens clustered.¹

Now the parts are all named, so here is the mismatch in one line. The vocabulary was built from one body of text, the model was trained on another, and a token frequent in the first but missing from the second sits inside the model wearing an embedding it never learned.

Why "just a rare word" is the wrong guess

The natural explanation is that the model rarely saw the word, the way it rarely sees any obscure word. That guess predicts something the evidence does not show. If mere rarity were the cause, every rare word would glitch, and they do not. A genuinely uncommon ordinary word still turns up now and then in training, gets its embedding nudged a little, and comes back correctly when you ask for it. The glitch tokens differ in kind, not degree: not rare, but very nearly absent from training while still holding a permanent slot in the vocabulary.

The second correction matters more, because the popular version gets it wrong. People say GPT-2 has these glitch tokens. It is loose. Sander Land and Max Bartolo measured under-trained tokens across 25 models. SolidGoldMagikarp verified as under-trained in GPT-J and Phi-2, which reuse the GPT-2 tokenizer but train on their own data. It did not verify in GPT-2 itself.⁷ The tokenizer is the constant. The training data decides whether a given token is actually dead. The original failures appeared on GPT-3 models and ChatGPT,¹ which share that tokenizer family. A glitch token is a property of one tokenizer-and-model pairing, never of a model name on its own.

Land and Bartolo also showed the effect is measurable rather than anecdotal. Their detector reads a model's weights alone and flags the under-trained tokens without being told what the model trained on. On one model whose training counts were made public, the detector's signal tracked how often each token appeared in training across ten orders of magnitude.⁷ Detecting these tokens is a solved problem. Across all 25 models, the severely under-trained ones ran between a tenth and one percent of the vocabulary.

The per-model counts carry the pairing argument on their own. The models that reuse the GPT-2 tokenizer but train on unrelated text run high. The two whose tokenizer was trained on the very same corpus as the model, the Pile, run far lower, because there the vocabulary and the training barely disagree about what is common.

Where the explanation still runs out

Two honest limits sit on top of all this. The first is that under-training is the main cause but not the only one. Land and Bartolo separate several other classes: fragments left over from an intermediate merge, tokens that normal text can never actually produce because of how a string is split into pieces before the merges apply, and plain configuration mistakes.⁷ Some of these are a mismatch between how text is split during training and during use, not a token that was merely rare. A token that sits in the vocabulary but is absent from training is the cleanest and most common case, and it is not the whole catalogue.

The second limit is that a token's looks do not settle anything; you have to measure. The same study found special strings such as <mask>, which the authors expected to be untrained, showing signs of training instead, because code and tutorials use those strings as ordinary text.⁷ A token can look exotic and be perfectly healthy, or look ordinary and be dead. The detector settles it, not the spelling.

Even the people who study these tokens cannot yet close the last step. What is settled is why the embedding is near-dead. What is still a guess is the exact path from a near-dead embedding to a specific wrong output. Why SolidGoldMagikarp returns distribute and not some other word, why one dead token makes a model evasive and another makes it rude, has no worked explanation in any of the sources. The phenomenon has since entered general reference works, which repeat the SolidGoldMagikarp behavior and treat detection and mitigation as an open research target.⁸

This puzzle has a mirror at the other end of the same series. A model's trouble counting the letters in a word comes from characters hidden inside a token the model knows perfectly well: it can use the word but cannot see into it. A glitch token is the reverse, a whole string the model treats as one unit it never learned to handle. Both grow from the single fact that the model reads tokens and not characters, worked from opposite directions.