A typo still breaks into pieces the model has already learned

Nothing in the pipeline checks your spelling

A message that reads "i dont no if this is neccessary" gets answered by a chatbot as though every word in it were spelled correctly. The easy explanation is that something quietly fixes the spelling before the model sees it, the way a word processor underlines a typo and offers the right one. That explanation is wrong, and no source says so directly. A complete description of the pipeline simply leaves the correction step out.

The paper that introduced the transformer architecture behind GPT models lays out every stage between raw tokens and a predicted next word. Each token is embedded, gains a signal for its position, passes through repeated blocks of self-attention and feed-forward transformation, and is projected through one linear layer and a softmax into a probability over the entire vocabulary.¹ Turning text into tokens, the step before any of that, is described in OpenAI's GPT-2 paper as a fixed merging procedure applied to raw bytes. The paper treats avoiding "lossy pre-processing" as a stated design goal, not a step it adds.² There is no spelling stage: the pipeline has tokenizing, embedding, attention, and next-token prediction, and nothing else.

A misspelling breaks into more, smaller tokens

Before any of those stages run, the raw text is cut into tokens: chunks that can be a whole common word or as short as a single character, chosen by an algorithm called byte-pair encoding, or BPE. BPE began as a text-compression algorithm and was later adapted for exactly this kind of tokenization.³ It learns its chunks once, from a large body of training text, before the model itself ever trains. It counts every pair of adjacent symbols in that text, merges the single most frequent pair into a new symbol, and repeats the merge a fixed number of times.⁴ Whatever symbols survive become the fixed vocabulary. New text is split into characters, then rebuilt by replaying those merges into the largest known chunks.⁴ A rare or misspelled word that never appeared whole in training simply comes out as more, smaller pieces instead of one piece.⁴

GPT-2 and its successors run that merging procedure over raw bytes rather than characters, so the starting alphabet is only 256 symbols instead of the 130,000-plus Unicode would need.² Any string can always be represented; the worst case is a word breaking all the way down to individual bytes.²

A live run of tiktoken, OpenAI's own tokenizer, using the cl100k_base vocabulary that backs GPT-3.5-turbo and GPT-4,⁵ shows exactly this split. "Necessary" comes out as a single token. "Neccessary" comes out as three: "ne", "ccess", and "ary".⁶ The same pattern holds across other ordinary misspellings:

The fallback goes all the way to raw bytes when nothing else matches: an emoji not covered by any learned merge splits into the UTF-8 bytes that encode it, three tokens for one emoji, nothing dropped.⁶ A tokenizer built this way never has an unrecognized word to substitute or correct, only bytes to fall back on.

That same subword mechanism runs the opposite way when the vocabulary itself misfires. The token a model can't repeat back describes strings that earned their own token during tokenizer training but almost never appeared during the model's training, so that token's meaning never formed. That is the inverse of a typo: one whole token with no learned meaning, instead of a misspelled word broken into pieces that already mean something.

Common pieces already carry the meaning

Once split into tokens, each piece enters the model as a vector: a long list of numbers learned during training. A transformer redraws that vector for the sentence a token is actually sitting in, but the vector it starts from already reflects everywhere that token showed up in training, whatever word it was part of.

Nothing in that pipeline defends against typos on purpose, either. Researchers building subword translation systems have said as much about their own work: rather than building in a defense, "we instead hope that the relevant noise will occur in the training data."⁷ Whatever tolerance for a misspelling exists has to come from what byte-pair encoding already does to ordinary text, not from a designed correction step.

That points to the real reason the pieces still mean something. BPE's merge rule only ever promotes the single most frequent symbol pair in the training text, repeated until its merge budget runs out.⁴ A fragment small enough to survive on its own is, by construction, one of the more common sequences in the language. "Ne", "ccess", and "ary" turn up constantly across ordinary, correctly spelled words. Each has had far more chances to earn a well-trained vector than the rare string "neccessary" ever would alone. A misspelling breaking into recognizable pieces is not the model recalling this exact typo. It is the tokenizer handing the model pieces it already understands, arranged in an order it hasn't seen but can still read from what those pieces mean elsewhere.

Context makes the intended word the most likely next one

The last stage in the pipeline turns every token's vector into a probability distribution over what comes next.¹ A model reading "...that isn't neccessary for the meeting" has already turned "ne", "ccess", and "ary" into vectors near "necessary". Every other token in the sentence is available too. An attention head is a weighted average that computes its own weights across exactly those earlier tokens, so the surrounding words vote on what the misspelled one most likely means, the same way they would for the correctly spelled version.

The model then predicts one token, appends it, and runs again, reasoning from that point about a sentence with "ne", "ccess", "ary" sitting where "necessary" would have. None of this rewrites those tokens into "necessary"; they stay exactly what the tokenizer produced. What changes is which continuation those pieces, read in context, make most probable.

The same flexibility that helps a typo also helps an attack

Put together, this is settled: fixed, frequency-learned pieces, already meaningful from training, read in context by a mechanism never built to notice a misspelling.¹²⁴ It is not a guarantee against every character-level change, though, and the line between an everyday typo and an adversarial input is fuzzier than a clean split.

A 2023 benchmark ran algorithm-searched, maximum-damage character edits against nine large language models, including GPT-3.5 and GPT-4, and found an average 20 percent drop in task performance. GPT-4 was among the most robust models tested, but not immune.⁸ The authors describe those same edits as "plausible user errors like typos,"⁸ blurring the line between an everyday typo and a crafted one. A 2025 study of GPT-4o and a large Llama model found something related: the model's own wording measurably changes between a clean prompt and one perturbed with synthetic, keyboard-adjacent typos.⁹ That is not evidence of a misread. The study measures wording overlap, not comprehension, so a changed answer can still be a correct one stated differently.⁹

The same study complicates the training-data story further: it describes large language model training sets as "curated datasets that lack human-induced errors, such as typos."⁹ That favors the reading this lesson has argued for, where "ne", "ccess", and "ary" are common pieces on their own, over the folk version where the model simply memorized this one typo.

The same subword flexibility that lets an accidental typo land safely gives a deliberate attacker more precise room to work. Testing a fine-tuned BERT sentiment classifier, researchers found that a single character added, dropped, or swapped, chosen to do maximum damage, dropped its accuracy from 90.3 percent to 45.8 percent.¹⁰ Subword and character-level models turned out more vulnerable to this than word-level models, because every character edit produces a genuinely different sequence for an attacker to search over.¹⁰

A further class of attack sits outside typo robustness altogether. Homoglyphs, a Cyrillic "а" standing in for a Latin "a", and invisible Unicode characters change what bytes a tokenizer receives without changing what a human reader sees.¹¹ A single substitution can measurably degrade a target system. Three can functionally break many of the machine-translation, moderation, and classification systems tested.¹¹ A typo is visible to everyone who reads it. An imperceptible encoding attack, by design, is not, and it targets the boundary between text and bytes directly rather than producing a spelling variant anyone would notice.

An explanation that only says the model "understood" a typo does not distinguish an ordinary misspelling from a Cyrillic "а" standing in for a Latin one, and that distinction is exactly what separates a solved problem from an open one.