The Mechanics
An agent reruns the step that just failed because the transcript still points to it
An agent is a stateless model re-reading a transcript and a harness running what it returns, and that plain design makes repeating a just-failed step the most likely next move.
A chatbot can't keep a word it never wrote down
Its answers to a run of guesses stop agreeing because a plain chat stores nothing between turns, so the word it promised to keep was never written anywhere and does not exist.
Chatbots pad simple answers because the reward model scored longer as better
Humans prefer the longer of two answers 62% of the time, and reinforcement learning amplifies that modest edge into the padded default you get on a question with a one-word answer.
A chatbot answers a false-premise question as fluently as a true one
Four datasets of real questions measure how often it happens, and the cause sits one level below the answer, in what a next-token predictor does with text already inside the prompt.
Tell a model a puzzle isn't Tower of Hanoi and it solves Tower of Hanoi anyway
A language model's training presses the textbook answer to a famous puzzle so hard into its probabilities that changing one detail in the question rarely stops it from answering as if nothing changed.
Chatbots keep naming the heroine Elara because alignment narrowed their range
Reward training concentrates a model's probability on a few favored answers, and turning up temperature samples that narrowed range more loosely without restoring the variety the base model started with.
A typo still breaks into pieces the model has already learned
Subword tokenizers split a misspelling into fragments that already carried meaning from training, and the same flexibility gives a deliberate attacker a way in.
A chatbot rhymes a poem but can't count its syllables
Its tokenizer hands it frequency-chosen chunks of text, never the syllables or stress a poet counts, so the rhymes survive on surface patterns while the meter falls apart.
Millions of watermarked stock photos teach image generators to draw one of their own
A diffusion model learns the regularities across billions of training photos, and a stock-photo mark common enough in that data becomes one of them, a mechanism distinct from the rarer case where a model reproduces one training photo almost whole.
A text-to-speech voice mispronounces a word for one of two different reasons
Siri says "Mobile, Ala." with the wrong vowel and "Des Moines" with its silent letters sounded out, and the two mistakes are made at different points in the same pipeline, before either one becomes a sound.
A clear question gets “it depends” because three training stages each push toward the hedge
OpenAI’s own paper says its labelers were told to reward epistemic humility and the reward model learned to prefer the hedge, the strongest of three pressures that all point the same way.
Whisper answers silence with a sentence no one spoke
The stage that writes the words is a language model, and a language model with almost nothing to go on still produces its most probable sentence instead of a blank.
A model's accuracy traces a U as the answer moves through a long prompt
A Stanford-led group buried one answer among nineteen distractors and moved only where it sat, and GPT-3.5-Turbo lost more than twenty points in the middle while a longer context window bought nothing.
A coding assistant invents the same fake package run after run
Because the same invented name recurs, an attacker can register it first, so a developer who pastes the assistant's import installs code the attacker wrote.
Ask DeepSeek's model who made it and it says OpenAI
A model has no way to look inward and check what it is, so unless its maker writes the answer in, it fills the gap from a web full of ChatGPT.
The same open model returns identical text two to three times faster on one service
A small draft model guesses the next few tokens and the big model checks them all in a single pass, so the words that would have come one at a time arrive in bursts.
Image generators put a prompt's colors on the wrong objects
On the standard benchmark, Stable Diffusion binds a color to the right object at 0.38 out of 1, because the prompt is encoded with no record of which adjective belongs to which noun.
gpt-3.5-turbo-instruct plays legal chess; the chatbots don't
No board is ever put in front of the model: it predicts the next move as text, and whether that text is legal rides on statistics one model learned well and the others mostly didn't.
A reasoning model spends 901 tokens to answer 2 plus 3
Reinforcement learning taught it to run long without a gauge for when a question is easy, so on simple inputs the extra thinking mostly burns compute and leaves the answer alone.
Ask an image generator for 3:15 and it hands back a clock at 10:10
The watches in its training data almost all point at ten and two, and nothing in the generator turns a requested time into an angle.
Rename the people in a word problem and the model does worse
A model loses a little accuracy when a problem's names change and a lot when it gains one true but useless sentence, and whether that proves it cannot reason is the part still in dispute.
Give GPT-4 the same options in a new order and it changes its answer
The swing runs from 13 to 75 points across five benchmarks, and the two papers that measured it most carefully split on whether position or the answer letters drive it.
An image generator turns "six apples" into "some apples" before it draws a single pixel
Accuracy holds up for the first handful of objects, then collapses the same way in system after system tested.
Quantization rounds a model's weights to a coarser grid, and a few oversized values decide what that costs
A careful 4-bit scheme costs a 7-billion-parameter model under 1% of its perplexity, but round the same weights naively, or below four bits, and its answers can collapse.
August 2026An image generator never counts the fingers it draws
Hands are the worst case for a model that learns how images look but not how a hand is built, and better data is slowly closing the gap.
Ask a chatbot for a random number and it says 7
A chatbot has no random-number generator to reach for, so it draws the answer from a probability distribution learned from human text, which is lopsided the same way people are.
'No onions' gets onions, 'no elephant' gets an elephant
Negation is rare in the text language models train on and nearly invisible to the encoders that read image prompts, so the one word meant to remove something is the easiest for a model to override.
The token a model can't repeat back
SolidGoldMagikarp reached the model's vocabulary from a Reddit counting forum and never reached its training.
A model can spell strawberry and still miscount its letters
Its tokenizer splits the word into str, aw, berry before it reads anything, and how much of the spelling survives that split is still partly unsettled.
A strict output format can block the reasoning a model needs to be right
The program holding a model's output to a schema blocks every next token that would break the format, sometimes the very ones the model needed to reason with.
A please or a threat shifts a model's answer without reliably improving it
The wording is ordinary text the model was trained to weigh, and on a hard current benchmark it swings a single question by up to 60 points in either direction while the average gain stays at zero.
A generated image gets its letters wrong before the picture is ever drawn
The word reaches the drawing step as a whole-word token with its spelling already blurred, and the fix is an encoder that reads the prompt one character at a time.
The wait before a chatbot's first word barely tracks the prompt's length
Anyscale measured a Llama 2 70b prompt token at about a hundredth of what an output token costs in end-to-end latency, well below the tens-of-thousands regime where quadratic attention drives the pause.
Post-training makes a model less able to tell when it's wrong
A base model's own probabilities track how often it is right, and the confidence a chatbot types out was never a reading of them.
Chatbots overuse 'delve,' and the popular explanation for it failed its one direct test.
‘Delves,’ ‘underscores,’ and a small set of cousins broke sharply upward in published writing the year ChatGPT shipped, and the training step that concentrated them is settled well before the reason it chose those words.
ChatGPT multiplies two five-digit numbers and misses by 913,200
With no calculator and no scratch paper, the model reaches its answer the only way it can, by predicting the next digit, which comes out right for the small sums it has seen everywhere and confidently wrong for a large one it has barely seen at all.
A model can't count the words it's writing
GPT-4 Turbo blew past its own word-count limit on 49.3% of prompts in one benchmark, because generation keeps no running count of length and can't take a word back.
A repeated phrase makes a model likelier to repeat it again
The loop shows up under greedy and beam search but not under sampling, and why a trained model favors its own recent words in the first place is still unsettled.
One line in the system prompt turns off the bullet points
The bulleted, bold-headed answer is a post-training habit, built mostly by a stage that rewarded structure and length largely regardless of whether they improved the answer.
GPT-4 answers the same exam 23 points worse in Telugu than in English
The model saw far less of almost every other language, and the tokenizer splits the same sentence into as many as 15 times more tokens in the hardest of them.
An image generator turns a field of static into your picture by removing the noise a little at a time
The network learned exactly one skill, predicting the noise added to an image, and running it in reverse over dozens of passes is all the generator ever does.
The model sees your whole conversation for the first time on every turn
To rebuild the thread each turn, the application sends the whole transcript back as fresh input, the fact underneath a blank new chat and a long chat that forgets its start.
One character of formatting decided whether the model got it right
To a model, a colon and a space are just different tokens, and on one task that gap was the difference between 4% and 83% accuracy on a 2023 open model.
Self-attention can't tell "dog bites man" from "man bites dog"
Position has to be supplied from outside the operation, and the scheme most models now use falls apart not far past the length it trained on.
A model ends its reply by choosing a stop token like any other word
The model learns that stop token in post-training, and a separate length cap can cut a reply off mid-word before it ever arrives.
A transformer can only compute more by writing more
A fixed network does a bounded amount of computation per token, so the only way to spend more on a hard problem is to write the intermediate steps and read them back.
A verbatim quote comes out of the weights, not a lookup
Ask a model for a famous opening line and the exact words come back, because duplication and scale can press a specific sequence into a network's weights.
A chatbot slices your photo into a few hundred squares and turns each into a word
To read it, the model maps each square to a vector in the same space as its words, then lets your question read across the grid.
A chatbot quotes your PDF without learning a word of it
To answer, the model rewrites your file as points in space and reads back the few passages that land nearest your question.
A chatbot reads your whole prompt at once but writes its reply one word at a time
The reply costs more than the prompt because generating each word means rereading the prompt's cached keys and values from memory.
A safe prompt gets refused because a trained switch fires on a word
Instruction-tuning trained the pattern into the model from examples labeled harmful, not from a theory of harm. That is why Llama-2 refuses a Minecraft weapons question 96% of the time, and GPT-4 almost never does.
A model that calls a tool never runs it
The JSON a model writes when it "calls" a tool executes nothing on its own — a program outside the model performs the real action and writes the answer back as text.
July 2026Language models draw no line between instructions and data
System text, user text, and anything pasted or retrieved land in one stream of tokens, and only training, not architecture, decides which span the model treats as a command.
Whoever else is in the batch settles a near-tied logit
At temperature zero the sampler is off, and a GPU still sums the model's math in an order that depends on which other requests share the batch.
GPT-4 names Tom Cruise's mother but not Mary Lee Pfeiffer's son
Berglund et al. tested about 1,500 real parent-child pairs and found forward accuracy at 79% and reverse accuracy at 33% — a training update that only ever runs in one direction, not a fact the model forgot.
A model given the right document can still score worse than one given none at all
In a 2023 study, GPT-3.5-Turbo answered correctly 53.8% of the time when the right document sat in the middle of 20 documents, worse than the 56.1% it scored with no documents at all.
A model's weights freeze the moment its training run ends
Everything a chatbot seems to learn afterward, from a correction to a live search result, arrives through the prompt: its weights were never touched again once training ended.
A transformer redraws a word's vector for every sentence it appears in
Word2vec's famous king-minus-man-plus-woman trick ran on a vector frozen once at training time, and most explanations of how today's language models represent a word still describe that same frozen kind.
The instant a model writes a token, it becomes fact
Once a language model appends a generated token to its own output, that token becomes ordinary input, indistinguishable from the prompt — and that, not the causal mask everyone points to, is why it never revises a word once written.
A reward model taught the assistant that agreeing with you scores well
When you doubt a chatbot and it reverses a correct answer, the cause is a training stage that scored answers by predicted human approval, with no test of whether they were true.
A model builds a fake citation the same way it builds a true one
No part of a chatbot checks whether a reference is real, and the way models are graded rewards a confident invention over an admitted blank.
How a model learns from examples it was never trained on
Show a model two or three input-to-output pairs and it continues the pattern with its weights frozen. A two-attention-head circuit that copies from earlier in the prompt is the part we can point to, and it is not the whole answer.
Two answers to one prompt diverge at a single draw
The weights and the prompt fix a probability distribution over the next token, and a sampler's single draw from it is where two answers to one question come apart.
A model can recognize strawberry's letters and still miscount them
The word arrives as the three integers 496, 675, and 15717, and counting its r's would mean recovering the spelling from those numbers and then running a tally the model was never built to run.
One gradient step drops the loss from 7.0 to 0.225
The rule subtracts from each weight a fraction of its share of the error, and that same rule, threaded back through the layers, is what sets every weight in a trained network.
An attention head is a weighted average that computes its own weights
Nine dot products across three tokens compute one attention head's output, and the same nine show why long context costs what it does and why the softmax weights are not, by themselves, an explanation.