A model builds a fake citation the same way it builds a true one

The model wrote the reference, it did not look it up

Ask a chatbot for the title of a specific researcher's doctoral thesis and it may answer with a clean, confident, wrong title. Ask the same system for that person's birthday, with a note to answer only if it knows, and one model gave three different wrong dates on three tries.¹ None of that is a search that went wrong. It is a hallucination: output presented as fact that nothing in the world backs up.²

The instinct is to say the tool reached for a reference and grabbed the wrong one. It did neither, because a plain chatbot has nowhere to reach. When it answers, a trained network reads the words so far and produces one next word, then the next, from patterns fixed in its weights while it was trained.³ There is no shelf of references inside it and no lookup against one. The lesson on how a word gets chosen covers the draw itself, and the lesson on what the model sees covers the units it works in. The citation was written, one word at a time, the same way the rest of the answer was written. It was not fetched.

So the useful question is not why a lookup failed. It is why a fake reference is the thing the writing produces.

A citation is a shape, and shapes are easy to predict

The model writes by predicting a likely next word given the words so far. A reference has a very regular shape: a surname, a year, a title in title case, a journal name, a volume, a page range or an identifier. The training text carries that shape a vast number of times. So once an answer reaches the point where a citation belongs, the words that make one are among the most probable continuations, whether or not a specific true source sits behind this instance. The model learned the form of a reference thoroughly. It did not, and could not, store which exact references exist.

A benchmark called TruthfulQA measured a close relative of this. It asked models questions where a common human answer is false, and found that models reproduce those learned falsehoods: the best model was truthful on 58 percent of the questions, against 94 percent for people.⁴ Whatever pattern the text taught is what comes back out, true or not. The survey that fixed the vocabulary for this field calls the case at hand extrinsic: output that cannot be verified from any source, neither supported nor contradicted by one.⁵ A fabricated citation is exactly that. Nothing grounds it, and nothing in the question rules it out.

No part of the model stops to ask if it is true

Between the ranked possibilities for the next word and the word that actually comes out, there is no separate step that asks whether the model knows this. A plausible string and a true string are produced by the same operation, and nothing in the system tells them apart. This is the step most casual explanations skip.

It is worth being careful, because models are not simply blind to their own uncertainty. A calibration study found that a large model, given the right format, can report a fairly accurate probability that its own answer is correct, and can partly predict whether it knows an answer at all.⁶ Later work reads a usable uncertainty signal directly from a model's internal state.⁷ So the information is present, in part. What a standard model lacks is a gate: a step that consults that signal and holds back the answer when confidence is low. The signal is real, it is imperfect, and later stages of training can distort it. A model can half-register that it is unsure and still emit the confident reference, because nothing wires the knowing to the saying.

The grading rewards a guess over an "I don't know"

Why would the process leave that gap? Because the way models are built, and above all graded, pays for guessing. One recent argument frames it as a school exam: a student who is unsure still writes something down, since a blank scores zero and a guess sometimes earns the mark. Most benchmarks score a model the same way, on whether the answer is right, with no credit for admitting ignorance and no extra penalty for a confident mistake.¹ A model tuned against that scoring learns to always answer. The authors argue the result is not a mysterious defect but the expected outcome of the objective. Producing a correct answer is in general harder than telling a correct answer from a wrong one, so some rate of confident error is built in.

Size does not rescue this. TruthfulQA found the largest models the least truthful of the group, because a bigger model imitates the training text, and its falsehoods, more fluently.⁴ The reason a fake citation matters more than most inventions is that it carries no alarm. A made-up function in code fails the moment the code runs, and the error is instantly obvious;⁸ an invented reference runs against nothing. It reads like a real one until a person tries to open it.

Hand it the source, and the inventing mostly stops

The cleanest evidence that the cause is missing grounding, rather than a broken model, is what happens when you supply the grounding. Put the relevant document in front of the model at answer time and let it work from that text, and the invention rate falls. That is what retrieval augmentation does: it pairs the memory baked into the weights with a live lookup into an outside store of text. The method's first paper reported that such a model generates "more specific, diverse and factual language than a state-of-the-art parametric-only" model.⁹ The "parametric-only" model there is the plain chatbot from the top of this lesson: a store of patterns with no way to look anything up. Give it a way, and the behavior changes. That is a controlled comparison, and it points straight at the cause.

It also marks where the settled account ends. Two things stay open even for the people who build these systems. The first is timing: a clean mechanical account of when a model invents versus when it recalls correctly does not yet exist. The uncertainty signal helps predict it but does not fully explain it.⁶ The second is whether the behavior can be removed entirely. Grounding handles facts that live in some document a system can fetch. For a fact that appears once or never in any text, there is no pattern to learn and nothing to retrieve, and the same argument that treats hallucination as expected holds that some rate of invention is then unavoidable.¹ The chain down to here is settled engineering: next-word prediction, no lookup, no truth gate, and grading that pays for a guess. The fine timing of a fabrication, and its full removal, are not.