A verbatim quote comes out of the weights, not a lookup

The sentence that comes back whole

You ask a model for the first line of Moby-Dick and it returns Call me Ishmael. Three words, exact, every time. It feels like the model looked the sentence up. It did not. At the moment it answers, there is no copy of the novel on hand for it to open, and no file to read the line out of. The words come out of the same weights, the millions of adjustable numbers set during training, that produce every other answer. And the effect is not confined to short famous lines. In a complaint filed in December 2023, The New York Times showed GPT-4 returning long stretches of its articles almost word for word when prompted with their openings.¹ So the specimen is that returned sentence, and the question is plain: how does a specific string get inside a network, and come back out intact?

Fit until the loss is gone

Training fits the weights by one repeated exercise: show the model some text, have it predict the next token, and nudge the weights whenever the prediction is wrong.² Do this across a whole corpus and the model learns the general shape of the language. After Call me a name is likely, and many names would serve. For most text that is all that survives, the shape, not the exact words. But when the model meets the same sequence often enough, the nudges keep pointing the same way. The probability it places on the one true continuation climbs until the model reproduces the string token for token. In the training arithmetic, the error on that sequence has been driven to near zero, and nothing pulls it back up. The sequence is now fixed in the parameters, and feeding the model its start returns the rest. That is memorization.

It is worth cashing the picture that wants to form here, of a saved file the model opens. There is no such file. The model does not keep Call me Ishmael. as characters and fetch them. It keeps a setting of weights that makes Call me continue to Ishmael and onward, the same next-token prediction it runs on any other prompt. That is where the file picture breaks. A saved file can be deleted, and its bytes sit somewhere definite. A memorized string is smeared across weights that also do everything else, with no row to point to.

That specific sequences really are in there, and not just common phrases the model would guess anyway, is what extraction studies established. Querying the public GPT-2 in 2021, Carlini and colleagues confirmed 604 unique sequences the model had memorized, recovered by prompting alone.² Some appeared in only a single training document, a lone email address or phone number the model had seen once and could still emit. A string that occurs once, pulled back out on demand, is stored, not inferred.

Copies raise the odds

A string seen once can stick, but the reliable lever is how many times it appears. Call me Ishmael. is not rare in the training data. It sits in the novel, in essay after essay about the novel, in list upon list of great opening lines, copied across the crawled web thousands of times. Each copy is another training example rewarding the exact continuation, so the odds the model reproduces it climb with the count.

The climb has no cliff. Prompting models with the true opening of a sequence, Carlini and colleagues in 2023 found extractability rising with the number of copies in the training data along a log-linear curve: more copies, steadily more likely, with no single count past which memorization switches on.³ Seen from the other direction, the same driver shows up as a cost. Lee and colleagues found that more than 1% of the unprompted output of standard language models is copied verbatim from the training set. One 61-word sentence sits in the widely used C4 web corpus more than 60,000 times. Training on de-duplicated data cut verbatim emission roughly tenfold.⁴ Remove the copies and the reciting mostly stops.

The pattern holds on exactly the specimen's kind of text. D'Souza and Mimno prompted ChatGPT for 240 poems by 60 American poets. It returned 72 of them word for word, while smaller models returned almost none. The strongest predictor of which poems came back was canonical prominence, inclusion in the standard anthology.⁵ Canon is duplication under another name: the poems everyone reprints are the poems the model has met the most.

Bigger models keep more

Duplication is one lever. Model size is the other. Hold the data fixed and enlarge the network, and it memorizes more of what it saw. Measuring the GPT-Neo models from 125 million to 6 billion parameters, Carlini and colleagues found that each tenfold increase in size raised memorization by 19 percentage points on their evaluation set, close to a straight line (an R² of 99.8%). The largest model memorized two to five times more than the smallest.³ The reason is capacity. A bigger network, having captured the shared patterns of the language, has weights to spare, and spare capacity is what lets it fit particular sequences exactly instead of compressing everything into general structure. Mark the limit on the figure: the slope is clean within one model family and grows more complicated across families, so it is the trend that is settled, not the exact number.³

Which raises the question the specimen keeps provoking: how much has a model actually memorized? There is no single answer, because the answer depends entirely on how you count. Counted as unprompted recovery of strings that appear in at most one document, GPT-2's memory is a rounding error, on the order of 0.00000015% of its data. Counted as exact suffixes recalled from their true prefixes, the 6-billion-parameter GPT-J has at least 1% of The Pile, its 825-gigabyte training set, sitting recoverable in its weights.³ Same phenomenon, four orders of magnitude apart, on the definition alone. Neither is the true number. Each answers a different question, and a percentage printed without its definition is not a fact.

How much of that comes out in practice was measured directly on a deployed model. Nasr and colleagues ran a divergence attack on ChatGPT. Instructing it to repeat a single word such as poem forever broke its assistant manner and dropped it back onto its training data, which it then emitted at 150 times the normal rate.⁶ About $200 of queries pulled out more than 10,000 unique memorized examples. In the strongest configuration more than 5% of the model's output was a verbatim 50-token copy of training text, real email addresses and phone numbers among it.⁷ At the time the work was published the trick had not been patched.⁸

The neighbors it is not

Memorization has three neighbors, and the confusions between them are where explanations of this behavior usually go wrong.

The first is retrieval. A system doing retrieval looks text up at answer time from an outside store and reads from what it finds; memorization holds the text in the weights, with nothing fetched. The same New York Times filing shows both at once. GPT-4 returning the paper's old articles is memorization, the text in the weights.¹ But the filing also shows Bing Chat reproducing an October 2023 article, one that postdates the model's April 2023 training cutoff and so could not have been in its weights at all.¹ That passage was fetched live. Identical-looking output, two different places the text lived.

The second is hallucination, where the model invents plausible text that matches no source. Memorization is the mirror image: the output is real and matches a specific document exactly. A hallucinated opening for Moby-Dick would be a fluent sentence Melville never wrote; the memorized one is his, to the character.

The third neighbor is the one with no clear fence. The same next-token training that memorizes a repeated sequence also generalizes across text the model saw once, and there is no measured line where storing a string ends and learning a pattern begins. Whether a given deployed model has memorized a passage or merely learned to reconstruct it, and how much any such model holds, cannot be read off from the outside. The number depends on the definition and on prompts no user sees.

So mark the ground. Settled: models memorize some of their training sequences, duplication and scale increase it, and de-duplication reduces it. Open: where memorization ends and generalization begins, and how much a specific model holds. The parties to the New York Times case divide on a further question, whether the behavior is routine. The Times presents the near-verbatim output as reachable with minimal prompting.¹ OpenAI calls regurgitation a rare bug it is working to drive to zero, and argues the Times used long, manipulated prompts and cherry-picked its examples.⁹ Weigh it on the evidence. Both sides agree duplication drives it, and OpenAI says so plainly. The extraction work shows the stored text is real and, under an adversarial prompt, not hard to pull out. Whether an ordinary request trips it easily is the genuinely contested part.