A chatbot quotes your PDF without learning a word of it

The document the model never trained on

Upload a forty-page apartment lease as a PDF and ask the chatbot a plain question: can you sublet the place while you travel. It answers in a sentence, and it quotes the exact clause that settles the matter, one line out of forty pages. But its training finished months ago, and your lease was never part of it. The model's weights were frozen when training ended, so it never learned your lease the way it learned the rest of the world. It still found the clause and used it. Where did that clause come from.

Rule out the obvious guess first: that the system handed the whole lease to the model and let it read the pages. It did not. For one lease it could, but the same feature runs over a folder of files, or a corpus of millions, and a model takes in only so much text at once. So the system does something narrower. It retrieves a few passages it judges relevant and inserts just those into the prompt, then asks the model to answer over them. This is retrieval-augmented generation, named by Lewis and colleagues in 2020.¹ Two mechanisms do the real work. A search over meaning picks the few passages worth showing, and the model reads them as ordinary prompt text and answers over them.

Relevance becomes a distance you can measure

Before the model sees anything, the system picks which passages of your lease bear on the word "sublet." It does this without reading for meaning the way you do, using the trick from word embeddings. An embedding model reads a span of text and turns it into a vector, a list of a few hundred numbers that names a single point in a high-dimensional space. Text that means similar things lands at nearby points. The word-embeddings lesson built this for single words. The same idea runs on whole passages, so one chunk of your lease becomes one point, placed by what the passage means rather than by any single word in it.²

The pipeline is short. The lease is split into chunks, and each chunk is embedded to a point once, in advance. Your question is embedded the same way the moment you ask it. Relevance is then just closeness: the system returns the chunks whose points sit nearest your question's point. Follow the sublet query through it. The governing clause reads "the Tenant shall not assign or transfer occupancy of the unit to another party," and it never contains the word "sublet." A plain keyword search for "sublet" finds nothing in that line. But "can I sublet" and "assign or transfer occupancy" mean nearly the same thing, so the embedding model places their points close together, and the clause rises to the top. Nearness of meaning found the line that shared no words with the question.

How is closeness measured. Two answers are common, and they nearly agree. Sentence-BERT, the method that made one fixed vector per passage practical, compares two vectors by cosine similarity, the angle between the directions they point.² The retrieval lineage RAG is built on, Dense Passage Retrieval, instead ranks by the raw dot product of the two vectors.³ Cosine is that same dot product after both vectors are scaled to the same length, so the two measure the same nearness. Cosine is the common default, not the metric every system uses. The claim that survives the choice is the one to hold: relevance is geometric closeness of whole-passage embeddings.

Does closeness of meaning beat matching words? In its own domain, clearly. The standard keyword baseline is BM25, a bag-of-words score that rewards a document for sharing query terms, with diminishing credit for repeats and an adjustment for length, and no notion of meaning at all.⁴ Measured against it on Natural Questions, dense embeddings win by a wide margin.

Across other open-domain sets that gap runs from 9 to 19 points.³ In the setting it was trained for, closeness of meaning beats shared words.

Finding the nearest points, at a billion of them

Closeness says which passages to return, not how to find them without measuring the distance to every point in the index. For one lease that cost is nothing. For a corpus of millions of chunks, checking every one on every query is unaffordable. This is approximate nearest-neighbor search, and it is the one step in the pipeline that counts as settled engineering. The trick is to avoid looking at most of the points. One standard method, HNSW, wires the points into a layered graph and searches from a coarse top layer down to a fine one, hopping toward the query's neighborhood and examining only a small fraction of the index. Its cost grows with the logarithm of the collection size rather than in proportion to it, so ten times the data adds a little, not ten times.⁵ At the largest scale the numbers are blunt: the FAISS library connects a billion vectors into a searchable index in under twelve hours on four GPUs.⁶ The lookup is not where the difficulty lives, so treat it as solved and climb back up.

What the model does with the passages it is handed

Now the model finally enters. The few top-ranked chunks, your lease's sublet clause among them, are placed into the prompt as ordinary text, right next to your question. The model does not open a database or follow a link. It reads the retrieved clause the same way it reads anything else in its input, as tokens in the prompt. It answers conditioned on those tokens, and it produces the citation by quoting or pointing back to the chunk it leaned on.

This is the whole arrangement, and it is looser than its famous name suggests. The 2020 paper that named RAG built something more integrated than the products that now borrow the term. It kept the document encoder and the document index fixed and fine-tuned the query encoder and the generator together, so the searcher and the writer were trained as a pair on knowledge tasks.¹ The RAG in a typical shipped product trains neither. It embeds with an off-the-shelf model, searches, and pastes the top chunks into the prompt of a frozen chat model that was never tuned for the job.

The confident citation the passage does not support

Anyone who has used this has also watched it fail in one particular way. The answer arrives with a citation, you click through to check, and the quoted passage does not actually say what the answer claimed. It is not rare. A human audit of four shipped generative search engines found that, on average, only 51.5% of their generated sentences were fully supported by the citations attached to them, and only 74.5% of citations actually supported the sentence they were pinned to.⁸

The ladder we climbed down lets us point at where this breaks, and the honest answer names two rungs. The same wrong quote can be born at either, and from the outside it looks identical.

The first rung is retrieval. The nearest passage is not always the right one. Closeness of meaning is a good guess, but it is a guess, and on text unlike what the embedding model was trained on it misses. The BEIR benchmark tested retrievers across eighteen datasets outside their training domains and found that plain dense retrievers often underperform, with real room to improve how they generalize.⁹ When the top passage handed to the model is the wrong one, the model can cite it faithfully and still be wrong, because it was pointed at the wrong line to begin with.

The second rung is generation. The right passage was retrieved and sat in the prompt, and the model answered from its own weights anyway, inventing support the passage does not contain. That is the failure the hallucination lesson describes, now wearing a citation. That the correct passage can be present and the model still not use it is its own open problem, the one the losing-the-thread lesson takes up.

No cited study cleanly separates the two. The audit measures the outcome across whole systems and cannot say, for any one bad citation, whether retrieval fetched the wrong line or generation ignored the right one.⁸ That is the honest edge of what is known. Below it the ground is firm. Embedding a query and passages, ranking them by closeness, and returning the nearest few fast is settled and works. Above it the open choices sit unresolved. How to cut a document into chunks has no settled rule and no paper that prescribes one. Whether to add a slower, more careful reranking pass is a cost you weigh, not a solved default.⁹ And whether the model uses the passage it was handed is, on the current evidence, not something you can count on.