Today's RAG pipelines fine-tune neither piece the paper trained together

The RAG in the paper and the RAG in the tab

The 2020 paper that named "retrieval-augmented generation," by Patrick Lewis and colleagues at Facebook AI Research, described one specific system.¹ A Dense Passage Retriever ran over 21 million Wikipedia passages,² a BART generator wrote each answer marginalized over the top few passages the retriever returned, and the query encoder was fine-tuned end-to-end together with the generator on question-answering data.

"RAG" today names something different. A Fudan University survey calls the current usage "Naive RAG": chunks of a document store, retrieved by an off-the-shelf embedding model, pasted into an off-the-shelf chatbot's prompt, with no training loop between the two pieces.³ Both are called RAG.

What Lewis et al. actually built

The system paired two pre-trained models. The retriever was DPR (Dense Passage Retriever), released the same year by an overlapping author team. It paired two BERT-base encoders, one for questions and one for passages, supervised on Natural Questions and TriviaQA to score question-passage pairs by dot product.² DPR indexed 21,015,324 passages, cut from the December 2018 English Wikipedia dump into non-overlapping 100-word blocks, into a FAISS approximate-nearest-neighbor index. The generator was BART-large, a 400-million-parameter sequence-to-sequence transformer.¹

Lewis et al. wired the two together and fine-tuned end-to-end, and then explicitly did not. Updating DPR's passage encoder during training would have required re-embedding and re-indexing 21 million passages every few steps, and the authors decided the cost was not worth paying.

So the "joint training" that later readers point at is narrower than the shorthand. One of DPR's two encoders and the generator were fine-tuned together on the question-answering data; the passage-side encoder and the FAISS index were left where DPR's own supervised training had put them. Nothing in the paper's system was trained from scratch. The retriever was initialized from a DPR checkpoint already supervised on Natural Questions and TriviaQA, and only one of its two encoders was moved further during RAG training.

The four benchmarks and the Jeopardy claim

Lewis et al. put the wired-together system against three kinds of baseline on four open-domain question-answering benchmarks. The comparison table (Table 1 in the paper) carried the numbers that made the paper influential.¹ On Natural Questions, RAG-Sequence answered 44.5 percent of the 3,610 test questions exactly right. The T5-11B closed-book baseline, made to hold the same world knowledge in its 11.3-billion parameters with no lookup at all, scored 34.5.¹ Adding salient-span-masking pre-training to T5-11B raised it to 36.6. REALM, Google's roughly 330-million-parameter open-book system that pre-trained a retriever alongside its generator, scored 40.4.⁴ DPR fed to a standard extractive reader, the strong 2020 open-book baseline, scored 41.5.¹

RAG-Sequence beat every baseline on Natural Questions, WebQuestions, and CuratedTrec, and lost to DPR-extractive on TriviaQA by 1.1 points on the open test (56.8 versus 57.9). The 8.0-point gap over T5-11B on Natural Questions is the number the paper is usually cited for. Its shape is worth naming: RAG's 400-million-parameter generator, paired with a retrieved-passage index, beat an 11.3-billion-parameter closed-book model on questions the closed-book model had to hold in its weights.

The paper also reported a factuality claim, and its evidence is narrower than the phrasing suggests. Lewis et al. generated Jeopardy-style questions and had human raters compare 452 pairs of RAG and BART generations. Raters called RAG more factual than BART in 42.7 percent of pairs, and BART more factual than RAG in 7.1 percent.¹ That is the paper's evidence for "hallucinates less." It is not a general hallucination rate. No experiment in the paper compares the same model with and without retrieval on the same prompt to give one.

The one demonstration that speaks most directly to today's use of RAG, swapping the index to update the model's world knowledge, is smaller than a reader might guess. Given a 2016 Wikipedia index, RAG named 70 percent of 2016 world leaders correctly and 4 percent of 2018 ones; given a 2018 index, RAG named 68 percent of 2018 leaders correctly and 12 percent of 2016 ones.¹ Change the index, change the answers. That worked because both indices were built by the same document encoder RAG's frozen query encoder had learned to talk to.

The pipeline that inherited the name

The "Naive RAG" pipeline the Fudan survey names trains nothing.³ A team wires an off-the-shelf embedding model to a vector database of chunked documents, sends the user's question through the embedding model, retrieves the top few chunks, and pastes them into an off-the-shelf chatbot's prompt. The primary studies on that pipeline measure different quantities than the ones the paper measured, and they do not all agree with each other.

Peng Xu and colleagues at NVIDIA, in a 2024 ICLR paper, ran the pipeline on Llama2-70B and a 43-billion-parameter GPT-style decoder, using Dragon, Contriever, and OpenAI's text-embedding-ada-002 as their three off-the-shelf retrievers.⁵ Averaging over seven long-context tasks, Llama2-70B with a 4,000-token context and retrieval scored 36.02, close to the 36.78 the 16,000-token variant scored without retrieval. At 32,000 tokens the retrieval-augmented model beat its non-retrieval counterpart, 39.60 to 37.36. Retrieval helped large long-context models. Xu et al. themselves flag a concurrent long-context benchmark that reached the opposite conclusion: retrieval helps small models but not long-context ones. They read the disagreement as a scale effect. The zero-shot ability to use retrieved chunks grows with parameter count, and the two primary sources measured models on opposite sides of the threshold.

Nelson Liu and colleagues at Stanford, in the paper picked up as "lost in the middle," measured what happens when the answer-bearing passage moves through the input context.⁶ GPT-3.5-Turbo scored 88.3 percent on their multi-document QA task when given only the answer-bearing passage, and 56.1 percent when given no passages at all. In the 20- and 30-document settings, with the answer in the middle of the context, it fell below that 56.1-percent closed-book number. Feeding the model the right documents in the wrong position was worse than feeding it none.

Cheng Niu and colleagues, from NewsBreak and the University of Illinois Urbana-Champaign, built RAGTruth, an 18,000-response corpus of RAG outputs annotated for hallucinations under retrieval.⁷ On the 450-instance test set, GPT-4-0613 hallucinated in 9.3 percent of responses and Llama-2-7B-chat in 51.8 percent. Across the full corpus, 43.1 percent of responses carried at least one hallucinated span. RAGTruth reports rates with retrieval only; there is no matched no-retrieval condition on the same prompts, so the corpus does not answer whether retrieval reduces or increases hallucination on a given model. It answers a different question, how often the LLM ignored or contradicted the passages it was handed.

Scott Barnett and colleagues at Deakin University, reporting on three deployed RAG systems in a 2024 experience paper, named seven engineering failure points in the pipeline itself.⁸ Chunks with the answer that failed to rank in the top few (FP2). Chunks that ranked but did not fit into the model's context (FP3). The model given the answer in context and still failing to extract it (FP4). Their empirical BioASQ run covered 4,017 open-access documents and 1,000 questions, though the paper's abstract states the size as 15,000 documents; the case-study section is the number to trust. Barnett's taxonomy was cataloged on production pipelines that train neither piece, and Lewis et al. never measured their system against it.