A Method That Pulls Out What Is Needed and Passes It On
RAG (retrieval-augmented generation) is a method of finding documents related to a question before an LLM answers, having it read them along with the question, and basing the answer on what they say. The name spread from a 2020 paper by Lewis and colleagues. With only the knowledge a model learned during training, it cannot answer about internal materials or information published after its training. RAG makes up for this by passing the needed parts just before the answer.
In a common setup, the material is split in advance into small chunks of a few hundred tokens each and stored in a form that can be searched by closeness of meaning. When a question arrives, the most relevant chunks are chosen and passed to the model along with the question.
Choosing Between RAG and Reading Everything
When the material is small, having the model read all of it without searching is simpler. In a September 2024 post, Anthropic says that if the material is under 200,000 tokens (about 500 pages), you can simply put all of it in the prompt. In that case, methods like RAG are not needed. RAG suits cases where there is more material than that or where the material grows often.
Reading more does not guarantee reading correctly, however. In a study by Liu and colleagues (published in 2023), answer accuracy dropped sharply when the needed information was in the middle of a long input. Keep this in mind when passing long material in full as well.
Two Parts That Decide Answer Quality
The quality of a RAG answer depends on both the part that finds documents and the part that reads them and answers. If the search misses, the model cannot write the right answer. Anthropic gives an example of chunks losing context when material is split. A chunk saying only "revenue grew by 3% over the previous quarter" does not tell you which company or which period it refers to. It then shows a technique of adding a short explanation of the surrounding context to each chunk.
On the pages for choosing by use, "RAG and document QA" compares the strength of the reading and answering part, ranked by accuracy on comprehension questions about a 120,000-token story. How well the search part works depends on the effort of those who build the system.