What is RAG, and why do AI systems use it?

In detail

The problem RAG solves

On its own, a language model only knows what it saw in training. It can’t see your company’s documents, last week’s data, or anything behind your login. Ask it about those and it will either refuse or, worse, invent a confident answer. RAG closes that gap by giving the model the source material at the moment you ask. Assistants that seem to know your files or this week’s news often use retrieval behind the scenes. Van Luijt described RAG’s spread in episode 8 at 13:41 as reaching the early majority: an ordinary developer at an enterprise gets an assignment and realizes it can be solved with RAG.

How it works

Three steps. First, your documents are split into chunks and turned into embeddings, numerical representations stored in a vector database. Second, when a question comes in, the system embeds it too and retrieves the chunks closest in meaning. Third, those chunks are added to the prompt, and the model answers using them. The model isn’t recalling the answer; it’s reading it.

Where the idea comes from

The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. It paired a pre-trained generator with a retriever over a dense vector index of Wikipedia, and argued that knowledge stored only in a model’s parameters is hard to update and hard to trace back to a source. With retrieval added, their models set new results on three open-domain question-answering tasks and generated more specific and factual text than the same generator without retrieval. Production RAG systems today are looser than that paper’s design (many bolt a retriever onto an off-the-shelf model rather than training the two together), but the motivation is the same.

Why it beats fine-tuning for fresh knowledge

Teaching a model new facts by fine-tuning is slow, expensive, and goes stale the moment the data changes. RAG keeps knowledge in a store you can update any time — add a document and the next query can use it, no retraining. That makes it the default way to put current, private, or fast-changing information in front of a model. Fine-tuning still has a place for teaching a format, a tone or a narrow skill; fine-tuning vs. RAG vs. prompting walks through when each one fits.

Where it falls short

RAG is only as good as what it retrieves. If the search returns the wrong chunks, the model answers from bad context, and a grounded-looking answer can still be wrong. That’s why retrieval quality, not just the model, decides whether a RAG system works — and why evaluating the retrieval step separately matters. Weaviate co-founder Bob van Luijt gives a concrete failure in episode 8 at 25:34: ask for a company’s Q2 revenue, and the right chunk may be a PDF page that never names the company in its text; the only clue is a logo. A retriever that sees only the text cannot connect that chunk to the question.

The common fixes target the retrieval step. Hybrid search combines keyword and vector matching so exact names and codes aren’t lost, and a reranker re-scores the top results before they reach the model. On the generation side, faithfulness checks catch answers that go beyond what the retrieved text says. How to evaluate a RAG system covers measuring both halves, and agentic RAG vs. traditional RAG covers systems that let an agent decide when and what to retrieve.

Disclosure: Episode 8 aired while Chain of Thought was Galileo’s podcast and Conor worked there; it includes a Galileo ad read at 22:30.