Architecture · LLM systems
The RAG pipeline, from indexing to answer
A language model only knows what its parameters hold: its memory is frozen at the end of training, and it cannot say where it came from. Retrieval-Augmented Generation (RAG) gives it a searchable external memory: before answering, the system fetches relevant passages and hands them to the model1.
Parametric memory
The model's weights. Vast, but frozen, without provenance, and costly to update.
Non-parametric memory
An index of documents. It can be corrected or extended without retraining the model1.
The link between them
A retriever that, for each question, selects the few passages to put in front of the model.
Two phases, two moments
Indexing (offline) and query (online)
- Indexing flow (once per document)
- Query flow (for every question)
- Reading the index
- Neural models
- Text sent to the model
- Chunk — Each document is split into chunks short enough to carry one idea, long enough to remain understandable out of context.
- Embed and store — An embedding model maps each chunk to a vector; vector, text and provenance are stored in the index.
- Embed the question — When a question arrives, the same model projects it into the same vector space.
- Retrieve — The index returns the chunks whose vectors are closest to the question's.
- Rerank — A costlier model re-reads each question–chunk pair and keeps only the most relevant.
- Generate — The prompt gathers the instructions, the selected passages and the question; the model writes the answer relying on those passages.
Finding the right passages
Question and passage are projected into one space; their closeness is measured by a dot product. Passage vectors are computed ahead of time, so a query reduces to a nearest-neighbour search2.
| Symbol | Meaning | Unit |
|---|---|---|
| Question | ||
| Passage (document chunk) | ||
| Question encoder, returning a -dimensional vector | ||
| Passage encoder, of the same dimension |
Sources:2
BM25 counts the question's words found in the document, giving more weight to rare words and correcting for length3. It remains hard to beat on exact terms: proper names, reference numbers, codes.
| Symbol | Meaning | Unit |
|---|---|---|
| -th term of the query | ||
| Number of occurrences of in document | ||
| Inverse document frequency: high for a term that is rare in the corpus | ||
| Length of the document | words | |
| Average document length in the corpus | words | |
| Free parameters: term-frequency saturation and length normalisation |
Sources:3
Question–chunk similarity
| Criterion | Lexical (BM25) | Dense (embeddings) | Hybridcommon |
|---|---|---|---|
| Matches | the same words | the same meaning | both |
| Strength | exact terms, identifiers, acronyms | paraphrases, synonyms | robustness |
| Weakness | different vocabulary in question and document | rare terms or terms unseen in training | two indexes to maintain, score fusion to tune |
| Model required | none | embedding model4 | embedding model |
Lexical (BM25)
- Matches
- the same words
- Strength
- exact terms, identifiers, acronyms
- Weakness
- different vocabulary in question and document
- Model required
- none
Dense (embeddings)
- Matches
- the same meaning
- Strength
- paraphrases, synonyms
- Weakness
- rare terms or terms unseen in training
- Model required
- embedding model4
Hybridcommon
- Matches
- both
- Strength
- robustness
- Weakness
- two indexes to maintain, score fusion to tune
- Model required
- embedding model
VerdictThe two families fail on different cases, which is why they are often combined, with a reranker doing the final sort5.
Generating from the passages
The retrieved passage is treated as a latent variable: the probability of the answer is an average over the top passages, weighted by their retrieval score1. Inserting the passages into a language model's prompt is a simplified version, in which the model reads all passages at once.
| Symbol | Meaning | Unit |
|---|---|---|
| Input (the question) | ||
| Generated sequence, with tokens | ||
| Retrieved passage | ||
| Retriever, with parameters | ||
| Generator, with parameters |
Sources:1
Instructions
Answer from the passages provided; say you do not know if the information is not there.
Numbered passages
Each passage carries an identifier and its provenance, so the answer can point back to it.
Question
The original question, as asked.
Cited answer
Each statement points to the passage that supports it, which makes the answer checkable.
Where the pipeline breaks
| Stage | Symptom | Likely cause | What to try |
|---|---|---|---|
| Corpus | The model says “I don't know” or makes things up | The information is simply not in the index | Extend the corpus; make the model decline rather than invent |
| Chunking | Partial or out-of-context answer | The chunk cuts through a table, a list or an argument | Split along structure; overlap; attach the section title |
| Retrieval | The right passage exists but is not returned | Vocabulary gap, too small, ambiguous question | Hybrid search; raise ; rewrite the question |
| Reranking | The right passage is retrieved, then dropped | Threshold too strict or context budget too short | Keep more passages; measure before and after reranking |
| Generation | The answer contradicts or ignores the passages | Conflicting or too many passages, or vague instructions | Fewer, better-sorted passages; require citations; check faithfulness |
| Index | Right yesterday, wrong today | Stale index or competing versions of one document | Incremental re-indexing; dates and versions as metadata |
A measure of retrieval alone: of the passages that had to be found, what share appears in the top ? It is computed without calling the language model.
| Symbol | Meaning | Unit |
|---|---|---|
| Set of passages relevant to the question (hand-annotated) | ||
| Set of the passages returned by the retriever |
Check your understanding
QuestionA RAG assistant answers a question badly. The logs show that the passage holding the answer was indeed in the prompt. Where should you look first?
- AOn the generation side: instructions, number and order of passages, conflicting passages.
- BAt the embedding model, which should be replaced.
- CAt chunking, by making the chunks smaller.
Show the answer
- A — Correct. Correct: retrieval did its job; it is the use of the passage that failed.
- B — Incorrect. The passage was found: the embedding is not at fault for this question.
- C — Incorrect. Possible if the chunk was truncated, but not the first lead when the full answer is in the prompt.
Does RAG eliminate hallucinations?
No. It reduces them when the right passages are supplied and the model sticks to them; the original paper observes more factual and more specific generations than with a purely parametric model1. But the model can still misread a passage, wrongly combine two, or fill gaps from memory.
What chunk size should I use?
There is no universal value: it is a trade-off to measure on your own questions. Too small and the chunk loses its context; too large and its vector blends several topics, diluting the useful passage. Following the document's structure (sections, paragraphs) is a better starting point than a fixed number of characters.
Common mistake: changing the embedding model without re-indexing
Vectors from two different models do not live in the same space: comparing a question encoded by one with chunks encoded by the other is meaningless. Any change of embedding model requires recomputing the whole index.
Does a very large context window make RAG unnecessary?
For a small, stable corpus, putting everything in the prompt may be enough. Beyond that, retrieval remains useful: it bounds the cost per request, keeps a changing corpus up to date, enforces per-document access rights and shows where each piece of information comes from.
References
- (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. arxiv.org/abs/2005.11401
- (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6769–6781. doi:10.18653/v1/2020.emnlp-main.550
- (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 4(1–2), 1–174. doi:10.1561/1500000019
- (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3980–3990. doi:10.18653/v1/D19-1410
- (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997. arxiv.org/abs/2312.10997