Aller au contenu

IAInformation RetrievalArchitectureIntermédiaire

Par · Publiée le · 10 min de lecture

Sommaire
  1. Two phases, two moments
  2. Finding the right passages
  3. Generating from the passages
  4. Where the pipeline breaks
  5. Check your understanding

Après cette fiche, vous saurez

  • Separate the indexing phase from the query phase.
  • Explain how a question is matched to passages, by dense or lexical search.
  • Locate each typical failure at the stage that causes it.
  • Choose metrics that evaluate retrieval and generation separately.

Formulaire

Les équations de la fiche, d’un coup d’œil

  1. 01Dense retrievalsim(q,p)=EQ(q)⊤EP(p)\mathrm{sim}(q, p) = E_Q(q)^{\top} E_P(p)
  2. 02Lexical retrieval: BM25score(D,Q)=∑i=1nIDF(qi)⋅f(qi,D) (k1+1)f(qi,D)+k1(1−b+b ∣D∣avgdl)\mathrm{score}(D, Q) = \sum_{i=1}^{n} \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \left(1 - b + b\,\frac{|D|}{\mathrm{avgdl}}\right)}
  3. 03RAG-Sequence modelpRAG-Sequence(y∣x)  ≈∑z ∈ top-k(pη(⋅∣x))pη(z∣x)∏i=1Npθ(yi∣x,z,y1:i−1)p_{\text{RAG-Sequence}}(y \mid x) \;\approx \sum_{z \,\in\, \text{top-}k\left(p_\eta(\cdot \mid x)\right)} p_\eta(z \mid x) \prod_{i=1}^{N} p_\theta\big(y_i \mid x, z, y_{1:i-1}\big)
  4. 04Recall at kRecall@k=∣ R∩Tk ∣∣ R ∣\mathrm{Recall@}k = \frac{\left|\,\mathcal{R} \cap \mathcal{T}_k\,\right|}{\left|\,\mathcal{R}\,\right|}

Demonstration example — not reviewed by the author

Architecture · LLM systems

The RAG pipeline, from indexing to answer

A language model only knows what its parameters hold: its memory is frozen at the end of training, and it cannot say where it came from. Retrieval-Augmented Generation (RAG) gives it a searchable external memory: before answering, the system fetches relevant passages and hands them to the model1.

  • Parametric memory

    The model's weights. Vast, but frozen, without provenance, and costly to update.

  • Non-parametric memory

    An index of documents. It can be corrected or extended without retraining the model1.

  • The link between them

    A retriever that, for each question, selects the few passages to put in front of the model.

Two phases, two moments

Indexing (offline) and query (online)

Two-lane diagram. Top lane, indexing: documents are split into chunks, each chunk is turned into a vector by an embedding model, then stored in a vector index. Bottom lane, query: the question is turned into a vector by the same model, the retriever queries the index and returns the k nearest chunks, a reranker reorders them, the prompt assembles question and passages, and the language model generates an answer together with its sources.INDEXING · OFFLINEQUERY · ONLINEDocumentsPDFs, pages, tickets…Chunkingchunks + metadataEmbedding modelchunk → vectorVector indexvectors + text + sourceQuestionEmbedding modelquestion → vectorRetrievalk nearest neighboursRerankingoptional, more precisePrompt assemblyinstructions + passages +questionLanguage modelSourced answerwritereadk candidatesthe best onesoriginal question
  • Indexing flow (once per document)
  • Query flow (for every question)
  • Reading the index
  • Neural models
  • Text sent to the model
  1. Chunk — Each document is split into chunks short enough to carry one idea, long enough to remain understandable out of context.
  2. Embed and store — An embedding model maps each chunk to a vector; vector, text and provenance are stored in the index.
  3. Embed the question — When a question arrives, the same model projects it into the same vector space.
  4. Retrieve — The index returns the kk chunks whose vectors are closest to the question's.
  5. Rerank — A costlier model re-reads each question–chunk pair and keeps only the most relevant.
  6. Generate — The prompt gathers the instructions, the selected passages and the question; the model writes the answer relying on those passages.
Indexing happens once per document; the query path runs again for every question. Both phases must use the same embedding model.

Finding the right passages

Dense retrieval
sim(q,p)=EQ(q)⊤EP(p)\mathrm{sim}(q, p) = E_Q(q)^{\top} E_P(p)

Question and passage are projected into one space; their closeness is measured by a dot product. Passage vectors are computed ahead of time, so a query reduces to a nearest-neighbour search2.

Variables
SymbolMeaningUnit
qqQuestion
ppPassage (document chunk)
EQE_QQuestion encoder, returning a dd-dimensional vector
EPE_PPassage encoder, of the same dimension

Sources:2

Lexical retrieval: BM25
score(D,Q)=∑i=1nIDF(qi)⋅f(qi,D) (k1+1)f(qi,D)+k1(1−b+b ∣D∣avgdl)\mathrm{score}(D, Q) = \sum_{i=1}^{n} \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \left(1 - b + b\,\frac{|D|}{\mathrm{avgdl}}\right)}

BM25 counts the question's words found in the document, giving more weight to rare words and correcting for length3. It remains hard to beat on exact terms: proper names, reference numbers, codes.

Variables
SymbolMeaningUnit
qiq_iii-th term of the query QQ
f(qi,D)f(q_i, D)Number of occurrences of qiq_i in document DD
IDF(qi)\mathrm{IDF}(q_i)Inverse document frequency: high for a term that is rare in the corpus
∣D∣|D|Length of the documentwords
avgdl\mathrm{avgdl}Average document length in the corpuswords
k1, bk_1,\, bFree parameters: term-frequency saturation and length normalisation

Sources:3

Question–chunk similarity

Matrix of three questions by five chunks. Each question has a high similarity with one or two chunks on the same subject and a low one with the rest; with $k = 2$, the two darkest cells of each row would be kept.ILLUSTRATIVE DATAReturnsShippingWarrantyBatteryCareersReturn period?Shipping fees?Battery warranty?0.860.410.330.120.050.380.840.150.100.060.290.130.810.770.040.040.86
Illustrative values, chosen by hand: they show what is expected of a good embedding model, not the output of a real one.
CriterionLexical (BM25)Dense (embeddings)Hybridcommon
Matchesthe same wordsthe same meaningboth
Strengthexact terms, identifiers, acronymsparaphrases, synonymsrobustness
Weaknessdifferent vocabulary in question and documentrare terms or terms unseen in trainingtwo indexes to maintain, score fusion to tune
Model requirednoneembedding model4embedding model

Lexical (BM25)

Matches
the same words
Strength
exact terms, identifiers, acronyms
Weakness
different vocabulary in question and document
Model required
none

Dense (embeddings)

Matches
the same meaning
Strength
paraphrases, synonyms
Weakness
rare terms or terms unseen in training
Model required
embedding model4

Hybridcommon

Matches
both
Strength
robustness
Weakness
two indexes to maintain, score fusion to tune
Model required
embedding model

VerdictThe two families fail on different cases, which is why they are often combined, with a reranker doing the final sort5.

Generating from the passages

RAG-Sequence model
pRAG-Sequence(y∣x)  ≈∑z ∈ top-k(pη(⋅∣x))pη(z∣x)∏i=1Npθ(yi∣x,z,y1:i−1)p_{\text{RAG-Sequence}}(y \mid x) \;\approx \sum_{z \,\in\, \text{top-}k\left(p_\eta(\cdot \mid x)\right)} p_\eta(z \mid x) \prod_{i=1}^{N} p_\theta\big(y_i \mid x, z, y_{1:i-1}\big)

The retrieved passage zz is treated as a latent variable: the probability of the answer is an average over the top kk passages, weighted by their retrieval score1. Inserting the passages into a language model's prompt is a simplified version, in which the model reads all passages at once.

Variables
SymbolMeaningUnit
xxInput (the question)
yyGenerated sequence, with tokens y1,…,yNy_1, \dots, y_N
zzRetrieved passage
pη(z∣x)p_\eta(z \mid x)Retriever, with parameters η\eta
pθp_\thetaGenerator, with parameters θ\theta

Sources:1

  1. Instructions

    Answer from the passages provided; say you do not know if the information is not there.

  2. Numbered passages

    Each passage carries an identifier and its provenance, so the answer can point back to it.

  3. Question

    The original question, as asked.

  4. Cited answer

    Each statement points to the passage that supports it, which makes the answer checkable.

Where the pipeline breaks

Typical failure modes, by stage. A summary of engineering practice5.
StageSymptomLikely causeWhat to try
CorpusThe model says “I don't know” or makes things upThe information is simply not in the indexExtend the corpus; make the model decline rather than invent
ChunkingPartial or out-of-context answerThe chunk cuts through a table, a list or an argumentSplit along structure; overlap; attach the section title
RetrievalThe right passage exists but is not returnedVocabulary gap, kk too small, ambiguous questionHybrid search; raise kk; rewrite the question
RerankingThe right passage is retrieved, then droppedThreshold too strict or context budget too shortKeep more passages; measure before and after reranking
GenerationThe answer contradicts or ignores the passagesConflicting or too many passages, or vague instructionsFewer, better-sorted passages; require citations; check faithfulness
IndexRight yesterday, wrong todayStale index or competing versions of one documentIncremental re-indexing; dates and versions as metadata
Recall at k
Recall@k=∣ R∩Tk ∣∣ R ∣\mathrm{Recall@}k = \frac{\left|\,\mathcal{R} \cap \mathcal{T}_k\,\right|}{\left|\,\mathcal{R}\,\right|}

A measure of retrieval alone: of the passages that had to be found, what share appears in the top kk? It is computed without calling the language model.

Variables
SymbolMeaningUnit
R\mathcal{R}Set of passages relevant to the question (hand-annotated)
Tk\mathcal{T}_kSet of the kk passages returned by the retriever

Check your understanding

QuestionA RAG assistant answers a question badly. The logs show that the passage holding the answer was indeed in the prompt. Where should you look first?

  1. AOn the generation side: instructions, number and order of passages, conflicting passages.
  2. BAt the embedding model, which should be replaced.
  3. CAt chunking, by making the chunks smaller.
Show the answer
  • A — Correct. Correct: retrieval did its job; it is the use of the passage that failed.
  • B — Incorrect. The passage was found: the embedding is not at fault for this question.
  • C — Incorrect. Possible if the chunk was truncated, but not the first lead when the full answer is in the prompt.
Does RAG eliminate hallucinations?

No. It reduces them when the right passages are supplied and the model sticks to them; the original paper observes more factual and more specific generations than with a purely parametric model1. But the model can still misread a passage, wrongly combine two, or fill gaps from memory.

What chunk size should I use?

There is no universal value: it is a trade-off to measure on your own questions. Too small and the chunk loses its context; too large and its vector blends several topics, diluting the useful passage. Following the document's structure (sections, paragraphs) is a better starting point than a fixed number of characters.

Common mistake: changing the embedding model without re-indexing

Vectors from two different models do not live in the same space: comparing a question encoded by one with chunks encoded by the other is meaningless. Any change of embedding model requires recomputing the whole index.

Does a very large context window make RAG unnecessary?

For a small, stable corpus, putting everything in the prompt may be enough. Beyond that, retrieval remains useful: it bounds the cost per request, keeps a changing corpus up to date, enforces per-document access rights and shows where each piece of information comes from.

References

  1. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. arxiv.org/abs/2005.11401
  2. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6769–6781. doi:10.18653/v1/2020.emnlp-main.550
  3. Stephen Robertson, Hugo Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 4(1–2), 1–174. doi:10.1561/1500000019
  4. Nils Reimers, Iryna Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3980–3990. doi:10.18653/v1/D19-1410
  5. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997. arxiv.org/abs/2312.10997

Aussi disponible en français

Partager cette fiche

Mots-clés

En vedetteIAApprentissage profond

Mélange d’experts et routage top-k

Un mélange d’experts confie chaque entrée à quelques sous-réseaux choisis par un routeur. Le nombre de paramètres augmente avec le nombre d’experts, mais le calcul par jeton ne dépend que de kk.

Avancé · 11 min de lecture ·

En vedetteIAApprentissage profondEN

Mixture of Experts and top-k routing

A mixture of experts hands each input to a few sub-networks chosen by a router. The parameter count grows with the number of experts, but the computation per token depends only on kk.

Avancé · 11 min de lecture ·

En vedetteÉconométrieFinance

Le modèle GARCH(1,1) et la dynamique de la volatilité

Comment une récurrence à trois paramètres reproduit les grappes de volatilité des rendements financiers, à quelles conditions elle est stationnaire et comment on l’estime par quasi-maximum de vraisemblance.

Intermédiaire · 10 min de lecture ·