Debugging RAG: retrieval failure versus generation failure

It is a binary partition of every possible RAG failure, drawn at the context boundary:

                    a wrong answer
                          │
        ┌─────────────────┴─────────────────┐
        │                                   │
  Was the correct passage IN THE CONTEXT the model received?
        │                                   │
       NO                                  YES
        │                                   │
  RETRIEVAL FAILURE                  GENERATION FAILURE
  the model never had it             the model had it and misused it
        │                                   │
  stages 1–7:                        stages 8–9:
  parse, chunk, embed,               assemble, prompt,
  index, query-prep,                 ground, cite
  retrieve, rerank                          │
        │                                   │
  fix: 06-01, 06-02, 07-02,          fix: 07-08, 07-11,
       07-04, 07-06, 07-07                 05-02, 09-12

Why the boundary is drawn at the context and not somewhere else. The context window is the only place in the pipeline where you can make a clean, verifiable statement about what the model knew. Everything before it is about getting evidence; everything after is about using it. That makes the boundary a genuine information barrier: no upstream fix can change how the model used a passage it received, and no downstream fix can conjure a passage that was never retrieved.

The two failure classes have disjoint fixes, and that is the whole payoff. This is not a taxonomy for its own sake. Consider the wasted work each misdiagnosis causes:

Both are common. The second is more expensive, because changing the embedding model means re-embedding the corpus (12-12).

A third class exists and must be named to keep the binary honest: the answer may be unavailable because the information is not in the corpus at all. That is neither a retrieval nor a generation failure — it is an ingestion or curation failure (06-01, 08-01), and in some cases the correct response is that the question is unanswerable and the system should say so (07-11). Folding it into "retrieval failure" is a mistake, because it sends you tuning a retriever over a corpus that does not contain the answer.

L1 — The intuition you can carry into an exam

Read the prompt the model actually received. Is the answer in there? If no, fix retrieval. If yes, fix generation.

Three facts to carry:

• The first question is always "was the correct passage in the context?"
• Retrieval and generation failures have disjoint fixes. Diagnose before intervening.
• You cannot run this diagnosis without logging the assembled context. That log is the prerequisite for all of it.

L2 — The four-question decision tree

The full procedure is four questions, each one narrowing further. Run them in order; do not skip ahead.

Question 1: Is the answer in the corpus at all?

Search the corpus directly — by keyword, by grep, by whatever crude means you have — for the fact the answer needed.

This question first, because it is the cheapest and because everything downstream is meaningless if it fails. It is also the question people never ask, which is why a parsing defect can survive months of retrieval tuning.

Question 2: Was the correct chunk in the retrieved candidate pool?

Retrieve the top-50 for the failing query — deeper than production k — and search the list for the chunk containing the answer.

If it is absent from the top-50, drill one level:

Question 3: Was the correct chunk in the assembled context?

Present in the top-3 does not mean present in the prompt. Read the logged context.

Question 4: The passage was in the context and the answer is still wrong. Which generation failure is it?

L3 — What makes the diagnosis possible, and what makes it impossible

The procedure above is trivial to run if you have one artefact and impossible without it: the logged assembled context. Not the retrieved ids, not the scores — the actual text the model received, or at minimum the ordered list of chunk ids with their scores plus a way to reconstruct the text.

The minimum viable trace, per request:

request_id
query_raw            the user's text
query_prepared       after rewriting (12-11)
filter_predicate     the entitlement + status filter applied (07-05)
sparse_top_n         [(chunk_id, bm25_score, rank), ...]
dense_top_n          [(chunk_id, cosine, rank), ...]
fused                [(chunk_id, rrf_score, rank), ...]
reranked             [(chunk_id, rerank_score, rank), ...]
assembled            [chunk_id in prompt order]
prompt_tokens        the count actually sent
answer               the generated text
citations            [claimed source ids]

Every field earns its place. Without sparse_top_n and dense_top_n separately you cannot tell which leg failed. Without filter_predicate you cannot see that an over-restrictive status filter excluded the answer. Without assembled in prompt order you cannot diagnose lost-in-the-middle. Without prompt_tokens you cannot see truncation.

One caution that is not optional. 07-05 established that retrieved chunk bodies may be restricted content, and that logs are a leak site. The resolution: log ids, scores, ranks, and predicates always; log chunk bodies only in environments whose corpus contains nothing restricted. In production, reconstruct the text from ids at debug time, under the same authorisation as any other read.

Why the order of questions is not arbitrary. Each question is chosen to eliminate the largest amount of pipeline per unit of effort, and to run before any question whose answer it could invalidate:

Asking Q4 first — "let me improve the prompt" — is the intuitive move and the wasteful one, because it is the question that eliminates the least.

Three readings that exam items probe.

Ranking failure is a sub-case of retrieval failure with a completely different fix. Both mean "the passage was not in the context", but "absent from the top-50" and "at rank 12 of 50" point at opposite ends of the retrieval stack — recall interventions versus precision interventions (07-06 versus 07-07). Collapsing them is the most common way this diagnosis goes half-right.

Ingestion failure is the only fully silent class, and it is the one whose misdiagnosis is most expensive. A parser that dropped every table produces a system that retrieves fluently, ranks well, assembles cleanly, and answers wrongly on every table-derived question — and every metric downstream of parsing looks healthy.

"Unanswerable" is a correct outcome, not a failure. If the corpus genuinely does not contain the answer, the right behaviour is a decline (07-11). A system that hallucinates instead has a generation failure on top of an unanswerable question, and the fix is the decline path, not better retrieval.

This is a constructed diagnostic walkthrough with invented values, built so each decision point is inspectable. The scores and counts are illustrative, not measured.

The system is the nine-stage pipeline from 07-09: 12,000 chunks, hybrid retrieval, cross-encoder reranking with a 0.5 floor, edge-loaded assembly of up to 3 chunks. Five user complaints arrive. Same symptom class — "the answer was wrong" — five different stages.

Case 1 — "It never finds anything about the licence renewal process"

Q1: is it in the corpus? Grep the indexed text for "licence renewal". Zero hits. The source document exists — a scanned PDF of the licensing agreement.

Parse log for licence-agreement.pdf:
  extracted characters: 41
  source estimate:      ~38,000
  assertion (0.8×–1.2×): FAILED — flagged, but nobody read the flag

Diagnosis: ingestion failure. The PDF is a scan with no text layer, so parsing yielded 41 characters of metadata and nothing else. Fix: OCR at the parse stage (06-01). Diagnosis time: ninety seconds, entirely because the assertion existed.

Note the counterfactual. Without that parse assertion the team would have seen a retrieval symptom, added hybrid search, changed the embedding model, and deployed a vector database — none of which can retrieve text that was never extracted.

Case 2 — "Asking about the 60-day international travel window returns nothing useful"

Q1: in the corpus? Yes — the sentence is present, intact.

Q2: in the top-50? No. Absent entirely.

Drill down:

Self-retrieval on that chunk's own text → rank 1, similarity 0.997     ✓ encoder fine
BM25 alone on "international travel expense window"   → rank 34         ← found, barely
Dense alone                                            → not in top-50
Chunk length                                           → 1,940 tokens   ← the problem

The chunk is 1,940 tokens covering the entire expenses policy — domestic windows, international windows, receipt rules, currency conversion, approval chains. Its single vector is the average of six topics, so it is moderately similar to everything and strongly similar to nothing. BM25 found it weakly on term overlap; the dense leg could not, because its vector does not point at "international travel" specifically.

Diagnosis: retrieval failure caused by chunking (06-02). Fix: re-chunk at ~400 tokens with overlap. After re-chunking, the international-travel paragraph becomes its own chunk and appears at dense rank 2.

The instructive part: this looked like an embedding-model problem and was a chunking problem. The self-retrieval check exonerated the encoder in one call, which is what made the next hypothesis cheap to reach.

Case 3 — "It knows about certificate rotation but the restart step comes out fourth"

Q1: in the corpus. Q2: in the top-50 — at fused rank 4.

Fused ranking:   c8801 (0.0325) > c5512 (0.0323) > c2170 (0.0161) > c6033 (0.0159)
                                                                     ↑ the answer
Assembled (k=3): c8801, c5512, c2170
                 → c6033 never entered the context

Q3: in the assembled context? No — cut by k=3.

Diagnosis: ranking failure (07-07). The chunk was retrieved and then discarded by the context cutoff. It fused to rank 4 because only the dense leg found it, and RRF rewards mutual presence over single-leg conviction (07-06).

Fix: add the cross-encoder reranker. It scores c6033 at 0.93 — highest of the pool — promoting it to rank 1. No retrieval change was needed at all; the passage was always there.

This is precisely the case where a team without the diagnosis reaches for the embedding model, re-embeds 12,000 chunks, and finds the ranking unchanged — because the retrieval was never the problem.

Case 4 — "Sometimes it gets the international travel exception, sometimes it doesn't"

Q1: in the corpus. Q2: top-50, rank 2. Q3: in the assembled context? Yes — but look at where.

Assembled context, naive rank order, k=8:
  position 1  e1   30-day rule            ← strong position
  position 2  e2   60-day intl exception  ← being pushed toward the middle
  position 3  e3   review timeline
  position 4  e4   portal location
  position 5  e6   currency conversion
  position 6  e7   receipt requirements
  position 7  e8   approval chain
  position 8  e5   revision history       ← the WEAKEST chunk in the strongest end position

Diagnosis: assembly failure (07-08). Both answer-bearing chunks were present. e1 at position 1 is used reliably, which is why the 30-day answer is always right. e2 at position 2 of 8 sits in the region the model attends to least reliably — the lost-in-the-middle U-curve — which is why the exception appears intermittently. Meanwhile a revision-history line occupies the strong final position.

The intermittency is the tell. A failure that appears and disappears for the same question, with the same retrieval, is a positional or attention effect, not a retrieval one.

Fix, in two lines of assembly code: apply the 0.5 relevance floor (dropping e4, e5, and others) and edge-load — e1 first, e2 last. Chunk tokens fall, and both answer-bearing chunks land in strong positions.

Case 5 — "It told a customer to disable certificate verification"

Q1: in the corpus — an archived forum post says exactly that. Q2: top-50, rank 1. Q3: in the assembled context, position 1.

Q4: which generation failure?

Context position 1: archived forum post — "just turned off cert verification, worked for me"
Context position 3: official spec — "the server MUST abort the handshake …"

Answer: "Disable certificate verification on the client."
Citations: none — no citation instruction was in the prompt

Diagnosis: this is not primarily a generation failure. The model faithfully used its highest-ranked source. It did exactly what it was told. The failure is upstream, in two places:

1. Authority blindness (07-03). The forum post ranked first because it matches the query's register — first-person, symptomatic, informal — while the specification is written like a specification. Fix: trust-tier metadata at ingestion and a filter or boost on it; ideally, do not index content you would not cite.
2. No citation requirement (07-11). With citations, a reader would see the answer came from an archived forum post and could reject it. Without them, a dangerous answer is indistinguishable from a good one.

There is a genuine generation-side fix too — instruct the model to prefer authoritative sources and to flag when its best source is community content — but the primary defect is that the corpus offered a bad answer at rank 1.

This case is why the diagnosis has four questions rather than two. A crude retrieval-versus-generation split would land on "generation failure: the model said something dangerous", and the prompt fix would be a patch over a corpus problem.

Reading the five cases together

Five identical-sounding complaints, five stages, five disjoint fixes. A team that responds to all five by changing the embedding model fixes none of them and re-embeds the corpus five times. The diagnosis costs minutes per case; the misdiagnosis costs weeks.

Use this as a lookup once you have run the four questions — it maps observations onto the stage to inspect.

Two entries in that table are worth memorising as signatures because they identify their stage almost uniquely:

"Confident, fluent, irrelevant, normal-looking scores, no errors" is the embedding-mismatch signature. Nothing else in the pipeline produces exactly that combination, and the self-retrieval assertion confirms or excludes it in one call.

"Intermittently correct for the same question with the same retrieval" is the assembly signature. Retrieval failures are consistent — the chunk is either found or it is not. Positional and attention effects are probabilistic, so intermittency points at assembly rather than upstream.

The official job-role frame describes an associate who implements testing and debugging, performs system analysis against specifications, and assesses and resolves performance issues [OFFICIAL]. That is a description of this lesson. The objectives it serves:

• 1.3 / 4.2 — Build LLM use cases such as RAG, chatbots, and summarizers. Building includes debugging.
• 4.5 — Monitor functioning of data collection, experiments, and other software processes [OFFICIAL]. The per-request trace is that monitoring.
• 4.1 / 1.1 — Assist in deployment and evaluation of model scalability, performance, and reliability under supervision of senior team members [OFFICIAL]. Attribution is what makes an evaluation actionable.
• 1.4 — Curate and embed content datasets for RAGs. Half of all diagnoses land in curation.
• 1.9 — Use prompt engineering principles to create prompts to achieve desired results. Generation-side fixes are prompt fixes.
• Derived Experimentation scope — separating retrieval quality from generation quality is exactly the RAG evaluation decomposition the course index names, and 09-07 owns the metrics.

The exam is scenario-heavy and calibrated to correct practice and tool choice rather than deep technical derivation [FIELD]. Debugging scenarios are the ideal vehicle for that: a symptom is described, four plausible interventions are offered, and the keyed answer is the one matching the stage the symptom implicates. These are among the most answerable questions on the exam if you know the symptom-to-stage map, and among the most guessable-wrong if you do not.

Question phrasings you should recognise:

Distractor families:

• "Use a larger LLM." The default wrong answer for retrieval failures. A larger model cannot read a passage it was not given.
• "Change the embedding model." The default wrong answer for generation and ranking failures, and the most expensive one because it forces a full re-embed (12-12).
• "Increase k." Superficially addresses recall, frequently worsens assembly (07-08).
• "Add hybrid search" offered for recency, authority, permissions, or ranking problems. It addresses lexical/semantic coverage only (07-06).
• "Fine-tune the model on the documents." Does not add retrievable facts reliably, removes provenance, and cannot be filtered per user (11-08, 13-05).
• "Lower the similarity threshold." The wrong passage often scored high (07-03).
• Prompt engineering offered for a retrieval failure, and retrieval changes offered for a generation failure. The two canonical inversions.
• "Check the logs" where no context trace is logged. The distractor is plausible precisely because most systems cannot do it.

Mistake 11 is the one that converts debugging from firefighting into progress. Every diagnosed failure should end as a new item in the evaluation set, labelled with the correct chunk id. Over a few months that turns a 20-item hand-written eval set (01-08) into a hundred-item suite grown from real failures (09-01) — and it means a fix that regresses an old failure fails the build instead of a customer (10-04).

Mistake 9 deserves a note too. A single wrong answer is a hypothesis generator, not evidence. Users report individual failures; the honest response is to reproduce it, diagnose it, add it to the eval set, and then measure whether the proposed fix helps the set rather than the anecdote. A change that fixes one query and breaks three is a regression that felt like a success.

Because it eliminates roughly half the pipeline with one cheap observation, and because the two halves have no fixes in common.

Count the stages. The nine-stage pipeline (07-09) splits into stages 1–7 (getting evidence) and stages 8–9 (using it). A single look at the logged context tells you which half to ignore. No other question in RAG debugging has that yield:

The asymmetry is not subtle. And the disjointness of the fixes is what makes it consequential rather than merely efficient: work done on the wrong half is not partial credit, it is zero. Prompt engineering applied to a retrieval failure yields nothing, however good the prompt gets, because the text is not there. A new embedding model applied to a generation failure yields nothing, because the same passage arrives at rank 1 and the model misuses it the same way.

There is a second-order benefit worth naming: the question is cheap to answer correctly, which means it is hard to get wrong. Judging whether a prompt is good enough is a matter of taste; judging whether a specific sentence appears in a logged context is a matter of fact. Diagnostic steps that produce facts should always precede ones that produce opinions.

And a third: it makes the conversation concrete. "RAG isn't working well" cannot be assigned to anyone. "For query X, chunk c6033 was retrieved at fused rank 4 and cut by k=3" has an owner, a fix, and a test. That translation is most of what makes a RAG system operable.

By reading the context and the answer side by side and asking whether every claim in the answer is supported by something in the context. This is the faithfulness (or groundedness) check, and it is the generation-side counterpart to retrieval recall.

The distinction that catches people: a model that faithfully answers from bad evidence is not hallucinating. It did its job. Scoring it as a generation failure sends you to fix the prompt when the corpus or the retriever is at fault. This is one reason RAG evaluation must report retrieval and generation metrics separately rather than as one quality number (09-07).

Three checks that make this mechanical rather than impressionistic:

Per-claim citation. Instruct the model to attach a source id to every factual claim (07-11). Then an unsupported claim is visibly unsupported — it has no citation, or a citation you can check.

Programmatic citation validation. For each cited id, verify the claim's key terms actually appear in that chunk. This is crude and catches the worst cases: citations pointing at chunks that do not contain the claim. It is cheap enough to run on every response in a test suite.

LLM-as-a-judge for faithfulness, with its biases understood (09-10). Give a judge model the context and the answer and ask whether every claim is supported. Useful at scale, unreliable enough that you should not trust it alone — and note this is a different use of an LLM than reranking (07-07).

Grounding via retrieval and citation of sources are named hallucination controls [NVIDIA-DOC], and NeMo Guardrails is the NVIDIA instrument for keeping applications accurate, appropriate, on-topic, and secure — 13-02 covers it and 09-12 covers hallucination types. The point for this lesson is narrower: without citations you cannot run this diagnosis at all, because you have no way to check which claim came from which source.

Thirty minutes, five steps, one artefact at the end. Concretely:

Step 1 — Reproduce with the trace (5 min). Re-run the failing query and capture the full trace: prepared query, filter predicate, per-leg candidates with scores, fused ranking, reranked ranking, assembled order, token count, answer, citations. If you cannot capture this, stop and build the logging — everything else is speculation.

Step 2 — Run the four questions (5 min). In corpus? In top-50? In context? Which generation failure? Write down the answer to each; do not skip forward because you have a hunch. The hunch is usually the interesting stage, not the broken one.

Step 3 — Name the stage (1 min). One stage, by name, from the nine. If you cannot name one, you have not finished step 2. "Retrieval is bad" is not a stage; "chunking produced a 1,940-token chunk whose vector averages six topics" is.

Step 4 — Add the query to the eval set (5 min). With the correct chunk id labelled. This is the step that makes the session cumulative instead of disposable. Do it before the fix, so you have a failing test.

Step 5 — Fix one thing and re-measure (10 min plus). The single intervention that matches the named stage. Re-run the whole eval set, not just the failing query, because a fix that helps one query and breaks three is a regression (03-04).

What you have at the end, and this is the point: a named stage, a permanent test, a measured before-and-after, and a one-line explanation someone else can read. Compare that to the alternative session, which ends with "I changed some things and it seems better."

Two habits that compound over a project:

Keep the four-configuration table current. Sparse alone, dense alone, hybrid, hybrid plus rerank, each with recall@3, recall@10, and MRR on the frozen set (07-01, 07-06, 07-07). It answers most architecture questions before they are asked, and it stops the recurring proposal to remove a component nobody can defend.

Put the assertions in CI (10-04). Self-retrieval similarity, parse-length ratios, maximum chunk length, retrieval recall thresholds, and faithfulness on the eval set. A CI gate that fails the build catches the silent stages; a dashboard does not, because nobody reads a dashboard on a Tuesday. That gate is the week-5 deliverable of this course, and it is the difference between a system you operate and one you hope about.

• Ask one question first: was the correct passage in the context the model received? It eliminates roughly seven of nine stages either way, and it costs one log line.
• Retrieval and generation failures have disjoint fixes. Work on the wrong half is not partial credit; it is zero.
• Run four questions in order: is it in the corpus, is it in the top-50, is it in the assembled context, and which generation failure is it. Each eliminates more than the next.
• The worked example's headline result: five identical-sounding complaints diagnosed to five different stages — parse, chunk, rank, assemble, and corpus authority — with five disjoint fixes. Changing the embedding model would have fixed none of them.
• Two signatures worth memorising. Confident-fluent-irrelevant with normal scores and no errors is embedding mismatch. Intermittent correctness with unchanged retrieval is assembly.
• Ingestion failure is the only fully silent class, and Q1 exists to catch it before you tune anything downstream.
• "Absent from the top-50" and "rank 12 of 50" are different problems with opposite fixes — recall interventions versus reranking.
• A model that faithfully answers from bad evidence is not hallucinating. Scoring it as a generation failure sends you to the wrong stage.
• You cannot run this diagnosis without the trace. Log ids, scores, ranks, filter predicates, assembled order, and token counts — bodies only where nothing is restricted (07-05).
• Every diagnosed failure becomes a permanent eval item. That is what makes debugging cumulative rather than disposable.

Several diagnoses in this lesson ended at the same place: the model produced a claim, and there was no mechanism to check whether the context supported it. Citations are what make that checkable, and a decline path is what makes an empty retrieval produce an honest answer instead of an invented one. Next: 07-11 covers grounding and citations as hallucination controls — how to instruct a model to answer only from provided sources, how to render source ids so citations are verifiable rather than decorative, how to validate them programmatically, and why letting a model say "I don't know" is a feature that has to be deliberately designed rather than a failure to be prompted away.