When RAG is the wrong tool: the counter-cases that matter

It means the problem's shape does not match what retrieval-augmented generation does. RAG does exactly one thing: it puts relevant existing text in front of a model at inference time so the answer is conditioned on that text rather than on the model's weights alone (07-09). Everything RAG is good at follows from that, and so does everything it cannot do.

The five counter-cases, stated as diagnostics:

What this lesson is not saying. It is not saying RAG is over-hyped or usually wrong. For the large class of problems where a document corpus holds the answer and freshness and provenance matter, RAG is the right architecture and the previous eleven lessons stand. The claim is narrower and more useful: RAG has a domain, the domain has edges, and the edges are identifiable in advance.

The single most useful diagnostic question, which subsumes most of the table: can you name the specific document that answers this question? If yes, RAG is probably right. If the answer is "no document — it's in the database", you need tool calling. If it is "no document — nobody wrote it down", you need to create the corpus. If it is "any document, that's not the problem, the problem is the tone", you need prompting or fine-tuning.

L1 — The intuition you can carry into an exam

RAG injects facts from documents. If the problem is not "the model lacks facts that exist in documents", RAG is not the tool. The three most examinable counter-cases: no corpus, style or format change, hard latency floor.

Three facts to carry:

• Facts from documents → RAG. Behaviour and format → fine-tuning. Live values and computation → tool calling.
• The RAG heuristic is [FIELD] calibration, not official guidance — a tie-breaker with known counter-cases.
• If the corpus fits in the context window, retrieval may be pure overhead.

L2 — The five counter-cases in detail

Counter-case 1 — there is no corpus.

RAG retrieves. If nothing exists to retrieve, there is no pipeline to build. Three variants, each with a different remedy:

The failure mode when this is misdiagnosed is instructive: a team builds the whole nine-stage pipeline over whatever documents they do have, the retriever dutifully returns the nearest passages, and the model produces confident answers derived from documents that never addressed the question. A retriever over an irrelevant corpus is a hallucination engine with provenance. Citations make this visible (07-11), which is exactly why they matter here.

Counter-case 2 — the requirement is behavioural rather than factual.

This is the RAG-versus-fine-tuning boundary and it is on the study guide's explicit confusable list. The test:

Retrieving a style guide does not change how a model writes. This is worth sitting with, because it is the single most common category error in applied LLM work. Putting your tone-of-voice document in the context gives the model information about your house style. It does not give it the style. Style, format discipline, and persona live in behaviour, and behaviour is changed by prompting first (cheap, immediate, revisable) and fine-tuning second (expensive, durable). 11-08 owns the full decision rule and 11-02 covers exactly what supervised fine-tuning can and cannot change.

Counter-case 3 — a hard latency floor.

Count the retrieval budget honestly (12-10):

query rewriting (multi-turn)     one LLM call, or a cheap heuristic     (12-11)
query embedding                  one encoder forward pass               (07-02)
sparse + dense retrieval         parallel; index-dependent              (07-01, 07-04)
fusion                           negligible                             (07-06)
reranking                        N forward passes — often the largest   (07-07)
longer prompt                    prefill scales with input tokens       (07-08, 12-10)
─────────────────────────────────────────────────────────────────────
total: a real, additive budget on every single request

If your end-to-end target is tight — an autocomplete, an inline suggestion, a high-QPS classification path — that budget may not fit. The honest options: drop the reranker (losing precision), shrink k (losing context), cache aggressively for repeated queries, precompute answers for a known question set, or decide the task does not need an LLM at all. Note that reranking is usually the largest single term, which makes it the first thing to trade away under latency pressure, and 07-07's pool-depth curve is where you find out how much quality that costs.

Counter-case 4 — the answer needs computation or a live system of record.

The pattern: RAG answers "what does our documentation say?"; tool calling answers "what is currently true in our systems?" These are different questions and they need different architectures, though they compose well — a support assistant may retrieve the refund policy (RAG) and fetch the customer's order status (tool call) in the same turn. 12-11 and the agent patterns it points at are where composition lives.

A specific and common misdiagnosis: indexing a nightly database export as documents so RAG can "answer" data questions. This produces a system that is confidently stale, cannot aggregate, cannot filter numerically, and cannot be audited against the source of truth. If the answer is a number from a database, query the database.

Counter-case 5 — the corpus fits in the context window.

If the entire corpus is a 40-page handbook — a few tens of thousands of tokens — you can send it whole and ask the question. That eliminates every retrieval failure mode in this module: no parsing granularity problem, no chunk boundaries, no encoder mismatch, no ANN recall, no fusion tuning, no reranking, no assembly ordering.

The honest reading: whole-corpus stuffing is right for a genuinely small, static, non-sensitive corpus and a low query volume. It becomes wrong quickly — as the corpus grows, as volume grows, or the moment different users must see different documents. But dismissing it reflexively is the same over-application error this lesson is about, and 07-04 made the parallel argument about vector databases: sometimes the simplest thing is correct.

L3 — The over-application patterns, and the honest ordering

Beyond the five clean counter-cases, there are patterns where RAG is technically applicable and still the wrong first move:

Building RAG before measuring the baseline. Sometimes the ungrounded model already answers the question, because the information is genuinely public and in its pretraining data. Measure it (01-08) before building a pipeline to inject facts the model already has.

Building RAG when the real problem is upstream. A team's answers are bad because their documentation is bad. RAG faithfully retrieves bad documentation and produces bad answers with citations. Retrieval quality is bounded by corpus quality (08-01), and no pipeline stage compensates for a corpus that is wrong, contradictory, or absent.

Building the full nine stages for a prototype. 07-09's build order exists for this: eval set, assertions, BM25, dense, exact search, fuse, rerank, assemble, ground — infrastructure last. Standing up a vector database in week one is 07-04's over-engineering warning.

Using RAG where determinism is required. If a question must always produce the identical answer — a regulatory disclosure, a policy statement, a price — retrieval plus generation is a poor mechanism, because both stages have variance (09-11). A templated answer from a maintained source of truth is correct and auditable. Not everything should be generated.

The customisation ladder in the right order, which is the general form of all of this (11-08, and 05-06 for the first decision rule):

1. Prompt engineering        cheapest, fastest, revisable      → try first, always
2. RAG                       adds facts, freshness, provenance → when facts are the gap
3. Prompt learning / PEFT    adds behaviour cheaply            → when behaviour is the gap
4. Full fine-tune            adds behaviour durably            → rarely, and last
5. Alignment                 adds preference shaping           → specialist territory

Rung 2 is not rung 1. A surprising fraction of "we need RAG" turns out to be "we need a better prompt", and prompting is a Tier-1 exam topic [FIELD] for exactly this reason.

This is the highest-value table in the lesson. The rows are the axes exam scenarios are built on.

Four readings, all directly examinable.

The RAG-versus-fine-tuning boundary is facts versus behaviour. RAG injects facts with provenance and freshness; fine-tuning changes how the model writes and behaves. This is on the study guide's explicit confusable list and 11-08 owns it. The compressed test: if the right answer changes when the documents change, that is RAG; if the right answer changes when you want a different kind of answer, that is fine-tuning.

Only RAG and tool calling give per-user access control. A fine-tuned model cannot unlearn a fact for one user (13-05), and a stuffed context is the same content for everyone. When a scenario mentions per-user permissions, fine-tuning is eliminated immediately.

Only RAG gives document provenance. Weights have no footnotes (07-11). When a scenario requires citing sources, fine-tuning is eliminated.

Only tool calling handles live values and computation. When a scenario says "current", "real-time", "balance", "count", or "calculate", RAG is the wrong option no matter how attractive the heuristic makes it look.

This is a constructed set of scenarios in the shape of exam items, written to exercise the counter-cases. They are illustrative, not reproduced from any question bank.

Scenario 1 — the style complaint

A legal team reports that an LLM assistant produces accurate summaries of their contracts but writes them in a conversational tone. They need summaries that follow their firm's formal drafting conventions, with defined terms capitalised and no contractions. What should be done?

The RAG-shaped distractor: "Index the firm's drafting style guide so the model can retrieve the conventions."

Why it fails: retrieving a style guide gives the model information about the conventions. It does not change how the model writes. The complaint explicitly says summaries are already accurate — there is no factual gap for RAG to fill.

Right answer: prompt engineering first — a detailed system prompt with explicit conventions and few-shot examples of correctly styled summaries (05-01, 05-02). If prompting proves insufficient at scale, supervised fine-tuning on a corpus of correctly styled summaries (11-02).

The tell: the facts are already right and the form is wrong. That is always a behavioural problem.

Scenario 2 — the live-value question

A customer service assistant must answer "what is the status of my order?" and "how much of my subscription credit remains?" for authenticated customers. What architecture should be used?

The RAG-shaped distractor: "Index order records and subscription data in a vector database and retrieve them per query."

Why it fails: order status and credit balance are live values in a system of record. An index is stale from the moment it is built; similarity search cannot filter numerically or aggregate; and an answer about a customer's balance derived from a nightly export is wrong in a way that generates support tickets rather than resolving them.

Right answer: tool calling. The assistant calls the order API and the billing API with the authenticated customer's id and answers from the live response. RAG may still serve the policy half of the same assistant — "what is the refund window?" — which is the composition case: RAG for what the documentation says, tools for what is currently true.

The tell: the words status, current, balance, remaining, how many. Live state and aggregation are tool-calling signals.

Scenario 3 — the latency floor

An IDE feature must suggest a completion within a very tight inline budget as the developer types. The team wants suggestions informed by the company's internal library documentation. What is the primary obstacle to a RAG approach?

The RAG-shaped distractor: "Add a reranking stage to improve the relevance of retrieved documentation."

Why it fails: the question asks about the obstacle, and adding a per-candidate model pass makes the obstacle worse. Reranking is typically the largest single latency term in a retrieval pipeline (07-07).

Right answer: the latency budget itself is the obstacle. Query embedding, retrieval, reranking, and a longer prefill are all additive on every keystroke-triggered request (12-10). Viable adaptations: precompute and cache suggestions for common contexts; retrieve without reranking; use a small local model with the relevant documentation already in a short static prompt; or accept that inline completion is not a RAG use case.

The tell: an explicit hard latency requirement in an interactive path.

Scenario 4 — the missing corpus

A company wants an assistant that answers "how do we decide whether to grant a customer a contract exception?" The decision is currently made case by case by three senior managers using judgement. There is no written policy. What should be done first?

The RAG-shaped distractor: "Build a RAG system over the company's existing contract repository."

Why it fails: the contract repository contains contracts, not the decision criteria. A retriever over it will return contracts, and the model will produce a confident answer about exception criteria synthesised from documents that never state any. The citations will point at real contracts that do not support the claims — which at least makes the failure detectable (07-11).

Right answer: create the corpus. Interview the three managers, document the criteria, review and approve the document, then index it. RAG cannot retrieve knowledge that was never written down, and the first deliverable of this project is a document, not a pipeline.

The tell: you cannot name the document that answers the question.

Scenario 5 — the small corpus

A team wants an assistant over a single 30-page onboarding handbook, used a few dozen times a day by new employees, with no per-user access restrictions. What is the simplest sufficient architecture?

The RAG-shaped distractor: "Chunk the handbook, embed it, load it into a vector database, and retrieve the top 5 chunks per query."

Why it fails — or rather, why it is not the simplest sufficient answer: the handbook is a few tens of thousands of tokens. It fits in a modern context window. Sending it whole eliminates chunking decisions, encoder consistency, index recall, fusion, reranking, and assembly ordering — the entire failure surface of this module — at a cost of tokens and prefill on a few dozen queries a day.

Right answer: send the handbook whole, with a grounding instruction and a citation requirement by section (07-11). Revisit if the corpus grows, volume grows, or access restrictions appear.

The tell: small, static, non-sensitive corpus and low volume. Note the qualifiers — change any one and the answer flips.

Reading the five scenarios together

In all five, a RAG option is present and plausible. That is what makes them the counter-cases to the heuristic. The discriminating skill is not "is RAG good?" — it is "does this problem's shape match what retrieval does?"

The last-but-one row deserves emphasis because it is the most common real situation. "It's bad" is not a diagnosis. Building RAG in response to an unmeasured complaint is how teams end up with a pipeline that solves a problem they did not have. The eval set comes first (01-08), then the diagnosis (07-10), then the tool.

Two reasons, and the second is the interesting one.

First, the objectives require tool choice. The official job-role frame describes an associate who performs system analysis against specifications, integrates new AI language models into existing systems, and identifies system data, hardware, or software components required to meet user needs (objective 4.4) [OFFICIAL]. Choosing an architecture is the associate's job, and choosing correctly requires knowing what each option cannot do. Objectives 1.3 and 4.2 (build LLM use cases such as RAG, chatbots, and summarizers) and 1.9 (prompt engineering) are both served, and the customisation ladder — prompt → RAG → prompt learning → PEFT → full fine-tune → alignment — is explicitly part of the course's scope, with 11-08 owning the full rule.

Second, and more specifically: the RAG heuristic needs a counterweight. Candidate reports converge on the finding that in scenario questions, when one option proposes building a RAG solution, it is usually the keyed answer [FIELD]. This must be carried with its uncertainty intact:

• It is [FIELD] calibration — published candidate reports and third-party breakdowns — not official NVIDIA guidance. It never overrides the blueprint.
• It is a tie-breaker for genuinely ambiguous items, not a rule to apply before reading the question.
• Its known counter-cases are no corpus, a need for style or format change, and a hard latency floor — the first three sections of this lesson.

Used properly, the heuristic is worth real marks: on an ambiguous scenario where two options are defensible and one builds RAG, lean RAG. Used improperly — as a reflex — it costs you exactly the questions that discriminate between a pass and a strong pass, because those are the questions written with a plausible RAG distractor and a different keyed answer.

The related [FIELD] calibration is worth pairing with it: NVIDIA-branded options tend to be favoured when two answers are technically defensible. Same status, same caution — a tie-breaker, not a rule. 12-13 and 13-01 cover the stack and the principles those questions draw on.

Question phrasings you should recognise:

Distractor families — and note that in this lesson the RAG option is often the distractor:

• "Index the style guide." RAG offered for a behavioural requirement. The most elegant trap in this family.
• "Index the database export." RAG offered for live values and aggregation.
• "Add reranking" offered where the stated constraint is latency. It makes the constraint worse.
• "Fine-tune on the documents." The mirror-image error: fine-tuning offered where facts, freshness, and provenance are needed. Unreliable at fact injection, and it destroys provenance and access control (13-05).
• "Use a larger context window" offered as though it removed retrieval's failure modes at scale. It does not, and it maximises lost-in-the-middle exposure (07-08).
• "Build RAG" offered before any measurement exists. Diagnose first (07-10, 01-08).
• "RAG eliminates hallucination" as a justification for choosing it. It reduces and makes detectable (07-11).

Mistakes 1 and 10 are mirror images and together they are the most consequential pair in applied LLM engineering. Facts and freshness and provenance are RAG. Style and format and behaviour are fine-tuning. Getting this backwards wastes a quarter in either direction, and it is on the study guide's explicit confusable list because it is that common.

Mistake 5 deserves the last word among the mistakes. A RAG system is a faithful mirror of its corpus. If the documentation is contradictory, out of date, or wrong, RAG will retrieve it accurately, cite it correctly, and answer wrongly — with provenance. The pipeline cannot be better than what it retrieves, and no lesson in this module changes that. Sometimes the right RAG project is a documentation project.

When the requirement is about how the model behaves rather than what it knows. The test is a single question: if you handed a human expert the right document, would they now produce the answer you want?

• Yes → the gap is knowledge → RAG.
• No, they'd still write it wrong → the gap is behaviour → fine-tuning.

The full comparison, with the properties that decide real cases:

Three notes that keep this honest:

They compose. A fine-tuned model that writes in your house style, retrieving current facts from your corpus, is a legitimate and common architecture. The question is never "which one" in the abstract; it is "which one addresses this gap".

Prompting comes before both. The customisation ladder starts at prompting for a reason: it is the cheapest, fastest, most revisable intervention, and a substantial fraction of both "we need RAG" and "we need fine-tuning" resolves to "we needed a better prompt" (05-02, 05-06).

Fine-tuning is a poor knowledge-injection mechanism, and this is the most important asymmetry in the table. Facts learned in weights cannot be cited, cannot be updated without retraining, cannot be filtered per user, and are absorbed unreliably — a fact seen a handful of times in a fine-tuning set may not be recalled at all. 11-02 covers what supervised fine-tuning can and cannot change and 11-08 owns the decision rule.

For a small corpus, yes. For a large one, no — and the reasons are cost, latency, positional attention, and access control rather than capacity.

What a bigger window does not change, restating from 07-08 because it is the load-bearing point:

Cost scales with tokens sent. A 128k-token prompt is a 128k-token bill on every request (12-09). Retrieval exists partly to send 600 tokens instead of 128,000.

Latency scales with input length. Prefill is proportional to input tokens and attention cost is quadratic in sequence length (04-01). Longer prompts are slower, per request, forever.

Positional effects persist. A longer context has more middle. A fact buried at position 400 of 600 is at maximum lost-in-the-middle exposure (07-08).

Access control is impossible. A stuffed context is the same content for every user. If different users must see different documents, retrieval with pre-filtering is the only mechanism (07-05). This alone eliminates whole-corpus stuffing for most enterprise systems.

Provenance is weaker. You can ask for citations against a stuffed document, but you have no retrieval trace, no per-chunk ids, and no way to validate that a cited section was actually consulted (07-11).

The synthesis, and the version worth carrying: context window and retrieval solve different problems. The window is how much the model can read at once. Retrieval is how you decide what is worth reading. A larger window makes retrieval's job easier — you can afford more chunks, and truncation pressure drops — and it does not make the job unnecessary at any scale where the corpus exceeds what you would want to pay to send every time.

RAG is the right tool when all of these hold:

1. The answer exists in text you control. Documents, not databases; written down, not tacit.
2. The corpus is large enough that sending it whole is impractical, or access control requires filtering.
3. Freshness matters more than a retraining cycle allows. Documents change and answers must follow.
4. Provenance matters. Someone needs to know where an answer came from.
5. The latency budget can absorb a retrieval round trip.
6. The requirement is factual, not stylistic or behavioural.

That is a large and important class of problems: enterprise documentation search, customer support over product docs, policy and compliance Q&A, technical assistance over manuals and runbooks, research assistance over paper collections. For all of them RAG is not merely defensible, it is the correct architecture, and the eleven lessons before this one are how you build it well.

RAG is the wrong tool when any of these hold:

1. There is no corpus — write it first.
2. The requirement is behavioural — prompt, then fine-tune.
3. The answer is a live value or a computation — call a tool.
4. The latency floor cannot absorb retrieval — cache, precompute, or reconsider the feature.
5. The corpus fits in the window and nothing needs filtering — send it whole.
6. The answer must be byte-identical every time — template it.
7. The corpus is wrong — fix the corpus.

Both lists matter, and a candidate who can produce both is in a different position from one who has only learned that RAG works. The exam heuristic — a RAG option is usually keyed [FIELD] — is a tie-breaker whose value depends entirely on knowing when it does not apply. Applied blindly it is a habit; applied with these counter-cases in mind it is judgement.

That distinction is the closing claim of this module. Eleven lessons taught the pipeline; this one drew its boundary. A technique you can only apply is a habit. A technique you can also decline to apply is engineering.

• Five counter-cases: no corpus, behavioural requirement, hard latency floor, computation or live state, and a corpus small enough to send whole. Each has a different right answer.
• The single best diagnostic: can you name the document that answers this question? No document because it is in the database → tool calling. No document because nobody wrote it → create the corpus. Any document, the problem is the tone → prompting or fine-tuning.
• Facts, freshness, provenance, and per-user filtering are RAG. Style, format, and behaviour are fine-tuning. Getting this backwards wastes a quarter in either direction, and it is an explicitly named confusable pair.
• Retrieving a style guide does not change how a model writes. The most elegant RAG distractor there is.
• Only RAG and tool calling support per-user access control; only RAG gives document provenance; only tool calling gives live values and aggregation. These three facts eliminate wrong options fast.
• The worked example's headline result: in all five scenarios a plausible RAG option was present and none of them was the right answer. The discriminating skill is matching the problem's shape to the tool, not rating the tool.
• The RAG heuristic is [FIELD] calibration, not official guidance. Use it as a tie-breaker on ambiguous items, and check no corpus, style change, and latency floor first.
• A bigger context window does not replace retrieval at scale — cost, latency, positional attention, and access control all persist (07-08).
• Retrieval quality is bounded by corpus quality. Sometimes the right RAG project is a documentation project.
• Prompting comes before both RAG and fine-tuning. A surprising fraction of "we need RAG" is "we need a better prompt".

This lesson kept arriving at the same place from different directions: the corpus. RAG cannot retrieve what was never written down, cannot improve on documentation that is wrong, and cannot filter on metadata nobody attached. Every one of those is a curation problem, and curation is the stage that decides the ceiling on everything the last twelve lessons built. Next: 08-01 covers how to curate a dataset for an LLM task — what belongs in it, how to define and label it, how to decide what to exclude, and how to handle the unanswerable cases that most curation guides omit entirely.