RAG Fundamentals for Agents: The Canonical Retrieval Pipeline

Retrieval-Augmented Generation (RAG) is a technique for grounding a language model's output in external information retrieved at query time, rather than relying solely on facts baked into the model's parameters during pretraining [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md). The word "augmented" is doing real work in that name: RAG does not replace generation, it supplements it. The model still generates the answer, token by token, the same way it always does. What is different is what gets placed in front of it before it starts generating — a set of passages pulled from a knowledge source specifically because they are relevant to the current question.

The property worth sitting with before anything else: retrieval happens at inference time, not training time, and no model weights change as a result of it [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md). Compare this against fine-tuning, where you take a base model and continue training it on new examples until its parameters shift to encode new behavior or knowledge. Fine-tuning is expensive, requires a training run, and once it's done, updating a single fact means running training again. RAG has none of that shape. Updating a RAG system's knowledge means editing a document and re-running three cheap pipeline steps — no GPU cluster, no loss curve, no risk of catastrophically forgetting something else the model used to know. This is why RAG is preferred specifically when the underlying knowledge changes often, needs to be citable back to a source, or is simply too large or too dynamic to have ever been a candidate for baking into weights in the first place.

An agent is the natural home for this pattern, more so than a standalone chatbot, because an agent already has a loop for calling external things and incorporating their output. A retrieval step is, from the agent's point of view, just one more tool: given a question, it returns text. The agent's reasoning loop treats a retrieval call exactly the way it treats a calculator call or an API call — invoke it, get a result, fold the result into the next decision. That is a genuinely different posture from the way RAG usually gets taught as a standalone information-retrieval application, and it matters enough that it's worth stating plainly now, before the pipeline mechanics: this lesson treats RAG as one tool an agent calls partway through a reasoning loop, not as the entire application. A dedicated RAG service built for a single question-answering interface has the luxury of assuming every request is a retrieval request. An agent does not have that luxury — it must first decide whether retrieval is even the right move for the current step, and only then invoke it. M6-03 returns to this distinction directly when it introduces agentic RAG, where the agent doesn't just call retrieval once but reasons about when and how many times to call it.

L1 — The intuition you can carry into an exam

Five verbs, in order: ingest and chunk, embed, store, retrieve, augment and generate [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md). Three of those verbs run once, offline, whenever your source documents change. Two of them run on every single question the agent is asked. Keep that split in your head — it is the same split that shows up in every serious discussion of retrieval systems, under different names, because it reflects a real difference in cost and frequency: preparing a library happens rarely; answering a question from it happens constantly.

L2 — What each stage does, and why it exists in that order

Stage 1 — Ingest and chunk. Source documents — PDFs, wiki pages, support tickets, whatever the agent needs to be grounded in — are pulled into the pipeline and split into smaller passages, typically a few hundred words each. Chunking exists because of a mechanical constraint: you cannot usefully embed or retrieve a 40-page document as a single unit, because a single embedding vector for 40 pages of mixed content represents none of it precisely, and because the whole point of retrieval is fetching only the passages relevant to the current question, not the entire source document every time. The chunk is the unit of everything that follows — the unit you embed, the unit you store, the unit you retrieve.

Stage 2 — Embed. Each chunk is passed through an embedding model, which converts the chunk's text into a dense numeric vector — a point in a high-dimensional space where semantic similarity between two pieces of text corresponds, approximately, to geometric closeness between their vectors. This is the stage that turns "text passages" into "things you can do math on," and it's the stage M6-02 picks up in detail, because the specific failure mode that governs vector retrieval — query and document embeddings needing to come from the same model and vector space — belongs to that lesson, not this one. For now, hold onto the shape: text in, vector out, one vector per chunk.

Stage 3 — Store. The vectors, along with enough metadata to trace each one back to its source chunk, are written into a vector database — a system built to hold large numbers of these vectors and search them efficiently. This is also M6-02's territory in depth; here it's enough to know that storage is a distinct stage from embedding, because you embed once per chunk at ingestion time but you search the resulting store on every single query afterward, and those are different operational problems with different performance requirements.

Stage 4 — Retrieve. This is where the online path begins. The user's (or agent's) query is itself embedded, using the same embedding model as stage 2, and compared against every stored chunk vector using semantic search — a similarity search over the vector space, returning the chunks whose vectors sit closest to the query's vector. The output of this stage is a short list, typically single digits to a few dozen chunks, of passages judged most relevant to the question actually being asked right now.

Stage 5 — Augment and generate. The retrieved chunks are inserted into the prompt sent to the language model, usually with instructions to answer using only (or primarily) the supplied context, and the model generates its response. This is the step that closes the loop: the model's output is now conditioned on real, current, specific text that was not part of its training data, rather than solely on whatever it happened to memorize about the topic during pretraining.

L3 — The exam-relevant edge case: what RAG demonstrably does not fix

The trap worth internalizing at depth, because it is exactly the kind of thing a scenario question is built to test: RAG reduces hallucination by grounding generation in retrieved evidence; it does not eliminate hallucination, and it does not repair bad data [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md). Both halves of that sentence matter and get missed for different reasons.

The first half — RAG reduces but does not eliminate hallucination — gets missed because the whole point of RAG is so obviously "stop the model from making things up" that it is tempting to round that benefit up to "solves hallucination entirely." It doesn't. A model given five retrieved passages can still misread them, conflate two of them, generate a plausible-sounding claim that isn't actually supported by any of the retrieved text, or simply ignore the retrieved context and answer from its own parametric memory anyway if the prompt doesn't force it to attend to what was retrieved. Retrieval narrows the space of ways the model can be wrong; it does not close that space to zero.

The second half is the one this lesson wants to leave you with most firmly, because it is the thread that M6-04 picks up as its entire subject: retrieval quality is capped by the quality of the underlying data, and no amount of retrieval sophistication compensates for source documents that were duplicated, outdated, contradictory, or simply wrong before they were ever chunked. If your knowledge base contains a support article that was correct two product versions ago and never got updated, a perfect retrieval pipeline will retrieve it perfectly, hand it to the model exactly as instructed, and the model will confidently generate an answer that is wrong — not because retrieval failed, but because retrieval succeeded at retrieving bad data. This is why ETL and data-quality work sit inside the same domain as the pipeline itself rather than being treated as separate infrastructure concerns: they are not optional plumbing underneath a RAG agent, they are a precondition for the pipeline's output being trustworthy at all [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md).

⭐ THE EARNED INSIGHT: The five-stage pipeline answers "how does an agent fetch external knowledge," but it deliberately does not answer "should the agent fetch it, how many times, and from where." Treating RAG as a single black-box tool call an agent's reasoning loop can invoke — rather than as the entire shape of the application — is what makes the rest of Domain 6 legible: M6-03's agentic RAG is what happens when an agent starts reasoning about that decision instead of always making the same one-shot call.

It is worth being explicit about a framing choice this lesson has made throughout, because a nearly identical topic gets taught elsewhere with a different emphasis, and the difference is not cosmetic. A standalone RAG application — the kind built as a dedicated question-answering service sitting behind a chat widget — treats every incoming request as a retrieval request by design, and the engineering problem is squeezing maximum accuracy out of each of the five stages: better chunking, a stronger embedding model, a reranking step, careful prompt assembly. That is a legitimate and deep engineering discipline in its own right, and it is not this lesson's subject.

An agent calling RAG as one tool among several faces a different first question, before any of those five stages even run: is retrieval the right move right now, for this step of this task? An agent might be mid-way through a multi-step plan where the current step is "call a calculator," not "look something up," and an agent architecture that always retrieves on every turn regardless of whether the current step needs it is wasting a tool call and potentially injecting irrelevant context into a prompt that didn't need it. This decision — whether to invoke retrieval at all, on this particular turn — sits in the agent's planning and reasoning layer (Domain 5's territory), one level above the five stages this lesson describes. Once the agent decides retrieval is warranted, the five stages run exactly as described above; what differs from the standalone case is that the decision to invoke them, and what to do with a thin or unsatisfying result, is itself a reasoning step the agent takes, not a foregone conclusion baked into the application's request-handling code. M6-03's agentic RAG pattern is the fullest expression of that difference: instead of one lookup per question, the agent plans sub-questions, retrieves per sub-question, and reformulates when a result looks thin — a posture available only once retrieval is understood as a callable tool an agent reasons about, not as the entire shape of the request.

The row worth memorizing as a decision rule: if the answer should change when the underlying documents change, that's a job for RAG; if the answer should change because you want a fundamentally different kind of answer — a different tone, a different output format, a different reasoning style — that's a job for fine-tuning. These two techniques are frequently presented as competitors in casual discussion, and they are not; they solve different problems, and a production agent commonly uses both — a fine-tuned model for behavior, sitting on top of a RAG pipeline for facts.

Long-context stuffing deserves a place in the table because it is a legitimate alternative at small scale, not a strawman: if your entire knowledge base is twenty pages, sending it whole in every prompt eliminates every retrieval failure mode this lesson describes, at the cost of paying to re-process those twenty pages on every single call and eventually hitting a context-window ceiling as the corpus grows.

This is a constructed scenario with invented numbers, built to make every stage boundary concrete — it is not a measurement of a real deployed system.

An internal agent answers employee questions about expense-policy rules. The knowledge base is 400 policy documents, most a page or two long.

STAGE 1 — Ingest and chunk
  Source documents:        400 policy PDFs and wiki pages
  Chunking:                ~300 tokens per chunk, small overlap
  Chunks produced:         2,150

STAGE 2 — Embed
  Embedding model:         one fixed model, applied to every chunk
  Output:                  2,150 vectors, one per chunk

STAGE 3 — Store
  Destination:             vector database, vectors + source metadata
  Metadata kept per chunk: source document id, section heading, last-updated date

STAGE 4 — Retrieve
  Query:  "Can I expense a conference ticket if my manager didn't
           pre-approve it in writing?"
  Query embedded with the SAME model as stage 2
  Semantic search over 2,150 stored vectors
  Top 4 chunks returned, ranked by similarity:
    1. "Pre-approval requirements for conference and travel expenses"
    2. "Manager approval workflow — written vs. verbal"
    3. "Expense report submission deadlines"
    4. "Reimbursement caps by expense category"

STAGE 5 — Augment and generate
  Prompt:  [system instructions] + [chunks 1-4 above] + [the question]
  Model output: "No — written pre-approval from your manager is
  required before the conference is booked, per the policy on
  pre-approval requirements. Verbal approval alone does not satisfy
  the requirement described in the manager-approval workflow policy."

Two things this trace demonstrates. First, notice that chunk 3 — submission deadlines — was retrieved into the top 4 despite being only loosely relevant; semantic search returns the closest chunks, not a guarantee that every returned chunk is actually useful, which is exactly why stage 5's prompt needs to instruct the model to use only what's actually relevant rather than treating every retrieved chunk as equally load-bearing. Second, notice that nothing about the embedding model or the language model changed during this exchange — if the pre-approval policy is updated next month, the fix is re-chunking and re-embedding one document, not retraining anything.

RAG fundamentals is objective 6.1 of Domain 6, Knowledge Integration and Data Handling, which carries a 10% exam weight [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md). The domain's own scope note is explicit that candidates are expected to know the canonical pipeline and be able to distinguish plain vector RAG — this lesson's subject — from the more elaborate patterns M6-03 covers [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md). Domain 6 is framed as the retrieval backbone behind two other domains: agent memory (Domain 5, where long-term memory is frequently implemented via exactly this kind of vector store) and grounding (Domain 9, where hallucination mitigation leans on retrieval-based evidence) [GROUND TRUTH] (Sources/ncp-aai/domain-6-knowledge-integration.md).

Expect the question shape to test two things primarily: whether you can name and correctly sequence the five stages (a scrambled-order multiple-choice item is a natural format for this), and whether you can identify the "RAG eliminates hallucination" trap when it's dressed up as a plausible-sounding answer option among four choices. A second recurring shape presents a scenario — "knowledge changes weekly and must be cited in every answer" — and asks which technique fits, testing the RAG-vs-fine-tuning distinction from §4 above rather than pipeline mechanics directly.

The canonical pipeline resolves the core grounding problem: how does an agent answer questions using knowledge it was never trained on, without retraining anything. Ingest, chunk, embed, store, retrieve, augment, generate — that five-stage loop is sufficient for a large class of real problems, particularly ones where a single, well-phrased query against a document collection reliably surfaces the right passage.

It leaves open several things this module addresses directly. It says nothing about what to do when the answer requires connecting facts across multiple documents through a relationship rather than through semantic similarity alone — M6-03 exists because plain vector retrieval genuinely cannot do multi-hop relational reasoning well. It says nothing about what happens before ingestion, when source data is duplicated, stale, or poorly formatted — M6-04 exists because that upstream work determines whether anything downstream can be trusted. And it says nothing about combining this text-centric pipeline with structured sources like databases and knowledge graphs, or with other modalities like images and tables — M6-05 closes the module by bringing all of that together.

Not generally, no — the two solve different problems and are frequently used together rather than as substitutes. RAG is the right tool when the underlying facts change and need to be reflected quickly and cited back to a source; updating a RAG knowledge base is a matter of editing and re-embedding documents, not retraining a model. Fine-tuning is the right tool when what needs to change is the model's behavior, output format, or style rather than the facts it has access to. A production agent commonly runs a fine-tuned model for behavior on top of a RAG pipeline for facts, and treating either one as a full substitute for the other is a common design mistake — RAG cannot change how a model reasons, and fine-tuning cannot cheaply keep pace with a knowledge base that changes daily.

Chunking matters because retrieval, not embedding-model capacity, is the constraint it serves. Even an embedding model capable of encoding a very long document into a single vector produces a vector that represents an average of everything in that document — if the document covers five distinct topics, the resulting vector is a blend of all five, and a query about any one specific topic will match that blended vector only weakly. Chunking into smaller, topically coherent passages means each stored vector represents one specific idea closely enough that a query about that idea can find it with high similarity, and it also means the pipeline retrieves only the relevant few hundred words rather than pulling an entire multi-page document into the prompt for every question, which would waste context budget on irrelevant surrounding material.

• RAG is five stages — ingest and chunk, embed, store, retrieve, augment and generate — split across an offline ingestion path and an online query path.
• No model weights change anywhere in the pipeline; grounding happens by inserting retrieved text into the prompt at inference time.
• RAG reduces hallucination through grounding; it does not eliminate it, and it cannot fix source data that was already wrong before ingestion.
• Retrieval quality is capped by data quality — a perfect pipeline retrieving a wrong document still produces a wrong, confidently stated answer.
• RAG and fine-tuning solve different problems: RAG injects facts with freshness and citability, fine-tuning changes behavior and style.
• For an agent specifically, RAG is one callable tool inside a reasoning loop, not a standalone application — the agent decides when to call it, not just how it works once called.
• Chunking exists because a single vector for a long, multi-topic document represents none of its specific content precisely.

Everything above assumes retrieval finds the right chunk through semantic similarity alone, which is exactly the assumption that starts to strain once the storage and search mechanics get specific. Next: M6-02 opens up the vector database this lesson treated as a black box — how approximate nearest-neighbor search actually finds those "closest" vectors, and the single correctness rule (query and document embeddings must share a model and vector space) that, if violated, makes every similarity score in stage 4 of this pipeline meaningless.