Prompt vs RAG vs Fine-Tuning: The Full Decision Rule

Each option modifies a different part of the system, and naming that part is the whole decision:

That table is the lesson. Everything below elaborates it, prices it, and turns it into a procedure.

The full customisation ladder, in cost order, is: prompt → RAG → prompt learning (prompt tuning, p-tuning) → PEFT/LoRA/adapters → full fine-tune → alignment. The first two rungs change no weights. The last four all do. That line — the weights line — is the single most useful structural fact about the ladder, because everything above it inherits the same set of drawbacks: frozen facts, no citations, no deletion path, and a training run to change anything.

Three rules govern movement on the ladder:

1. Start at the bottom. Prompting is the fastest, cheapest, most reversible option and it frequently suffices.
2. Climb only on evidence. An evaluation set that predates the change, showing the cheaper rung plateaued. 01-08 built it; 10-04 automated it.
3. Some requirements skip rungs downward, not upward. "Must cite the source" does not mean climb higher; it means go to RAG specifically, regardless of what else is true.

L1 — The intuition: knowledge, behaviour, or wording?

Ask what kind of thing is wrong.

• The model does not know something, or knows a stale version, or cannot show where an answer came from → knowledge problem → RAG.
• The model knows the right thing but says it the wrong way — wrong length, wrong voice, wrong structure, should have refused → behaviour problem → prompting first, fine-tuning if prompting plateaus.
• The model could have done better if you had asked more clearly → instruction problem → prompting.
• The model cannot do the task at any prompt → capability problem → a stronger base model. None of the three options fixes this.

That fourth case is the one people miss, and it is worth checking first because it invalidates all other work. Capability is set at pretraining (11-01) and no adaptation raises it.

L2 — The three-gate procedure

Run these gates in order and stop at the first that fires.

Gate 1 — the disqualifying clauses. Scan the requirement for these phrases. Any hit sends you to RAG, immediately, regardless of anything else:

These are not preferences to be weighed. They are architectural constraints, and a fine-tune that promises to satisfy them is a commitment you cannot honour.

Gate 2 — the capability check. Can the base model do the task at all, at the best prompt you can write? Test it zero-shot and few-shot (10-02). If not, the answer is a stronger base model, and this is both the cheapest experiment available and the one that most often makes the rest of the project unnecessary.

Gate 3 — the plateau check. Has prompting demonstrably stopped improving on a real evaluation set? "We tried some prompts" is not a plateau. A plateau is a versioned series of prompt variants (05-04) with scores that stopped moving. Only then does climbing to a weight-changing rung have an evidence base.

If Gate 1 fires → RAG. If Gate 2 fires → change the model. If Gate 3 has not fired → keep prompting. If you clear all three → fine-tune, starting with the cheapest weight-changing rung that could work.

L3 — Where the answer is "both", which is most of the time

The exam presents these as choices because multiple-choice questions require choices. Production systems are usually compositions, and the composition is not a compromise — it is the correct architecture.

The standard production shape:

   user query
      │
      ▼
 ┌─────────────────────────────────────────────┐
 │ retrieval: search the index, filter by      │  ← RAG supplies knowledge,
 │ permission, rerank, assemble context        │    freshness, citations
 └─────────────────────────────────────────────┘
      │
      ▼
 ┌─────────────────────────────────────────────┐
 │ prompt: system instruction + retrieved      │  ← prompting supplies task
 │ context + user query, in a tested template  │    framing and constraints
 └─────────────────────────────────────────────┘
      │
      ▼
 ┌─────────────────────────────────────────────┐
 │ model: base + small LoRA adapter for house  │  ← fine-tuning supplies voice,
 │ voice, format, refusal style                │    format, refusal behaviour
 └─────────────────────────────────────────────┘
      │
      ▼
 ┌─────────────────────────────────────────────┐
 │ guardrails: topical, safety, security rails │  ← a boundary outside the weights
 └─────────────────────────────────────────────┘
      │
      ▼
   grounded, cited, on-brand, bounded answer

Each layer does what only it can do. Removing retrieval makes the system stale and unciteable. Removing the adapter makes it off-brand. Removing the guardrails makes it unbounded regardless of how well-aligned the weights are (13-02).

Why the composition order matters. Retrieval before prompting, because retrieved context is part of the prompt. Prompting before fine-tuning, because a fine-tune should encode behaviour you have already validated as correct via prompting — fine-tuning an unvalidated behaviour bakes in a mistake. And guardrails outside everything, because a rail that lives outside the weights survives a weight update, which 11-03 showed you cannot say of alignment behaviour.

The one genuine trade-off between them. RAG and fine-tuning have opposite cost profiles, and this is the only place where they truly compete:

So: high request volume with stable behaviour favours folding that behaviour into weights; changing knowledge favours retrieval no matter the volume. The arithmetic in section 4 makes this concrete.

The three sentences to memorise, because they answer most scenario questions on their own:

1. RAG is right whenever facts change, citations are needed, or one user's data must be deletable.
2. Fine-tuning changes style, format, and behaviour reliably, and installs facts only brittlely.
3. Prompting is right first, because it is the only option you can revert in seconds.

The confusable that is most often tested backwards: using RAG to change writing style, and using fine-tuning to install current facts. Both are wrong in the same way — they apply a tool to the object it does not modify. RAG changes the input, so it cannot change the voice. Fine-tuning changes the function, so it cannot keep facts current.

A team has 40,000 internal documents and 2 million requests per month. They want answers grounded in those documents, in their house format. Price all three options over twelve months. Every figure is a constructed scenario for illustration, not a measurement or a vendor quote.

Shared assumptions:

Requests per month              2,000,000
Base prompt (system + query)          400 tokens
Retrieved context when using RAG    1,600 tokens
Output per response                   250 tokens
Input token price          $0.15 / 1M tokens
Output token price         $0.60 / 1M tokens
House-format instruction in prompt    350 tokens (prompt-only route)
Engineering cost                 $120 / hour
GPU cost for training              $2.50 / GPU-hour

Option A — prompt engineering only (long system prompt carrying the format rules).

Input per request  = 400 + 350            =   750 tokens
Output per request =                          250 tokens

Monthly input  = 750  × 2e6 = 1.50e9 tokens → 1,500 × $0.15 = $225
Monthly output = 250  × 2e6 = 0.50e9 tokens →   500 × $0.60 = $300
Monthly inference                                            = $525
Annual inference                                             = $6,300

One-off engineering: 40 h prompt work × $120                  = $4,800
────────────────────────────────────────────────────────────────────
Year 1 total                                                 = $11,100

Cheapest and fastest. But it answers only from the model's own knowledge — no grounding in the 40,000 documents, no citations. It does not meet the requirement.

Option B — RAG plus the same prompt.

Input per request  = 400 + 350 + 1,600    = 2,350 tokens
Output per request =                          250 tokens

Monthly input  = 2,350 × 2e6 = 4.70e9 tokens → 4,700 × $0.15 = $705
Monthly output =   250 × 2e6 = 0.50e9 tokens →   500 × $0.60 = $300
Monthly inference                                            = $1,005
Annual inference                                             = $12,060

Vector DB hosting: $400 / month × 12                          = $4,800
Ingestion + pipeline build: 160 h × $120                      = $19,200
Ongoing re-indexing ops: 8 h / month × 12 × $120              = $11,520
────────────────────────────────────────────────────────────────────
Year 1 total                                                 = $47,580

Meets the grounding and citation requirements. Note the token cost: retrieved context nearly tripled the input bill, from 225 to 705 per month.

Option C — RAG plus a LoRA adapter carrying the house format (so the 350-token format instruction is removed from every prompt).

Input per request  = 400 + 1,600          = 2,000 tokens   (350 saved)
Output per request =                          250 tokens

Monthly input  = 2,000 × 2e6 = 4.00e9 tokens → 4,000 × $0.15 = $600
Monthly output =   250 × 2e6 = 0.50e9 tokens →   500 × $0.60 = $300
Monthly inference                                            = $900
Annual inference                                             = $10,800

Vector DB hosting                                             = $4,800
Ingestion + pipeline build                                    = $19,200
Ongoing re-indexing ops                                       = $11,520

Fine-tuning, one-off:
  Data curation (1,800 pairs): 120 h × $120                  = $14,400
  LoRA training: 12 GPU-h × $2.50                            =      $30
  Eval + validation: 40 h × $120                             =   $4,800
  Subtotal                                                    = $19,230
────────────────────────────────────────────────────────────────────
Year 1 total                                                 = $65,550

Option D — for contrast only: RAG plus a full fine-tune instead of LoRA.

Everything as Option C, except training:
  Full fine-tune of a 7B model needs ~84 GB static (from 11-04)
  → multi-GPU: assume 4 GPUs × 20 h = 80 GPU-h × $2.50       =     $200
  Distributed-training engineering: 80 h × $120              =   $9,600
  Storage/deployment of a 14 GB checkpoint per revision       = marginal
  Subtotal vs LoRA: +$9,770
────────────────────────────────────────────────────────────────────
Year 1 total                                                 = $75,320

Step 1 — compare the totals.

Option A  prompt only                $11,100   ← does not meet the requirement
Option B  RAG + prompt               $47,580   ← meets it
Option C  RAG + LoRA                 $65,550   ← meets it, better format control
Option D  RAG + full fine-tune       $75,320   ← meets it, no extra benefit here

Step 2 — find where the fine-tune pays for itself. Option C's fine-tuning adds 19,230 one-off and saves 105/month in tokens (1,005 − 900).

Payback period = $19,230 / $105 per month = 183 months ≈ 15.3 years

At 2 million requests per month, folding a 350-token instruction into weights does not pay for itself on token cost. The honest justification for Option C is not cost — it is format reliability, which is a quality argument, and that is a legitimate reason. But do not claim the cost argument when the arithmetic says otherwise.

Step 3 — find the volume at which the cost argument does work. Let R be monthly requests:

Monthly saving = 350 tokens × R × $0.15 / 1e6 = R × 5.25e-5 dollars
Break even in 12 months:  R × 5.25e-5 × 12 = $19,230
                          R = 19,230 / (6.3e-4) ≈ 30,500,000 requests/month

Around 30 million requests per month before a twelve-month payback on token savings alone — roughly 15× this team's volume. This is the honest shape of the "fold your prompt into the weights to save tokens" argument: it is real, and it is a high-volume argument. Below that scale, fine-tune for reliability, not for the bill.

Step 4 — note what changes the answer completely. If the 350-token instruction were 3,000 tokens instead — a large few-shot block, say — the saving becomes $0.45 per thousand requests and the break-even volume falls by roughly 8.6×, to about 3.6 million requests per month, which is in range for many products. The lever is the size of the prompt you can remove, not the existence of a fine-tune.

Step 5 — cost the option nobody prices: doing nothing correct. Option A is cheapest and fails the requirement. Choosing it because it is cheapest is the most expensive decision on the list, because the project does not work. Cost comparison is only meaningful among options that meet the requirement — which is why Gate 1 runs before any arithmetic.

The last row is the one to be ready to defend in real life. "The model should know about X" is the most common way a knowledge requirement gets phrased as a fine-tuning request. Translate it: does the answer need to be correct, current, and traceable? Then it is retrieval.

This is a top-tier exam item and it is examinable from several blueprint directions at once. The Core Machine Learning and AI Knowledge domain — 30% of the exam, the largest — carries the customisation ladder as must-know content and asks candidates to use prompt-engineering principles and to assist in evaluating model scalability and performance. The Software Development domain asks for identification of the components required to meet user needs and for building LLM use cases such as RAG, chatbots, and summarisers. RAG versus fine-tuning is also named explicitly among the confusable pairs the exam is known to contrast.

More practically: this exam is dominated by scenario questions, and a large share of them are this decision wearing a costume. A paragraph describes a team, a constraint, and a complaint, and four options propose different rungs. If you can run the three gates in twenty seconds, you convert a category of questions from reasoning to recognition.

The RAG answering heuristic, and its counter-cases. Candidate reports converge on a useful pattern: in scenario questions, when one option proposes building a RAG solution, it is frequently the keyed answer. Use it as a tiebreaker, not a reflex, because the counter-cases are real and the exam tests them:

Question phrasings to expect:

• "A company's policies change monthly and answers must cite the policy. Which approach best meets this?" — RAG.
• "A team wants consistent tone and formatting. Which approach is most appropriate?" — prompting, then fine-tuning if it plateaus.
• "Which approach does NOT modify the model's weights?" — prompting and RAG.
• "Place these in order of increasing cost and complexity." — prompt, RAG, prompt learning, PEFT, full fine-tune, alignment.
• "What is the main disadvantage of fine-tuning for knowledge-intensive tasks?" — facts are frozen, unciteable, and undeletable.
• "What is the main disadvantage of RAG?" — added latency and per-request token cost, and dependence on retrieval quality.
• "A team must customise a model on one GPU. What should they use?" — parameter-efficient fine-tuning.
• "Which approach allows a specific customer's data to be removed on request?" — RAG.

Distractor families:

RAG if the answer must be correct, current, or traceable. Fine-tuning if the answer must be shaped a particular way. Both if both, which is the usual case.

The reason this is not a close call in either direction is that the two operate on different objects. RAG changes what the model can see; fine-tuning changes what the model does with what it sees. A perfectly fine-tuned model with no retrieval is stylish and stale. A perfect retrieval system behind an unadapted model is accurate and off-brand.

The three-clause test settles the RAG side decisively — facts change, citations needed, or data must be deletable — and each clause is architectural rather than a matter of degree. The fine-tuning side has a softer test, because prompting is a genuine competitor for most behavioural requirements: fine-tune when the behaviour must be reliable at volume, prompting has measurably plateaued, and you can afford the data-curation work, which is the dominant cost.

And be explicit about the asymmetry of failure. When RAG fails, it fails visibly and diagnosably: the wrong chunk was retrieved, or none was, and you can attribute the failure to a stage (07-10). When a fact-installing fine-tune fails, it fails invisibly: the model produces a confident near-miss with no way to trace it. Prefer the failure mode you can debug.

More often than teams expect, and the honest answer is that you cannot know until you have tried properly. "Properly" has a specific meaning here: a structured prompt with explicit format constraints and delimiters (05-02), few-shot examples where they help (05-01), constrained decoding or schema enforcement if you need valid JSON (05-05), versioned variants scored on a fixed eval set (05-04), and a documented plateau.

Prompting is enough when:

• The behaviour can be described in words and the model follows it consistently on your eval set.
• Request volume does not make the prompt's token cost material.
• Latency has room for the prompt length.
• The behaviour changes often enough that you want to edit rather than retrain.

Prompting is not enough when:

• Consistency matters more than the model reliably delivers, even with good instructions.
• The instruction is long and the volume is high, so you are paying for it millions of times — and the section-4 arithmetic clears the break-even bar.
• The behaviour is subtle enough that describing it is harder than demonstrating it. This is the genuinely strongest case for SFT: some behaviours are easy to show and hard to specify.

There is also an argument for prompting that has nothing to do with capability: reversibility. A prompt is a string in version control that you can change and redeploy in minutes. A fine-tune is a training run, an artifact, and a deployment. When a requirement is still moving — and early in a product it always is — the ability to change your mind cheaply is worth more than the marginal consistency.

The ladder is prompt → RAG → prompt learning (prompt tuning, p-tuning) → PEFT/LoRA/adapters → full fine-tune → alignment, and it is ordered by increasing cost, increasing commitment, and decreasing reversibility.

The ordering is not arbitrary; each of the three axes moves the same way, which is why one ordering serves all of them. And the weights line between RAG and prompt learning is the sharpest boundary on the ladder: cross it and you inherit frozen facts, no provenance, no deletion, and a training run for every change.

Two important qualifications:

The ladder is not a maturity model. Being on the top rung is not an achievement. The correct rung is the lowest one that satisfies the requirement, and a team running RLHF when a prompt would do has made a mistake, not progress.

Some requirements move you sideways, not up. Citations, freshness, and deletability all send you to RAG specifically. No amount of climbing gets you there, because the higher rungs do not have those properties at all.

It can, and the mechanism is specific: if a behaviour currently requires a long instruction or a block of few-shot examples in every prompt, encoding that behaviour in the weights lets you delete those tokens from every request. Since input tokens are billed and processed on every call, removing them reduces both cost and prefill latency.

Whether it is worth it is arithmetic, not intuition, and the section-4 example is the template:

monthly saving   = tokens_removed × requests_per_month × input_price
payback_months   = one_off_finetune_cost / monthly_saving

In the constructed example, removing 350 tokens at 2 million requests per month against a $19,230 fine-tuning cost gave a payback of about 15 years — so the cost argument failed and the reliability argument had to carry the decision. Scale either the removed-token count or the request volume up by an order of magnitude and the same arithmetic flips. The break-even volume there was roughly 30 million requests per month for a 350-token saving, and roughly 3.6 million for a 3,000-token saving.

The trap to avoid is asserting the cost benefit without computing it, because it is the most quoted justification for fine-tuning and the least often checked. And note two costs that push the other way: a fine-tuned model is another artifact to version, evaluate, and re-validate whenever the base model changes, and if you keep the adapter unmerged you pay a small amount of extra computation per request (11-05). The 12-09 cost framework is where the full per-token accounting lives.

• The three options modify different objects: prompting modifies the instruction, RAG modifies the input, fine-tuning modifies the function. Identify the object and the answer follows.
• RAG is right whenever facts change, citations are needed, or one user's data must be deletable. Any one clause is sufficient and none is negotiable.
• Fine-tuning changes style, format, tone, and refusal behaviour reliably, and installs facts only brittlely. That single sentence answers most exam confusables in this family.
• Prompting is right first, always, because it is the fastest, cheapest, and only instantly reversible option — and because a fine-tune should encode a behaviour prompting has already validated.
• Run three gates in order: disqualifying clauses → capability check → plateau check. Stop at the first that fires.
• Climb the ladder one rung at a time, and only on evidence from an evaluation set that predates the change. The top rung is not an achievement.
• Some requirements move you sideways to RAG rather than up the ladder, because the higher rungs do not have provenance or deletability at all.
• Cost the per-request side, not just the up-front side. In the constructed example RAG nearly tripled the input token bill, and folding a 350-token prompt into weights had a 15-year payback at 2 million requests per month.
• The token-saving argument for fine-tuning is a high-volume argument. Compute the break-even before asserting it.
• Composition is the normal answer: retrieval for knowledge, prompt for framing, adapter for voice, guardrails for boundaries. The exam forces a single choice; production rarely does.
• Prefer the failure mode you can debug. RAG fails visibly and attributably; a fact-installing fine-tune fails invisibly.

The rule above assumes you get to choose on the merits. Real decisions arrive pre-loaded with constraints that were fixed before you were consulted: one GPU, six weeks, no annotation budget, a compliance requirement, a base model somebody already licensed, a latency SLA in a contract. Those constraints eliminate options before the technical argument starts, and knowing which they eliminate is a distinct skill from knowing which option is best. Next: 11-09 runs the decision under real limits — hardware, data, time, budget, governance, and latency — and gives you a defensible one-page recommendation you can put in front of a senior reviewer.