Choosing Prompting vs. RAG vs. Fine-Tuning Under a Stated Constraint

The choice between prompting, RAG, and fine-tuning is not a preference among three roughly equivalent tools. Each one repairs a different kind of gap, and the professional-level skill is reading a described failure correctly enough to know which gap you are actually looking at.

[GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md) states the rule in exactly the terms this domain expects you to reproduce: "Situation: Task steering, formatting, quick iteration — Prefer: Prompt engineering. Situation: Knowledge changes often / must be cited / is large — Prefer: RAG (retrieve at inference). Situation: New skill/behavior/style baked in permanently — Prefer: Fine-tuning / PEFT."

[GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md) is explicit about the constraint that governs movement between the second and third rungs: "Exhaust prompting and RAG before fine-tuning when data/compute is limited or knowledge is dynamic." That sentence carries two separate, independently sufficient reasons to hold off on fine-tuning, and section 3 below takes each one apart on its own terms.

L1 — Intuition: most "the model can't do this" complaints are actually "I never asked correctly" complaints

Before asking whether a gap is about knowledge or behavior, ask whether it is about wording at all. A request that produces inconsistent length, wrong tone, or a format the model keeps drifting from is frequently not a capability problem — it is an instruction problem, and M2-02's prompt-template treatment (role, context, constraints, format, delimiters) exists specifically to close that gap without spending a single token on RAG's retrieval hop or a single hour on a training run.

L2 — Mechanism: a diagnostic sequence that uses each of this module's lessons in order

Run these checks in sequence, and stop at the first one that resolves the failure. Each step is cheaper and faster to try than the one after it, so working in this order never wastes more than the previous step's small cost before finding the actual fix.

Step 1 — tighten the instruction and add a structured template. Name the audience, the required length, the forbidden content, and the exact output shape (M2-02's five slots: role, context, constraints, format, delimiters). A large share of apparent capability failures are actually unspecified-requirement failures, and this step costs nothing beyond editing a string.

Step 2 — add in-context exemplars targeted at the task's real decision boundaries. If the failure is a specialized label set or a house convention prose cannot state cheaply, M2-01's few-shot treatment is the next-cheapest lever — a handful of worked examples, chosen at the boundaries you actually got wrong, not the easy middle.

Step 3 — add reasoning structure if, and only if, the task has genuine multi-step dependencies. M2-02's chain-of-thought treatment applies here, and only here: a task with no real decomposition gains nothing from this step and simply pays its cost, so this step should be skipped entirely for single-step lookups and classifications.

Step 4 — constrain the output if the remaining failures are structural. M2-04's constrained decoding and validation wrappers close a specific and common residual gap: a response that is correct in substance but wrong in shape, or that needs a guaranteed-parseable structure a prompt alone cannot promise.

Step 5 — ask the knowledge question. Does the model produce confident, wrong statements about facts specific to your situation — facts that change, facts that must be cited, or facts simply too voluminous to fit in any prompt? If yes, no amount of steps 1 through 4 fixes it, because none of them add anything the model was never shown. This is where RAG enters.

Step 6 — consider fine-tuning, and only after every step above has been tried and measured, the residual failure is genuinely behavioral rather than knowledge-based, and the two constraints in section 3 have been checked and cleared.

L3 — The exam-relevant edge case: "the model doesn't know X" is ambiguous, and resolving the ambiguity is the actual skill

A stakeholder complaint phrased as "the model doesn't know our product line" can describe either a knowledge gap or a behavior gap, and mistaking one for the other sends you to the wrong rung. If the model has never seen your product names and specifications at all, that is missing knowledge, and RAG is the fix — retrieving the actual specification sheet at query time. If the model has been shown your product names inside a long prompt already, and the complaint is that it does not consistently use the right tone or format when discussing them, that is a behavior gap wearing a knowledge complaint's clothing, and the fix is further down M2-02's template work, or fine-tuning if the volume justifies it — not more retrieval, because retrieval was never the missing piece.

[GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md) names two conditions under which prompting and RAG should be exhausted before fine-tuning: limited data, and limited compute. These are independently sufficient reasons, and each has a distinct mechanism worth understanding rather than memorizing as a slogan.

The data constraint. A supervised fine-tune needs a labeled dataset large enough for the model to generalize a pattern from, rather than memorize a handful of specific examples in a way that does not transfer to new inputs. When that dataset does not exist — a specialized domain with only a few dozen historical examples, say — a fine-tune attempted anyway tends to overfit: the model becomes very good at reproducing its training examples and poor at everything else, which is a worse outcome than a well-constructed prompt that at least generalizes across the model's existing pretraining. In-context learning (M2-01) is specifically well-suited to exactly this data-scarce situation, because it needs only as many worked examples as fit in one prompt, not enough to train reliably from.

The compute constraint. A fine-tune, even a parameter-efficient one, requires a training run: GPU time, an evaluation pass to confirm it worked, and a deployment step to serve the resulting weights or adapter. Prompting and RAG both require none of that — a prompt change is a string edit deployed in minutes, and a RAG pipeline's ongoing cost is retrieval latency and index maintenance, not a training job. Under a genuinely stated compute ceiling — a single available GPU, a hard timeline, no training infrastructure — the rungs that require no training run are not just cheaper, they may be the only ones actually available to try at all.

Both constraints point the same direction for the same underlying reason: fine-tuning is the one rung on this ladder that is expensive to attempt and expensive to undo. A prompt can be reverted in seconds. A RAG index can be swapped or disabled without touching the model. A fine-tuned checkpoint represents a completed training run, and reversing course means either serving the older checkpoint or running another training job — never a one-line edit. That asymmetry, not a general suspicion of fine-tuning as a technique, is the actual reason to exhaust the cheaper rungs first when either constraint is genuinely in force.

The row worth carrying into a scenario question is reversibility. A stated constraint of limited data or limited compute is not really an argument that fine-tuning "doesn't work" under those conditions — it is an argument that the one option requiring the most upfront investment and offering the least room to change your mind is the wrong one to reach for first when you are least equipped to recover from a bad bet.

Constructed scenario, illustrative only. A mid-sized logistics company runs an internal assistant for warehouse staff. Three separate complaints arrive in the same week, and the team has a stated constraint: no dedicated ML engineer, and a training budget that would fund at most one small experiment this quarter.

Complaint A — "It gives long, rambling answers when people just want a one-line status." Diagnosis: instruction gap. The assistant was never told the required response length or format, and nothing about this complaint suggests missing knowledge or unreliable behavior at scale — it suggests an unspecified constraint. Fix: apply M2-02's template pattern, adding an explicit length and format constraint. Cost: an afternoon of prompt editing. Given the stated compute constraint, this is not just the cheapest option, it is the only one the team can attempt without spending any of its one-experiment budget.

Complaint B — "It confidently states the wrong current inventory count for SKU-4471, which changed yesterday." Diagnosis: knowledge gap, and specifically a dynamic one — the correct figure changes daily. No amount of instruction tightening or example curation installs a fact that changes every day; the model would need to be retrained daily to keep pace via fine-tuning, which is absurd on its face. Fix: RAG against the live inventory system, so the correct count is retrieved at query time rather than memorized anywhere. This also happens to satisfy the compute constraint better than fine-tuning would, since indexing a structured inventory feed is a smaller build than a training pipeline.

Complaint C — "Every shipping-exception report needs to follow our incident-report template exactly, and staff keep having to manually reformat what the assistant produces, at a volume of around 400 reports a week." Diagnosis: this is the case worth slowing down on. The content is largely right; the persistent problem is form, at real and repeated volume. Before reaching for fine-tuning, the team should first exhaust M2-02's template pattern and M2-04's constrained decoding — a schema-constrained output could plausibly solve this without a training run at all, and given the stated one-experiment-this-quarter compute constraint, that cheaper attempt should be made and measured first. If a carefully constrained, templated prompt still leaves staff reformatting output at that volume after a real attempt, that measured, persistent gap — not the original complaint — is what would justify spending the quarter's one experiment on a small fine-tune, and only then because both the behavioral diagnosis and the compute budget check out.

The pattern across all three: the stated constraint did not change the diagnosis. It changed the bar for how thoroughly the cheaper rungs had to be exhausted before fine-tuning could be justified at all.

Constructed scenario, illustrative only. Return to Complaint C. Suppose the templated, constrained-decoding attempt genuinely plateaus, and the team wants to check whether the data constraint from section 3 is actually cleared before spending the quarter's one experiment on a fine-tune.

Step 1 — count what is actually available. The team has eleven months of past shipping-exception reports in the required format, written by hand before the assistant existed.

Historical hand-written reports available:  11 months x ~35 reports/month = 385 reports

Step 2 — filter for quality, since not every historical report is a clean example of the target format; some predate the current template revision.

Reports matching the current template exactly:   385 x ~60% = 231 reports

Step 3 — compare against the volume a reliable supervised fine-tune typically needs. A parameter-efficient fine-tune for a narrow, well-specified behavioral pattern like a fixed report format commonly needs several hundred to a few thousand high-quality labeled pairs to generalize reliably rather than overfit to its training examples. ⚠️ UNVERIFIED: this domain's source material does not state a specific minimum example count for a PEFT run, and this lesson will not assert one as ground truth — what follows is the professional judgment call the exam expects you to be able to reason through, not a memorized threshold.

Available, quality-filtered examples:        231
Low end of a typically workable PEFT range:  ~300-500

Step 4 — read the result honestly. 231 clean examples sits below even the low end of a typically workable range for a reliable fine-tune, which is exactly the data-constraint signal section 3 described: attempting the fine-tune anyway risks a model that reproduces its 231 training reports well and generalizes poorly to the shipping-exception scenarios those 231 reports did not happen to cover. The correct move under this specific number is not "fine-tune anyway because the plateau was real" — it is to spend more effort on the prompting-side attempt (a stricter schema, a second exemplar set drawn from the highest-quality subset of the 231 reports as few-shot demonstrations rather than training data) while the additional months of reports needed to clear a workable data volume accumulate. The plateau justified investigating fine-tuning; the data count is what actually decides whether to execute it this quarter.

Step 5 — the counterfactual that would flip the decision. If the same company had three years of reports instead of eleven months, quality-filtered to roughly 750, the data constraint would clear the same threshold this example failed, and — with the behavioral diagnosis and compute budget already confirmed in Complaint C's narrative — fine-tuning would become the defensible recommendation. The diagnosis does not change with more data; only the feasibility check does, which is precisely the distinction section 3 draws between the behavior-versus-knowledge diagnosis and the data-versus-compute feasibility gate.

Objective 2.5 is this decision rule directly, and it sits inside Prompt Engineering's 13% share of the blueprint, tied for third-largest. [GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md): the domain's scope note states that professional-tier questions should "ask you to choose a prompting strategy under constraints (small dataset, specialized domain, strict output format) and to distinguish prompting from fine-tuning."

The word "constraints" in that scope note is doing real work — an associate-level version of this question might just ask you to name the three options and what each changes; the professional-level version states a specific data or compute limitation and expects you to route the decision through it, not around it.

Expect the question in these shapes:

1. A constraint-first scenario. A description states a limited dataset or a limited compute budget explicitly, alongside a complaint that sounds like it could be fixed by fine-tuning. The keyed answer routes to prompting or RAG first, specifically because the stated constraint rules out a reliable fine-tune, not because fine-tuning is wrong in the abstract.
2. A knowledge-versus-behavior discrimination, phrased as a stakeholder complaint rather than a technical description. "The model doesn't know our latest policy" versus "the model knows the policy but won't format the answer consistently" are different gaps requiring different fixes, and the exam tests whether you can extract the right diagnosis from natural, ambiguous phrasing.
3. A sequencing question. A scenario asks what should be tried before fine-tuning is considered, and the keyed answer names the specific cheaper techniques — exemplars, templates, reasoning structure, output constraints — this module built in M2-01 through M2-04, rather than a vague "try prompting."

What the distractors typically look like

The standing trap in this domain's house style is a real technique, applied past the constraint that would rule it out. The most common distractor patterns are: recommending fine-tuning to install facts that change frequently, when no retraining cadence can track daily or weekly change; recommending fine-tuning under an explicitly stated data or compute shortage, ignoring the constraint stated directly in the scenario; and treating RAG as a style or behavior fix, when retrieval changes what the model can see, not how it behaves once it sees it.

Ask what would fix it. If showing the model the correct facts — in a retrieved passage, in a longer prompt, in any form — would resolve the failure, it is a knowledge gap, and RAG is the durable fix because it can keep supplying those facts as they change. If the model already has access to the correct facts, whether through a long prompt or retrieved context, and the failure persists as inconsistent formatting, tone, or task execution regardless of how the facts are presented, it is a behavior gap, and the fix is further prompt engineering from this module or, once the two constraints in section 3 are cleared, fine-tuning. A useful diagnostic test: paste the correct fact directly into the prompt by hand and see whether the failure disappears. If it does, the gap was knowledge. If the same failure persists even with the fact stated explicitly in front of the model, the gap is behavioral.

When three conditions hold together, not any one alone. First, the diagnosis is genuinely behavioral — content is broadly correct and the persistent problem is form, consistency, or a specific skill, not missing or changing facts. Second, prompting and RAG have actually been tried and measured, not just assumed to be insufficient, and a real plateau shows up in that measurement. Third, the stated data and compute constraints are both cleared: there is a real dataset in the hundreds-to-thousands range of high-quality examples, and there is compute and time available for a training run, an evaluation pass, and a deployment step. Section 5's Complaint C shows exactly this sequence — the recommendation to fine-tune only becomes defensible after the cheaper rungs have been tried and have measurably failed to close the gap, and after confirming the quarter's single-experiment compute budget can actually support it.

⭐ THE EARNED INSIGHT Every technique this module built — exemplars, reasoning structure, templates, decoding constraints — is not a warm-up act for fine-tuning; it is frequently the entire fix, and reaching past it is usually a failure of diagnosis rather than a failure of those techniques. The professional-level judgment this lesson asks for is not "know that fine-tuning exists as an option," it is the discipline to treat fine-tuning as the most expensive, least reversible rung on the ladder, and to climb to it only when the gap has been correctly diagnosed as behavioral, the cheaper rungs have been genuinely exhausted and measured, and the constraint you were actually handed — limited data, limited compute, or both — has been checked rather than assumed away.

• Diagnose the gap before picking a rung: an instruction gap points at this module's prompting toolkit, a knowledge gap points at RAG, and a behavior gap points at fine-tuning only once it is confirmed genuinely persistent.
• Exhaust prompting and RAG before fine-tuning when data or compute is limited — these are two independently sufficient reasons, not a single vague caution.
• Fine-tuning is the one rung that is expensive to attempt and expensive to undo. That asymmetry, not a blanket preference for cheaper tools, is why it comes last.
• A stated data constraint argues for in-context learning specifically, because a reliable fine-tune needs far more labeled examples than a few-shot prompt does.
• A stated compute constraint argues for prompting and RAG specifically, because neither requires a training run, an evaluation pass, or a deployment of new weights.
• "The model doesn't know X" is genuinely ambiguous between a knowledge gap and a behavior gap wearing a knowledge complaint's phrasing — resolving that ambiguity correctly is the actual professional skill this objective tests.
• Run this module's own techniques in order first: template and instruction tightening, targeted exemplars, reasoning structure where genuinely needed, output constraints — and only then ask the knowledge question, and only after that consider fine-tuning.

Every decision this module has covered assumed the model's weights stay exactly as they are. The moment a diagnosis genuinely calls for fine-tuning — durable behavior at scale, sufficient labeled data, sufficient compute — an entirely different set of concerns opens up, starting well before any training run: what the data that behavior is learned from needs to look like, how it must be cleaned and organized, and which early decisions about it cannot be undone once training begins. That is the subject the next module in this course picks up, moving from what a prompt can buy without touching a weight to what a dataset must look like before you are ready to touch one at all.