SFT vs. RLHF vs. DPO vs. GRPO: Which Alignment Method Needs a Reward Model

Supervised fine-tuning (SFT) is the baseline alignment step: take a pretrained model and fine-tune it on curated instruction-response pairs, so the model learns to follow instructions in the specific form the demonstrations model. Its supervision is demonstrative — a human writes down an example of the right answer, and the model learns to imitate it. SFT is necessary but structurally limited: a human can write down what a good response looks like for cases they can anticipate, but nobody can write down an exemplar for every subtle judgment call a model will face, and imitation caps out at the quality and coverage of the exemplars themselves.

Alignment, as this lesson uses the term, is the stage that layers a comparative signal on top of an SFT policy: instead of "here is the right answer," the supervision is "this answer is better than that one." [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) frames the whole family this way, walking through "SFT, RLHF, DPO, GRPO" as the domain's alignment subsection, and stating plainly that RLHF trains a separate reward model and a PPO critic, DPO drops the reward model, and GRPO drops the critic while keeping a reward signal. Three methods, three different answers to "what extra machinery turns preference comparisons into a weight update," and mixing up which method needs which piece is, in the source material's own words, "the domain's signature trap."

Two framings are worth holding before the mechanism gets detailed:

• The supervision changes shape as you move down this list. SFT: write the answer. RLHF and DPO: compare two answers. GRPO: rank several answers relative to each other. Each shift is a response to a real cost or stability problem with the method before it.
• Every one of these methods assumes an SFT policy already exists. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the classic pipeline directly: "SFT → train a reward model on human preference rankings → RL optimization (PPO) with a KL penalty." Reward-model training needs candidate responses worth comparing, and a base model's raw continuations are not worth ranking — they are frequently incoherent relative to the instruction. SFT is what makes the candidates worth comparing in the first place.

L1 — Intuition: three ways to teach without writing the answer

Imagine three ways to coach a writer without ever handing them a finished essay to copy. The RLHF way: train a separate judge who has learned your taste from thousands of side-by-side comparisons, then have the writer draft freely and adjust toward whatever the judge scores highly — while a second helper, the critic, estimates in advance how much better or worse each draft-in-progress is likely to score, so the writer's adjustments are less noisy. Two extra people are now part of the process: the judge (the reward model) and the estimator (the critic).

The DPO way skips both helpers. You show the writer pairs of their own past drafts, one you preferred and one you did not, and adjust their writing habits directly toward the preferred one — no separate judge is ever trained, and no estimator is needed, because the comparison itself is the whole lesson.

The GRPO way keeps a judge — something still scores each draft — but drops the estimator. Instead of a helper predicting in advance how good a draft is likely to be, the writer produces several drafts of the same piece at once, the judge scores all of them, and the writer's habits shift toward whichever drafts scored better relative to the others in that same batch. The comparison group itself replaces the need for a separate prediction of expected quality.

L2 — Mechanism, stage by stage

RLHF (PPO) — reward model plus critic. The classic pipeline, per [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md), runs in a strict order: "SFT → train a reward model on human preference rankings → RL optimization (PPO) with a KL penalty." The reward model is a separate network, typically initialized from the same model family, trained on human-labeled preference pairs (prompt, response_chosen, response_rejected) with a pairwise ranking loss, so it learns to output a scalar score that agrees with human rankings. That reward model's score is then the signal that PPO (proximal policy optimization) — the reinforcement-learning algorithm running stage three — optimizes the policy against, subject to a KL-divergence penalty that keeps the policy from drifting too far from the SFT starting point. PPO's standard implementation additionally trains a critic (value) network, a second auxiliary model whose job is to estimate the expected future reward from a given partial output, so the training signal driving each gradient step (the "advantage" — how much better this response scored than the critic expected) is lower-variance than the raw reward alone would be. That is four moving pieces potentially resident at once: the policy being trained, the frozen SFT reference the KL penalty measures against, the reward model, and the critic.

DPO (Direct Preference Optimization) — no reward model, no RL loop. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) is direct about the mechanism: DPO "fine-tunes directly on preference data; no separate reward model and no RL sampling loop." Rather than learning an explicit reward function and then running an RL algorithm to maximize it, DPO uses a loss function derived so that minimizing it directly moves the policy toward the chosen responses in the same preference-pair data RLHF would have used to train a reward model — the source names this precisely as "the language model itself implicitly plays the reward model" [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md), a description worth reading literally: there is no separate scalar-output network anywhere in the pipeline, only the policy and a frozen reference copy of it. The training loop looks like ordinary supervised fine-tuning — a fixed dataset, a forward and backward pass, no policy generating fresh samples mid-training — which is what makes DPO markedly simpler to implement and to run than PPO-based RLHF.

GRPO (Group Relative Policy Optimization) — reward signal, no critic. GRPO is a PPO variant, introduced alongside DeepSeekMath, and [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) is explicit about what it removes and what it keeps: GRPO "drops PPO's separate value/critic network and estimates advantages from group-relative (normalized) scores of multiple sampled outputs per prompt, lowering memory cost while improving reasoning." Concretely, for a given prompt, the policy samples several outputs — a group — rather than one. Each output in the group is scored by a reward signal, and instead of a critic network estimating in advance what a "typical" score should be, the advantage used for the policy-gradient update is computed by normalizing each output's score against the mean and spread of scores within its own group. The critic's job — providing a baseline to compare each score against, so the gradient signal is not dominated by noise — is done instead by the group itself. This is the exact meaning of "critic-free": GRPO still needs a reward signal (a scalar quality judgment for each sampled output), it just no longer needs a second trained network to estimate the baseline that signal is compared against.

L3 — The exam-relevant edge case: reward model and critic are two separable pieces, not one bundle

The trap this domain names directly is treating "does it need a reward model" and "does it need a critic" as the same question, when they are independently variable. The table below is the whole answer, and it is worth internalizing as a 2×2 rather than a linear list:

RLHF occupies the "both" cell: a reward model scores outputs, and a critic estimates a baseline to compare those scores against. GRPO occupies the "reward model, no critic" cell — it still needs something producing a reward signal, it simply gets its baseline from the sampled group instead of from a trained critic. DPO occupies the "neither" cell — the preference data itself, via the DPO loss, plays the role a reward model would have played, and there is no RL loop generating fresh samples for a critic to score in the first place. The single most tested confusable in this table, stated as the domain's own trap, is offering "DPO trains a reward model, then runs PPO" as a description of DPO — that sentence correctly describes RLHF, and an item testing this trap is checking whether you know DPO's entire point is to skip exactly that.

All figures are a constructed scenario built to illustrate the mechanism, not a benchmark of any specific training run. Take a 7-billion-parameter model, and count what must be resident in GPU memory during each method's training step, at a coarse "how many model-sized objects" level rather than a byte-exact estimate.

Step 1 — SFT.

1. the policy being trained (weights + gradients + optimizer state)
total distinct model-sized objects: 1

Step 2 — RLHF with PPO.

1. the policy being trained          (weights + gradients + optimizer state)
2. the frozen SFT reference policy   (weights only, for the KL penalty)
3. the reward model                  (weights only, for scoring)
4. the critic/value network          (weights + gradients + optimizer state)
total distinct model-sized objects: 4

Step 3 — DPO.

1. the policy being trained          (weights + gradients + optimizer state)
2. the frozen reference policy       (weights only, for the DPO loss's reference term)
total distinct model-sized objects: 2

Step 4 — GRPO.

1. the policy being trained          (weights + gradients + optimizer state)
2. the reward signal source          (a reward model, or a rule-based scorer —
   the source material does not commit to one implementation, so treat the
   exact form of GRPO's reward signal as ⚠️ UNVERIFIED beyond "a reward
   signal exists and is used")
   -- no critic/value network occupies a third slot
total distinct model-sized objects: 2 (plus whatever the reward source costs,
which is smaller than a fourth full trainable model in the PPO case)

Step 5 — read the count as the argument, not just the arithmetic. RLHF's four-object count is the direct, countable consequence of needing both a reward model and a critic; DPO's two-object count is the direct consequence of needing neither; GRPO's count sits between the two specifically because it kept the reward signal but removed the critic. This is a constructed illustration of [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md)'s stated properties, not a new fact: the object count is a mechanical readout of exactly the property the source names — reward model and critic are separable requirements, and each method's cost profile is a direct function of which of the two it actually needs.

⭐ THE EARNED INSIGHT

The exam's likely framing treats "does it need a reward model" as the whole question, which invites a two-way sort: methods with a reward model versus methods without one. The professional-depth trap underneath that framing is that a reward model and a critic are two independently variable pieces, not a bundle that comes and goes together. GRPO is the method that proves the point: it keeps the reward signal RLHF has and still drops a full auxiliary network, because the piece it removes — the critic's baseline estimate — is not the same piece as the reward model at all. Sort by "reward model, yes or no" and GRPO looks like RLHF's twin. Sort by "critic, yes or no" and GRPO looks like DPO's cousin instead. The only sort that is actually correct is both axes at once.

All figures below are a constructed scenario, not a measurement. A team is tempted to save time by training the reward model directly on outputs from the raw pretrained base model, skipping SFT, on the theory that "we need preference data regardless, so let's collect it as early as possible."

Step 1 — what the reward model actually learns to rank. A reward model's pairwise ranking loss trains it to separate two responses to the same prompt along a quality axis humans can perceive and agree on. For that ranking to be learnable at all, the two responses being compared need to differ in some way a human annotator can consistently judge — tone, completeness, correctness, adherence to the instruction.

Base-model continuations for an unseen instruction-style prompt:
  candidate A: a plausible-sounding continuation of the prompt's *text*,
               not an attempt to follow it as an instruction
  candidate B: a different plausible-sounding continuation, also not
               instruction-following

Annotator's task: "which of these two better follows the instruction?"
Annotator's actual experience: neither candidate is really attempting to
follow an instruction at all, because the base model was never trained to
recognize "this text is an instruction I should act on"

Step 2 — the failure mode this produces. Annotators forced to rank two non-comparable failures either produce near-random labels (both are bad in different, incomparable ways) or latch onto a superficial signal — length, confident phrasing — that has nothing to do with actual instruction-following quality. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) frames the fix precisely: the strict order is "SFT → train a reward model on human preference rankings → RL optimization (PPO)," and SFT's role in that order is to produce a policy whose outputs are coherent enough, and instruction-shaped enough, that ranking them measures something real.

With SFT run first:
  candidate A: a genuine attempt to follow the instruction, executed well
  candidate B: a genuine attempt to follow the instruction, executed poorly
  annotator's task is now answerable, because both candidates are
  comparable on the axis being asked about.

Step 3 — the number that makes the shortcut visible as a shortcut, not a saving. This constructed scenario illustrates why [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the pipeline order as strict rather than as a preference: skipping SFT does not remove any labeling cost, it relabels the same annotation budget onto a signal too noisy to train a useful reward model from, so the team still pays for preference labeling and additionally gets a reward model that barely beats chance, with no cheap recovery except going back and running SFT anyway — the apparent time saved at the start is spent again, with interest, diagnosing why stage three is not improving anything.

Fine-Tuning is tied for third-largest domain on the NCP-GENL blueprint at 13%, and the module's own weight note names this comparison directly as one of the domain's three most-tested differentiators, alongside PEFT's latency property: "DPO's missing reward model, and GRPO's missing critic." [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the professional-level framing for the whole domain — questions probe "the distinguishing properties of each method," not surface familiarity with the names — which for this lesson's subject means the exam rewards knowing precisely which auxiliary network each method needs, rather than a general sense that "RLHF and DPO are both about preferences."

Expect this material in a few recurring shapes: a direct identification question ("which alignment method needs no separate reward model?"), a reversed-fact question that states DPO's or GRPO's mechanism backwards and asks you to catch it, and a scenario question describing a resident-model count or a memory constraint and asking which method is consistent with it. The distractors in this domain's house style are, true to form, a real technique described with one property swapped for another method's property — "DPO trains a reward model then runs PPO" (that is RLHF), or "GRPO removes the reward signal" (it removes the critic, not the reward signal) are the two standing traps this lesson has been built specifically to immunize against.

How the question tends to be phrased

Professional-level items favor a short factual stem paired with four named methods as options: "Which alignment method optimizes directly on preference pairs with no separate reward model?" with DPO as the keyed answer against RLHF, GRPO, and SFT as distractors, each a real technique that fails the stated property for a specific, nameable reason. A second common shape names a resident-object count or a described training loop and asks which method it describes — testing the worked-example arithmetic from section 4 rather than the name recognition alone.

What the distractors typically look like

The standing traps are: describing RLHF's pipeline but naming DPO (a real pipeline, wrong label); claiming GRPO removes the reward signal instead of the critic (conflating the two auxiliary pieces this lesson keeps separate); and reversing the RLHF stage order by putting reward-model training before SFT (a real stage, wrong position — SFT must produce the candidates the reward model learns to rank).

Work through each item before checking the answer key. Every option names a real technique — the task is matching the described property to the right method, not spotting an invented one.

1. Which alignment method trains a separate reward model AND a separate critic network?
   • A. SFT
   • B. RLHF with PPO
   • C. DPO
   • D. GRPO
2. Which method optimizes directly on preference pairs with no separate reward model and no RL sampling loop?
   • A. RLHF
   • B. GRPO
   • C. DPO
   • D. SFT
3. GRPO is described as "critic-free." What does it use instead of a critic network to estimate advantage?
   • A. A second reward model
   • B. Group-relative normalized scores across several sampled outputs per prompt
   • C. A fixed constant baseline
   • D. The KL penalty alone
4. A team trains a reward model directly on a raw pretrained base model's outputs, skipping SFT. What is the most likely consequence?
   • A. No consequence — reward models work equally well on any model's outputs
   • B. The reward model trains faster because there is less data to process
   • C. Annotators struggle to rank incomparable, non-instruction-following outputs, producing a noisy signal
   • D. The RL loop becomes unnecessary
5. Which statement correctly describes DPO?
   • A. DPO trains a reward model, then runs PPO against it
   • B. DPO requires online generation from the current policy at every training step
   • C. DPO optimizes the policy directly on preference pairs with a supervised-style loss
   • D. DPO removes the need for any preference data at all
6. What is the correct order of RLHF's three stages?
   • A. Reward model, then SFT, then policy optimization
   • B. SFT, then reward model training, then policy optimization
   • C. Policy optimization, then SFT, then reward model training
   • D. SFT and reward model training happen simultaneously, then policy optimization
7. A described training run keeps a frozen reference policy and one trainable policy, with no reward model and no critic anywhere in the pipeline. Which method does this describe?
   • A. RLHF
   • B. GRPO
   • C. DPO
   • D. SFT
8. Which of the following is true of GRPO relative to standard PPO-based RLHF?
   • A. It adds a second critic network for stability
   • B. It removes the reward signal entirely
   • C. It drops the critic/value network and estimates advantage from group-relative scores
   • D. It requires full-parameter fine-tuning rather than any parameter-efficient method

Answers

1. B. RLHF with PPO is the only one of the four that trains both a separate reward model and a separate critic/value network.
2. C. DPO's defining property is skipping both the reward model and the RL sampling loop, optimizing directly on the preference data.
3. B. GRPO replaces the critic's baseline-estimation role with normalized scores computed across a group of sampled outputs for the same prompt.
4. C. A base model's outputs are not instruction-shaped, so annotators cannot reliably rank them on the intended axis, producing a reward model that barely beats chance.
5. C. This is DPO's defining mechanism; option A describes RLHF, and DPO uses no online generation and no reward model.
6. B. SFT produces the policy whose outputs are worth ranking; the reward model is trained on those rankings; policy optimization uses the reward model last.
7. C. Exactly two resident objects — a trainable policy and a frozen reference — with no reward model and no critic, is DPO's signature resident-model footprint.
8. C. GRPO's entire distinguishing property relative to PPO is dropping the critic and substituting group-relative advantage estimation; it keeps the reward signal.

DPO is the method built for exactly this situation. It consumes the same pairwise preference data RLHF would use to train a reward model, but its loss function is derived so that the policy can be optimized directly on that data with an ordinary supervised-style training loop — no reward model to train separately, and no reinforcement-learning sampling loop generating fresh completions mid-training. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) frames this as DPO's whole appeal: it reaches a similar destination to RLHF "without a separate reward model" and "no RL sampling loop," which is precisely the engineering capacity most teams lack when they lack RL infrastructure specifically.

No, and this is the single fact worth being unable to get wrong. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states it directly: GRPO is "critic-free," and it "drops PPO's separate value/critic network," replacing the critic's job — estimating a baseline to compare each output's score against — with group-relative normalization across several outputs sampled for the same prompt. Per [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md), GRPO still uses a reward signal; what it specifically removes is the second trained network that PPO's implementation otherwise needs to estimate an expected-score baseline.

• SFT is demonstrative; RLHF, DPO, and GRPO are comparative or group-relative. Every preference-based method here assumes an SFT policy already exists, because its outputs are what get compared or scored.
• RLHF needs both a separate reward model and a separate critic — the pipeline [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states as SFT, then reward model, then PPO with a KL penalty.
• DPO needs neither. It optimizes directly on preference pairs with a supervised-style loss, and the source names this precisely: "the language model itself implicitly plays the reward model."
• GRPO needs a reward signal but not a critic. It replaces the critic's baseline-estimation job with group-relative normalization across multiple sampled outputs per prompt.
• Reward model and critic are two separable requirements, not a package deal — the 2×2 in section 2's L3 is the single fact this domain rewards knowing cold.
• The standing trap is describing one method's pipeline while naming another. "DPO trains a reward model then runs PPO" describes RLHF; catch the mismatch by asking which auxiliary piece the sentence actually names.
• Resident-model count is a mechanical readout of these requirements, not an arbitrary implementation detail: RLHF's four objects, DPO's two, and GRPO's two-to-three all follow directly from which of reward model and critic each method needs.

Knowing which alignment method needs which supporting piece answers "how do you change a model's disposition once you've decided to change weights at all." It says nothing about the other half of Fine-Tuning's territory — training the smaller, more specialized models that make retrieval and search work in the first place. Next: M5-03 covers contrastive loss for embeddings, the training objective that pulls semantically similar pairs together and pushes dissimilar pairs apart, and that underlies every retrieval and semantic-search model a RAG pipeline depends on.