RLHF: Reinforcement Learning from Human Feedback Explained

RLHF is a three-stage training procedure that aligns a language model with human preferences by learning a model of those preferences and then optimising the language model against it.

Stage 1 — Supervised fine-tuning (SFT). Start from a pretrained base model and fine-tune it on high-quality instruction-response demonstrations. The output is a policy: a model that follows instructions well enough that its outputs are worth comparing. This stage is 11-02 in full.

Stage 2 — Reward model (RM) training. Collect human preference labels: show annotators a prompt and two or more candidate responses from the SFT model, and have them rank or pick a winner. Train a separate model — usually initialised from the same family — to take a (prompt, response) pair and output a scalar score that agrees with the human rankings. The reward model is a learned proxy for human preference.

Stage 3 — Policy optimization. Fine-tune the language model with reinforcement learning to produce responses the reward model scores highly. The classical algorithm is PPO (proximal policy optimization). A penalty term keeps the optimised policy from drifting too far from the SFT policy, because unconstrained optimisation against a proxy reliably breaks the proxy.

Three framings to hold on to:

• The supervision signal is comparative, not demonstrative. In SFT a human writes the answer. In RLHF a human judges answers. Judging is cheaper per item and more consistent across annotators than authoring, which is exactly why the method scales.
• The reward model is a proxy and every proxy can be gamed. This is not an incidental risk; it is the central pathology of the method, and it has a name — reward hacking — and its own lesson, 11-07.
• Alignment is not capability. RLHF changes which of the responses a model could produce it actually reaches for. It does not teach the model anything it could not already do. A model that cannot reason through a problem will not learn to by being told which of two failed attempts was nicer.

L1 — The intuition: a taste teacher, then practice

Imagine coaching a writer. The demonstration approach is to hand them a hundred finished essays to imitate — that is SFT. It works, and it is limited by your ability to produce finished essays and by the fact that imitation caps out at the exemplars' quality.

The preference approach is different. You cannot write a hundred perfect essays, but you can look at any two drafts and say which is better. So you do that a few thousand times, and from your judgements you train an assistant editor who has learned your taste and can score a draft without you — that is the reward model. Then you have the writer produce drafts, the assistant editor score them, and the writer adjust toward higher-scoring drafts — that is policy optimization.

Two consequences follow from the analogy and both are real. The assistant editor is only as good as your judgements, so bad labels produce bad taste, faithfully learned. And a writer optimising hard against an assistant editor will eventually discover what the editor over-rewards — flattery, length, confident phrasing — and exploit it. Hence the constraint that keeps the writer near their original voice.

L2 — The mechanics, stage by stage

Stage 1 mechanics — SFT. Standard supervised fine-tuning as covered in 11-02: instruction-response pairs, next-token cross-entropy over the response tokens, chat template applied consistently. Increasingly this stage is run with LoRA rather than full-parameter training (11-05). Its output is called the SFT policy or the reference policy, and it plays two roles later: it is the model that generates candidates for preference labelling, and it is the anchor the optimised policy is penalised for drifting away from.

Stage 2 mechanics — the reward model.

• Data collection: sample a prompt, generate two or more responses from the SFT policy, present them to a human annotator, record which is preferred. The output is a set of records of the form (prompt, response_chosen, response_rejected).
• Architecture: take a language model and replace the token-prediction head with a scalar head, so it outputs a single number per (prompt, response) pair.
• Objective: a pairwise ranking loss. The model is trained so the score of the chosen response exceeds the score of the rejected one. Note carefully what this means — the reward model learns a relative ordering, not an absolute quality scale. Its outputs are only meaningful in comparison.
• Evaluation: the reward model's own accuracy is measured as agreement with held-out human preferences. A reward model that agrees with humans barely better than chance will produce an alignment stage that does nothing useful, so this is a checkpoint you cannot skip.

Stage 3 mechanics — policy optimization with PPO. This is the reinforcement-learning loop, and mapping RL vocabulary onto language modelling is what makes it comprehensible:

The loop: sample a prompt, have the policy generate a response, score it with the reward model, compute the KL divergence between the policy's token distribution and the reference policy's, subtract a penalty proportional to that divergence, and take a policy-gradient step that raises the probability of high-scoring behaviour.

objective ≈  E[ reward_model(prompt, response) ]
           − β · KL( policy ‖ reference_policy )

The KL penalty term is doing essential work. Without it, the policy will wander far from the SFT model in pursuit of reward-model score and will discover degenerate outputs that score highly and read terribly. The coefficient β trades alignment strength against fidelity to the reference: too small and the policy drifts into reward hacking, too large and nothing changes.

The word "proximal" in PPO refers to the same instinct implemented differently — PPO constrains how much the policy may change per update step, which is what makes policy-gradient training stable enough to run on a model this large.

What is resident during stage 3, and why RLHF is expensive. This is the operational fact that decides whether a team can do RLHF at all:

1. the policy being trained          — weights + gradients + optimizer state
2. the frozen reference policy       — weights only (for the KL term)
3. the reward model                  — weights only (for scoring)
4. optionally a value/critic model   — weights + gradients + optimizer state

Three to four models in memory simultaneously, plus a generation loop inside the training loop, because every step requires the policy to actually produce text before it can be scored. Apply the memory arithmetic from 11-04 and it is obvious why RLHF is mostly run by well-resourced organisations and why simpler alternatives were sought.

L3 — DPO, SteerLM, RLAIF, and why the alternatives exist

The complexity of stage 3 is the motivation for everything in this section.

DPO (Direct Preference Optimization) takes the same preference dataset and optimises the policy on it directly, with a supervised-style loss, skipping both the separate reward model and the RL loop. Instead of learning an explicit reward function and then maximising it, DPO uses a loss derived so that minimising it moves the policy toward preferring the chosen responses over the rejected ones, with a reference-policy term playing a role analogous to the KL penalty.

DPO's practical appeal is that it turns alignment back into something that looks like SFT — a dataset, a loss, a training loop — which brings it within reach of teams that could never stand up a PPO pipeline. What you give up is the online element: PPO scores fresh samples from the current policy, whereas DPO learns from a fixed set of comparisons collected earlier. The exam framing to remember is simply: DPO is a simpler alternative to RLHF that achieves preference alignment without a separate reward model or an RL loop.

SteerLM [NVIDIA-DOC] is NVIDIA's alignment technique and appears in NVIDIA's own trustworthy-AI materials as a method for aligning models with human feedback. Its distinguishing idea is attribute conditioning: rather than collapsing human preference into a single scalar reward, responses are labelled along multiple named attributes — helpfulness, quality, toxicity, humour, creativity and similar — and the model is trained to condition its generation on requested attribute values. The practical consequence is steerability at inference time: instead of one globally-aligned model, you get a model whose behaviour can be requested per call. Know SteerLM by name, know it is NVIDIA's, know it is an alignment approach using labeled attributes rather than a single preference score, and know it avoids the full RL machinery. Its appearance in the NVIDIA stack alongside NeMo Guardrails and the Model Card Generator is 13-02 territory.

RLAIF (reinforcement learning from AI feedback) replaces some or all human labelling with judgements from a strong model given a written constitution or rubric. Its appeal is cost and throughput — preference labelling is the bottleneck in RLHF and human annotators are slow and expensive. Its risk is that you are now aligning to a model's judgement, inheriting that model's biases wholesale, and the biases of LLM judges are well documented enough to take seriously (09-10).

Why the third stage cannot simply be dropped. A reasonable question: if preference data exists, why not just SFT on the chosen responses and ignore the rejected ones? Because doing so discards the contrast, which is where the information is. "A is better than B" tells you something that "A is good" does not — it tells you what to move away from. Preference optimisation, whether by PPO or DPO, uses both sides. SFT on the winners only uses half the dataset and cannot express dispreference at all. That is the cleanest statement of why alignment is a different stage rather than more SFT.

The two rows that matter most for exam questions are "separate reward model" and "prerequisite." RLHF has a reward model; DPO does not. Both need an SFT'd policy first. A question that offers "skip SFT and train the reward model directly on base-model outputs" is testing exactly that dependency.

And the confusable inside the confusable: alignment versus capability. These are orthogonal and the exam can test the distinction directly.

A team wants to align a support assistant so it hedges appropriately and refuses out-of-scope legal questions cleanly. They compare authoring demonstrations against collecting preferences. All figures are a constructed scenario, not measurements.

Assumptions:

Annotator loaded cost               $40 / hour
Time to author one good response     12 min  = 0.20 h
Time to judge one A/B comparison      1.5 min = 0.025 h
Target: 4,000 labeled items either way
Inter-annotator agreement checks: 10% double-labeled

Step 1 — cost of the SFT-only route (authoring).

4,000 items × 0.20 h            = 800 annotator-hours
+ 10% double-labeling           = 880 hours
880 h × $40/h                   = $35,200

Step 2 — cost of the preference route (judging).

4,000 comparisons × 0.025 h     = 100 annotator-hours
+ 10% double-labeling           = 110 hours
110 h × $40/h                   =  $4,400

Step 3 — the per-item ratio.

authoring : judging  =  0.20 h : 0.025 h  =  8 : 1
cost ratio           =  $35,200 : $4,400  =  8 : 1

Eight-to-one is the number that explains why preference-based alignment exists at all. For the same annotation budget you get eight times the labeled items — or the same number of items at one-eighth the cost. That is the economic engine of the whole method, and it follows only from the assumption that judging is faster than authoring, which is the assumption the field is built on.

Step 4 — but you still need SFT. The preference route does not replace stage 1; it follows it. So the honest comparison is not "authoring versus judging" but "authoring only" versus "some authoring plus much more judging":

Route A — SFT only:
    4,000 authored demonstrations                   $35,200

Route B — SFT then preference alignment:
    1,000 authored demonstrations (0.20 h each)
      1,000 × 0.20 × 1.1 × $40                   =  $8,800
    6,000 preference comparisons (0.025 h each)
      6,000 × 0.025 × 1.1 × $40                  =  $6,600
    ─────────────────────────────────────────────────────────
    total                                           $15,400

Route B buys a smaller SFT set and 6,000 preference comparisons for 44% of Route A's annotation cost. Whether that is the better model is an empirical question for the eval set — this arithmetic establishes only that it is affordable, which is the precondition for asking.

Step 5 — add the compute cost asymmetry, which runs the other way. Annotation is not the only budget. Using the memory arithmetic from 11-04, for a 7B model at bf16:

Route A (SFT with LoRA):
    1 model resident, adapter training      ≈  14 GB   → single GPU

Route B stage 3 with PPO:
    policy (trainable, LoRA)                ≈  14 GB
    frozen reference policy                 ≈  14 GB
    reward model (7B, inference only)       ≈  14 GB
    value/critic model (if separate)        ≈  14 GB
    ────────────────────────────────────────────────
    total                                   ≈  56 GB   → multi-GPU
    plus generation inside the training loop → far slower per step

Step 6 — read the trade honestly. Route B saves annotation money and spends compute and engineering complexity. That is precisely the trade DPO was designed to improve:

Route C — SFT then DPO:
    annotation: same as Route B              $15,400
    training:   policy + reference only     ≈  28 GB
    no reward model, no RL loop, no generation-in-the-loop

Route C keeps the annotation saving and removes most of the compute and complexity premium. For a team without a dedicated RL infrastructure group, that is usually the defensible plan — and "DPO as a simpler alternative to RLHF" is exactly how the exam frames it.

Step 7 — the number that gates everything. Before spending any of this, check the reward model's agreement with held-out human preferences (Route B) or the internal consistency of the preference set (Route C). If annotators only agree with each other 60% of the time, the preference signal is close to noise and no amount of optimisation extracts a real preference from it. Inter-annotator agreement is 09-03, and it is the gate, not a formality.

The last row is the one most teams skip. RLHF optimises toward a proxy for a preference; if you cannot measure the preference independently, you cannot tell whether the optimisation worked or whether it found a shortcut. The proxy-decoupling problem that 09-02 and 09-06 introduced with perplexity and ROUGE reappears here in its most consequential form, and 11-07 takes it apart.

RLHF is named directly in the Experimentation domain's own scope statement, which describes that domain as covering how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback. That is as explicit as blueprint coverage gets. The domain is 22% of the exam. Candidate reports independently place RLHF basics in the second tier of reported topic frequency, which means it appears regularly without being the exam's centre of gravity.

The examinable content is narrow and specific, and it favours structure over mechanism:

Question phrasings to expect:

• "Place the stages of RLHF in the correct order." — SFT, reward model, policy optimization. This is the single highest-probability RLHF question form.
• "What is the role of the reward model in RLHF?" — to score responses as a learned proxy for human preference, providing the signal the policy optimises against.
• "What kind of human input does RLHF require?" — comparative preference labels, not written target responses.
• "Which algorithm is classically used for the policy-optimization stage?" — PPO, proximal policy optimization.
• "What is DPO and how does it differ from RLHF?" — direct preference optimization; no separate reward model and no RL loop.
• "Which NVIDIA technique aligns a model using labeled attributes rather than a single reward score?" — SteerLM.
• "What is the purpose of the KL penalty during policy optimization?" — to keep the policy near the SFT reference and limit reward hacking.
• "Does RLHF improve a model's capability?" — no; it aligns behaviour with preferences. Capability is set by pretraining.

Distractor families:

One calibration note. This exam favours NVIDIA-stack answers when two options are technically defensible, and SteerLM is the NVIDIA-branded alignment answer. If a question asks how to align a model with human feedback using NVIDIA technology, SteerLM is the name being fished for.

Supervised fine-tuning, then reward model training from human preference labels, then policy optimization. Memorise it in that direction and be able to justify each dependency, because the justification is what distinguishes a memorised answer from an understood one:

The most common wrong ordering puts the reward model first, presumably because it is the stage that sounds most distinctive. Reject it on mechanism: the reward model is trained on comparisons between responses, and the responses come from the SFT policy. Without stage 1 there is nothing to compare.

Also note the two roles the SFT policy plays in stage 3 — it is both the starting point that gets optimised and, as a frozen copy, the reference the KL penalty measures drift against. That dual role is a nice detail to have ready.

RLHF trains an explicit reward model and then optimises the policy against it with reinforcement learning. DPO optimises the policy directly on the preference pairs with a supervised-style loss, with no reward model and no RL loop. Both consume the same kind of data — pairwise human preferences — and both require an SFT'd model first.

What that buys, concretely:

• Fewer models in memory. PPO keeps the policy, a frozen reference, a reward model, and often a value model. DPO keeps the policy and a reference. Applying the arithmetic from 11-04, that is roughly half the resident footprint for a same-size model.
• No generation inside the training loop. PPO must have the policy produce text at every step so the reward model can score it. DPO trains on a fixed dataset, so each step is an ordinary forward and backward pass. This is a large difference in both wall-clock time and implementation surface.
• No explicit proxy to hack. With no reward model, there is no scalar function for the policy to exploit. Preference-data flaws still propagate — if annotators systematically preferred longer answers, DPO will learn that too — so the pathology is reduced, not eliminated.

What you give up: PPO is on-policy, scoring fresh samples from the current policy, so it can respond to the policy's evolving behaviour. DPO learns from comparisons collected earlier against an earlier policy. Whether that matters depends on how far the policy moves.

For the exam, the compact statement is the one the study index uses: the pipeline is SFT → reward model → PPO, and DPO is a simpler alternative that skips the reward model and the RL loop.

Not directly, and treating it as a truthfulness mechanism is a category error worth naming. RLHF optimises for what annotators preferred, and annotators prefer responses that seem correct, well-written, confident, and helpful. Seeming correct and being correct are different properties, and where they diverge the preference signal follows the appearance.

That said, alignment can install two behaviours that reduce visible falsehood, and both are behaviours rather than knowledge:

1. Calibrated hedging — expressing uncertainty when uncertain, if annotators consistently preferred hedged answers over confidently wrong ones.
2. Abstention — saying "I do not have that information," if annotators preferred honest declines over invented answers.

Both are genuinely valuable and both are exactly the kind of thing preference data expresses well and demonstrations express poorly. Neither makes the model know more.

There is a specific risk running the other way. If annotators reward agreeable, confident, pleasant answers — and humans reliably do — the optimisation can produce a model that is more prone to telling you what you want to hear. That failure has a name, sycophancy, and it is a preference-data problem rather than an optimisation problem, which means the fix is in the annotation guidelines. The hallucination mitigation ladder — grounding via retrieval, citation, constrained decoding, guardrails, human review — is 09-12, and none of its rungs is an alignment stage.

Rarely the full PPO pipeline, and usually they should not try. The blockers are not conceptual, they are logistical, and they come in three kinds.

The data blocker. You need a preference-labelling operation: written guidelines, trained annotators, agreement measurement, adjudication of disagreements, and a pipeline that keeps feeding the process as the policy changes. This is an ongoing programme with people in it, not a dataset you buy once. 09-03 is what good rubric design looks like and it is a serious undertaking.

The compute blocker. Three to four models resident, plus text generation inside the training loop. From the section 4 arithmetic, a 7B alignment run lands around 56 GB before activations, against 14 GB for a LoRA SFT run — and it is far slower per step because of the generation.

The engineering blocker. PPO has more moving parts than any other stage in this module: reward normalisation, advantage estimation, KL coefficient scheduling, and a stability profile that punishes misconfiguration with silent degeneration rather than a clean error.

What a small team can realistically do, in escalating order:

The fourth row deserves emphasis because it is the answer most teams should take: modern instruct models arrive already aligned, and adapting one with a LoRA SFT run inherits that alignment work for nothing. The blueprint's framing of the associate role — contributing under senior supervision — is consistent with understanding RLHF thoroughly and running the pipeline rarely. Know it cold for the exam; reach for DPO or inherited alignment in practice.

• The order is strict and it is the most examinable fact here: SFT → reward model trained from human preference labels → policy optimization (PPO). The reward model ranks the SFT policy's outputs, so SFT cannot be skipped.
• RLHF's supervision is comparative, not demonstrative. Humans judge which of two responses is better rather than authoring the ideal one — roughly an eight-to-one labour advantage in the constructed example, and the economic reason the method exists.
• The reward model is a learned proxy for human preference, trained with a pairwise ranking loss, so its scores are meaningful only in comparison.
• The KL penalty against the frozen SFT policy is essential, not optional. Without it the policy drifts and discovers degenerate outputs that score well and read badly.
• Alignment is orthogonal to capability. RLHF changes which achievable output the model reaches for; it does not raise the ceiling set at pretraining.
• RLHF is expensive: three to four models resident during stage 3 plus generation inside the training loop, which is why simpler alternatives were sought.
• DPO is the simpler alternative — same preference data, no explicit reward model, no RL loop, roughly half the resident footprint.
• SteerLM is NVIDIA's alignment approach [NVIDIA-DOC], conditioning generation on labeled attributes and giving inference-time steerability rather than one globally-aligned model.
• RLHF does not make a model truthful. It optimises for what annotators preferred, which can mean confident and agreeable. It can install hedging and abstention, which are behaviours.
• Most teams should inherit alignment from an already-aligned instruct model and adapt lightly, rather than standing up a PPO pipeline.

There is a hole in everything above, and it is the hole every proxy metric in this course has had. The reward model is not human preference — it is a model of human preference, trained on a finite set of labels by fallible annotators, and the policy is being optimised as hard as possible against it. You already know from perplexity and ROUGE what happens when you optimise a proxy instead of the thing it stands for. Next: 11-07 takes the reward model apart — how preference data is collected and what corrupts it, what reward hacking looks like when it happens, and why a rising reward curve is not evidence that anything got better.