Reward Models, Reward Hacking, and Preference Data

A reward model is a neural network that takes a prompt and a candidate response and outputs a single scalar score intended to predict which response a human would prefer. It is normally built by taking a language model, removing the token-prediction head, and attaching a scalar output head, then training it on a dataset of human comparisons.

Its training objective is a pairwise ranking loss, and the shape of that objective explains almost everything about how reward models behave. Each training record is a prompt with a chosen response and a rejected response, and the loss pushes the model to score the chosen one higher than the rejected one. It is never told how much higher, and it is never told what a good absolute score is. Consequently:

• A reward model's scores are meaningful only in comparison. A score of 3.2 means nothing on its own. It means something relative to another response to the same prompt. This is why "the reward went from 0.8 to 2.4" is not a claim about quality; it is a claim about the proxy.
• A reward model learns the annotators' preferences, including their biases. If the annotators consistently preferred longer answers, the reward model learns that longer is better, and it will say so about answers no human ever saw.
• A reward model has a distribution it was trained on, namely the outputs of the SFT policy at the time the data was collected. Score a response far outside that distribution and the model is extrapolating with no basis for it — which is exactly the situation a policy-optimization run creates as the policy drifts.

Reward hacking — also called reward gaming, specification gaming, or in its economics form Goodhart's law — is what happens when a policy discovers behaviours that score highly under the reward model while being worse by the standard the reward model was supposed to represent. It is not an exotic edge case; it is the expected outcome of hard optimisation against an imperfect proxy, and preventing it is a design constraint on the whole alignment pipeline rather than a bug to fix afterwards.

L1 — The intuition: the exam that the student can see

You want to teach good writing, so you write a rubric, grade a thousand essays with it, and then let a student optimise their grade for as long as they like. If the rubric awards a point per paragraph, you will get essays with a great many very short paragraphs. The student is not cheating in any sense they would recognise — they are doing exactly what you measured — and the essays are worse.

The reward model is the rubric, learned from examples rather than written down. The policy is the student, and it is an extremely persistent one: policy optimization will run millions of samples looking for whatever the rubric over-rewards. If any exploitable regularity exists in the reward model, the policy will find it, because finding it is the only thing the policy is being paid to do.

This is why the interesting question is never "is the reward model good?" It is "how hard can I optimise against this reward model before it stops correlating with quality?"

L2 — The mechanics of preference data collection, and the six things that corrupt it

Collection. Sample a prompt from a distribution that reflects real usage. Generate two or more responses from the SFT policy, typically with sampling temperature high enough to produce genuine variety (04-05) — if both candidates are near-identical the comparison carries no information. Present them to an annotator, usually blinded to which is which and with randomised presentation order. Record the preference. Optionally collect a strength-of-preference rating or a free-text rationale.

What corrupts the data, in rough order of how often it bites:

Note what all of these have in common: they are data problems, not algorithm problems. No amount of clever optimisation repairs a preference set that systematically prefers longer, more confident, more agreeable answers. The fix is upstream — in guidelines, in annotator training, in randomisation, in agreement measurement, and in adjudicating disagreements. 09-03 is the rubric-design discipline this depends on and 08-02 is the label-noise framing.

Training the reward model. Ordinary supervised training on the pairwise loss, with two evaluations that matter:

1. Held-out agreement. What fraction of unseen human comparisons does the reward model order correctly? This is the reward model's accuracy, and a reward model near chance is worthless. Interpreting a number like this needs the sample-size discipline from 09-09, because a small held-out set cannot distinguish 62% from 68%.
2. Agreement on the hard pairs. Overall agreement is inflated by easy comparisons where one response is obviously bad. The informative measure is agreement on pairs where humans found the choice difficult, because those are the comparisons that encode real taste.

L3 — The mechanics of reward hacking, and what actually defends against it

Reward hacking follows a characteristic trajectory. Early in policy optimization, reward-model score and genuine quality rise together, because the easy wins — being more helpful, more complete, better structured — are wins under both. Then they separate. The policy has exhausted the improvements that are genuinely good and starts finding the ones that are only good according to the proxy.

      quality
        ▲
        │        ┌──────╮
        │       ╱        ╲___________     ← true quality plateaus, then declines
        │      ╱
        │     ╱
        │    ╱ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─  ← reward-model score keeps rising
        │   ╱
        └──┴────────────────────────────────▶  optimisation steps
              ↑
        the divergence point

Constructed illustration of the shape, not measured data. The important structural fact is that the divergence is invisible if the reward model is your only measurement. By construction, the reward curve keeps going up. That is what it means for the proxy to be gamed.

What the hacks look like in practice, as concrete named patterns:

The last row is the pathological case and the reason the KL penalty exists. Once the policy has drifted far enough from the SFT policy, the reward model is being asked to score text unlike anything it was trained on, and its outputs there are essentially arbitrary — often arbitrarily high. A policy that finds such a region will camp in it, and the output is gibberish with an excellent score.

The three defences, and why you need all three:

Defence 1 — the KL penalty. Penalise divergence from the frozen SFT reference policy, as 11-06 describes. This does not make the reward model better; it limits how far the policy may travel to exploit it, keeping the policy inside the distribution where the reward model's scores mean something. Monitor the KL divergence as a first-class metric: a sharply rising KL is the earliest available warning that the policy is heading somewhere the reward model cannot evaluate.

Defence 2 — an independent held-out evaluation the optimizer cannot see. This is the load-bearing defence and it is the same principle as a test split (01-07). The reward model is now part of the training loop, which means it has the epistemic status of training data, not of evaluation. You need a separate measurement — human ratings on a frozen set, or task metrics, or a general-capability eval — that is never used as an optimisation target. If it were used, it would be gamed too.

Defence 3 — preference data designed to resist the shortcut. Guidelines that instruct annotators explicitly to disregard length, to penalise confident inaccuracy, to reward appropriate uncertainty, and to resist agreeable framing. Randomised presentation order. Measured inter-annotator agreement with adjudication. Prompts sampled from real traffic. This is the only defence that addresses the cause; the other two contain the symptom.

Two further techniques worth knowing by name:

• Reward model ensembles. Several reward models trained on different data splits, with the policy optimised against a conservative combination such as the minimum score. Exploiting one model's quirk is easier than exploiting several simultaneously.
• Iterative data collection. Rather than collecting all preferences once, collect a round, optimise, sample fresh responses from the new policy, and collect preferences on those. This keeps the reward model's training distribution near the policy's current output distribution, which is exactly the mismatch that lets hacking flourish. It costs an ongoing annotation programme rather than a one-off dataset.

And the honest framing to keep: the reward-hacking problem is not solved. These are mitigations that push the divergence point later and make it easier to detect. Treat any claim that a technique eliminates reward hacking with the scepticism you would apply to a claim that a metric cannot be gamed.

Four things that all output numbers about model output, routinely confused. Distinguishing them is genuinely useful and is fair game on an exam that likes metric-to-purpose matching.

Two clarifications this table is designed to force.

A reward model is not an evaluation metric, even though it produces a quality-shaped number. The instant it enters the optimisation loop it becomes training signal, and reporting the reward score as evidence of improvement is like reporting training loss as evidence of generalisation. If you want to know whether the aligned model is better, you need a measurement the optimizer never touched.

An LLM-as-a-judge is not a reward model, though people use the phrase loosely. A judge is a general model prompted to grade; a reward model is a purpose-trained scalar scorer. They share a failure family — both are proxies with biases including a preference for longer and more confident text — and they occupy different positions: the judge sits outside the loop as an evaluator, the reward model sits inside it as the objective. Using a judge as a reward signal is one way to do RLAIF, and it inherits both sets of problems at once.

A team runs policy optimization against a reward model and tracks four things: the reward-model score, a frozen human-rated eval slice, average response length, and the KL divergence from the reference policy. All numbers are a constructed scenario, not measurements.

Step   RM score   Human eval (%)   Avg length (tok)   KL from ref
──────────────────────────────────────────────────────────────────
   0      0.00         62.0              180              0.00
 200      0.41         67.5              205              0.9
 400      0.78         71.0              240              2.1
 600      1.12         72.5              310              4.4
 800      1.49         71.0              420              8.7
1000      1.83         66.5              610             15.2
1200      2.16         58.0              880             24.9
──────────────────────────────────────────────────────────────────

Step 1 — find the divergence point. Human eval rises to step 600 and falls after. Reward-model score rises monotonically throughout.

Human eval peak:      step 600, 72.5%
RM score at peak:     1.12
RM score at step 1200: 2.16  (+93% beyond the peak)
Human eval at 1200:   58.0%  (-14.5 points from peak,
                              -4.0 points below the starting model)

Between step 600 and step 1200 the reward model reports a 93% improvement while human raters report a model that is worse than the one they started with. This is reward hacking in a single table.

Step 2 — identify the mechanism from the diagnostics.

Length at step 600:   310 tokens
Length at step 1200:  880 tokens
Growth factor:        2.84×

Response length nearly tripled past the divergence point. That is the length-inflation signature, and it points straight at the preference data: annotators mildly preferred longer answers, the reward model encoded it as a general rule, and the policy exploited it to the point of absurdity.

Step 3 — check the KL divergence as the early-warning signal.

KL at step 600 (human eval peak):  4.4
KL at step 800 (eval turns down):  8.7   ← doubled
KL at step 1200:                  24.9   ← 5.7× the peak-quality value

KL divergence roughly doubles at exactly the step where human eval turns down, and then runs away. That is the value of monitoring KL: it flags the drift without needing the human eval, so it can be watched cheaply at every step while human evaluation runs only occasionally.

Step 4 — cost the mistake of not measuring independently. Suppose the team had only tracked the reward score, as teams do when human evaluation is expensive:

Observed:  RM score 0.00 → 2.16, monotonic, no anomalies
Conclusion they would draw: a large, clean success
Reality:   62.0% → 58.0% human-rated quality, a 4-point regression

They would have shipped a worse model with a triumphant metric. The reward curve cannot detect reward hacking, because reward hacking is defined as the reward curve going up.

Step 5 — compute the per-token cost consequence, because it compounds. Length inflation is not only a quality problem; it is a bill. Using the cost framing from 12-09, at a hypothetical $0.60 per million output tokens and 2 million requests per month:

Before:  180 tok × 2e6 = 3.6e8 tokens → $216 / month
Step 600: 310 tok × 2e6 = 6.2e8 tokens → $372 / month
Step 1200: 880 tok × 2e6 = 1.76e9 tokens → $1,056 / month

The hacked model costs 4.9× the original in output tokens to deliver worse answers, and it is slower per response as well (12-10). Reward hacking is expensive in three currencies at once.

Step 6 — state the corrective actions in order.

1. Roll back to the step-600 checkpoint (best independent eval).
2. Raise the KL coefficient β so the policy cannot travel as far.
3. Add an explicit length instruction to the annotation guidelines
   and re-label a sample to check whether length bias is present.
4. Add average response length as a monitored guardrail metric
   with a hard ceiling.
5. Keep the human eval slice strictly out of the optimisation loop.
6. Consider a reward-model ensemble, or iterative data collection
   so the RM's training distribution tracks the policy.

Step 1 is only possible because checkpoints were kept and an independent evaluation existed to select among them. Both are process decisions made before the run, not fixes available after it.

The two rows worth internalising as rules rather than entries: never optimise against a reward model you have not measured against held-out human preferences, and never treat the reward score as evidence of improvement. The first stops you from optimising noise. The second stops you from shipping a hack.

The Experimentation domain's own scope statement names the use of human subjects in labeling and in reinforcement learning from human feedback, and that domain is 22% of the exam. Reward models are the artifact that human labelling produces in an RLHF pipeline, and reward hacking is the named failure mode of the technique. The blueprint also asks the associate to compare models using statistical performance metrics and to identify factors that could affect the results of research — and "our optimisation target is a proxy that decouples from the goal under pressure" is precisely such a factor.

There is a second reason this content earns exam weight: it is the general lesson about proxy metrics, in its sharpest form. A candidate who understands reward hacking understands why perplexity is not quality, why ROUGE is not usefulness, why a benchmark score is not capability, and why an LLM judge is not a user. That transfer is why this lesson arrives late in the course rather than early — it lands as an earned insight rather than an assertion.

Question phrasings to expect:

• "What is the purpose of the reward model in RLHF?" — to provide a scalar approximation of human preference as the optimisation signal.
• "What is reward hacking?" — the policy finding behaviours that maximise the reward model's score without genuinely improving quality.
• "How is a reward model trained?" — on human pairwise preference comparisons, with a ranking loss.
• "Which of the following best mitigates reward hacking?" — a KL penalty against the reference policy, plus independent held-out evaluation.
• "A team observes a rising reward score and falling user satisfaction. What is happening?" — reward hacking; the proxy has decoupled from the objective.
• "Why must the evaluation used to judge an aligned model be independent of the reward model?" — anything used as an optimisation target becomes gameable and stops measuring.
• "What data-collection defect makes a model produce needlessly long answers?" — length bias in the preference labels.
• "What does inter-annotator agreement tell you about a preference dataset?" — whether the preference signal is consistent enough to learn from at all.

Distractor families:

That last one is worth dwelling on, because it is the sophisticated mistake. Removing the explicit reward model removes one surface for exploitation. It does not remove length bias, sycophancy bias, or annotator inconsistency, because those are properties of the labels. If your annotators preferred longer answers, DPO will produce a model that prefers longer answers.

Reward hacking is the policy learning behaviours that maximise the reward model's score without improving — and often while degrading — the quality the reward model was meant to represent. The essential point is that nothing is malfunctioning. The reward model is scoring as trained. The optimizer is maximising as designed. The policy is doing exactly what it was asked. The failure is in the specification: the reward model is a proxy, and maximising a proxy is not maximising the goal.

The general principle has a name outside machine learning. Goodhart's law: when a measure becomes a target, it ceases to be a good measure. RLHF is the most literal instantiation of that law available in engineering practice, because it takes a measure of quality and makes it, explicitly and single-mindedly, the target of a training loop with millions of samples of search behind it.

The diagnostic signature to recognise on an exam:

reward-model score          ↑  rising, often monotonically
independent human eval      ↓  flat then falling
KL divergence from ref      ↑  rising, often accelerating
some surface feature        ↑  length, confidence, bullets, disclaimers

Any two of those four in combination should be read as reward hacking, and the fourth row is usually the one that tells you which hack.

You contain it rather than eliminate it, with three layers that address different parts of the problem.

Constrain the policy. The KL penalty against the frozen reference policy keeps the optimised model near the distribution the reward model was trained to evaluate. Tune the coefficient deliberately: too weak and the policy escapes into territory where the reward model's scores are meaningless, too strong and alignment does not happen. Monitor KL divergence continuously; it is the cheapest early-warning signal available.

Measure outside the loop. Maintain a held-out evaluation the optimizer never sees — human ratings on a frozen slice, task-specific metrics, general-capability slices. Select checkpoints on that, not on reward. This is the same discipline as a test split (01-07) applied to a stage where people frequently forget it, and it is the defence that actually catches the failure rather than delaying it.

Fix the data. Every hack traces back to something the preference labels rewarded. Guidelines that name the shortcuts and forbid them, randomised presentation, measured agreement, adjudicated disagreements, and prompts drawn from real traffic. This is the only layer that addresses the cause.

Two supplements worth knowing:

• Reward-model ensembles with a conservative aggregation, so a quirk in one model is not a free lunch.
• Iterative preference collection, keeping the reward model's training distribution near the policy's current outputs.

And the general habit that outlives this lesson: whenever you make a metric an optimisation target, add a second measurement that is not. It applies to reward models, to eval metrics, to benchmark scores (10-01), and to the CI gates in 10-04. One number you optimise, one number you watch.

Because policy optimization needs a score for every sampled response, and a run involves an enormous number of samples. Consider the arithmetic under a constructed but reasonable setup:

Optimisation steps                       1,200
Responses sampled per step                  64
Total responses needing a score          76,800

Human judging at 1.5 min per item:
  76,800 × 0.025 h  =  1,920 annotator-hours
  at $40/h          =  $76,800
  at 8 h/day, 1 annotator = 240 working days

Nearly $77,000 and a year of one person's time to score a single training run — and the run cannot proceed while waiting, because each step's samples depend on the previous step's weights. Human-in-the-loop scoring at that cadence is not merely expensive, it is architecturally impossible: the loop would run at human speed.

The reward model solves this by amortising the human effort. A few thousand human comparisons train a model that then scores tens of thousands of responses for the cost of a forward pass. That is the trade, stated plainly: you exchange a direct but unusable signal for a cheap, fast, approximate one — and reward hacking is the price of the approximation.

This is also the cleanest way to see what DPO changes. DPO trains directly on the fixed preference pairs, so it never needs to score fresh samples and never needs the amortising model. It pays for that with being offline: it cannot respond to how the policy evolves. Different point on the same trade-off surface, and 11-06 has the comparison.

There is no published number that transfers across tasks, and the useful framing is a quality-before-quantity ordering with three gates. The volumes below are illustrative reasoning about ratios, not sourced recommendations.

Gate 1 — agreement before volume. If annotators do not agree with each other on the same pairs, additional data adds noise rather than signal. Measure agreement on a small double-labeled sample first and fix the guidelines until it is acceptable. A thousand consistent comparisons beat ten thousand inconsistent ones, because inconsistency is exactly what a ranking loss cannot learn from.

Gate 2 — coverage of the behaviours you care about. A preference set that only contains easy comparisons teaches the reward model to detect obviously-bad answers, which the policy already avoids. The informative comparisons are the hard ones, where two plausible responses differ in a way that matters. Deliberately sample prompt types and response pairs that probe the distinctions you actually want expressed.

Gate 3 — enough held-out data to measure agreement meaningfully. You need a test split large enough that the reward model's agreement figure is not noise. 09-09 covers why a 50-item held-out set cannot distinguish a 62% reward model from a 70% one, and that distinction may be exactly the one you need.

The scaling intuition that follows: preference data volume should track the number of distinct behavioural dimensions you are trying to align, not the size of the model. Aligning one dimension — say, response length appropriateness — is a small-data problem. Aligning helpfulness, calibration, tone, refusal appropriateness, and format simultaneously is five problems sharing a budget, and it is where multi-attribute approaches like SteerLM [NVIDIA-DOC] become interesting, because they keep the dimensions labelled separately rather than collapsing them into one scalar where they can trade against each other invisibly.

• A reward model is a learned proxy for human preference, trained with a pairwise ranking loss on human comparisons. Its scores are meaningful only relatively, never absolutely.
• Reward hacking is the expected consequence of hard optimisation against a proxy, not an exotic bug. Nothing malfunctions; the specification is simply imperfect.
• The reward curve cannot detect reward hacking, because rising reward is what hacking is. You need a measurement the optimizer never sees.
• The named hacks are length inflation, confidence inflation, sycophancy, format spamming, boilerplate hedging, refusal creep, and degeneration — and each traces back to something the preference labels rewarded.
• Every hack is a data problem first. Guidelines that name and forbid the shortcuts, randomised order, measured agreement, and real-traffic prompts are the only fix that addresses the cause.
• Three defences, all needed: a KL penalty to constrain the policy's travel, an independent held-out evaluation to detect divergence, and preference data designed to resist the shortcut.
• Monitor KL divergence continuously. In the worked example it doubled at exactly the step human-rated quality turned down, giving a cheap signal that needed no human labels.
• Measure the reward model's own held-out agreement with humans before optimising against it. A near-chance reward model turns your alignment run into an expensive random walk.
• Human feedback cannot be used directly in a policy-optimization loop — the constructed example needed 76,800 scored responses for one run — which is precisely why the amortising proxy exists.
• Removing the reward model does not remove preference bias. DPO learns length bias and sycophancy from the labels just as faithfully.
• The transferable rule: whenever a metric becomes an optimisation target, add a second measurement that is not. This is the same lesson perplexity and ROUGE taught, at maximum pressure.

You now have the complete inventory of ways to change a model's behaviour, from a one-line prompt edit to a full RLHF pipeline, along with what each costs and how each fails. What remains is the decision itself — the rule that takes a described situation and picks the rung. 05-06 gave you a first pass at it before you knew what fine-tuning actually cost or what RAG actually required. Next: 11-08 settles it properly, with cost arithmetic on all three options and the specific clauses in a requirement that decide the answer before any technical argument begins.