LLM-as-a-Judge: How It Works, How to Validate It, and Where It Fails

An LLM judge is an evaluation function implemented as a prompt. You supply a rubric and the material to be judged; the model returns a verdict. Three architectures, and the difference between them matters more than most teams realise:

A fourth mode worth naming because it appears throughout RAG evaluation: evidence-grounded checking, where the judge is given a claim and a passage and asked only whether the passage supports the claim. This is the narrowest and most reliable judge task, because it is close to a natural-language-inference decision rather than an aesthetic judgement, and it is how faithfulness is usually computed in 09-07.

What a judge is not. It is not ground truth. It is an approximation of human rubric scoring, and its quality is defined by how well it agrees with human labels on the same items. That framing dictates everything: you cannot report a judge score without knowing its human agreement, and you cannot interpret that agreement without knowing the human-human agreement it is trying to match (09-03).

L1 — Intuition: a rubric with a model attached

Writing a rubric forces you to say what "good" means in observable terms. A judge is that rubric handed to a model instead of a person, thousands of times, for a fraction of a cent each. Everything that makes a rubric good for humans — observable criteria rather than adjectives, a defined decision order, named edge cases — makes it good for a judge too. Everything that makes a rubric ambiguous for humans makes the judge silently inconsistent, which is worse, because a judge never tells you it was unsure.

L2 — Mechanism: building and validating a judge

Step 1 — Write the rubric as observable conditions. "Rate helpfulness 1–5" invites drift. Instead: "Score 1 if the answer does not address the question. Score 3 if it addresses the question but omits a stated constraint. Score 5 if it addresses the question and satisfies every stated constraint." Anchor each level with a worked example inside the prompt. This is the same discipline as 09-03, applied to a model.

Step 2 — Constrain the output. Ask for a structured verdict — a JSON object with a score field and a short reason field — so parsing is reliable and the reason is auditable. 05-05 covers getting structured output reliably. Order matters: asking for the reason before the score generally produces more stable verdicts than asking for the score first, because the score then follows stated evidence rather than being justified after the fact. Do not over-read this as the model "reasoning"; 05-03 explains why emitted reasoning is not a faithful trace. Its value here is auditability — you can read why a verdict was wrong.

Step 3 — Neutralise position. For pairwise judging, evaluate every pair twice with the order swapped, and count a win only when both orders agree. Disagreement between the two orders is a tie — and the rate of such disagreements is a direct, free measurement of your judge's position bias.

Step 4 — Validate against human labels. This is the step that is skipped, and it is the step that makes the metric legitimate.

1. Human-label a subset (50-100 items) with two annotators, independently
2. Compute human-human agreement (Cohen's kappa) — this is your ceiling
3. Run the judge on the same items
4. Compute judge-human agreement (same statistic, judge vs the adjudicated human label)
5. Compare: judge-human kappa relative to human-human kappa

If human-human agreement is 0.62 and judge-human agreement is 0.58, your judge is nearly as good as an additional human annotator, and you should use it. If human-human is 0.85 and judge-human is 0.45, the task is crisp and the judge is not up to it — fix the rubric, or use a stronger judge model, or keep humans. The absolute judge-human number is meaningless without the human-human number, and this is the single most important methodological point in the lesson.

Step 5 — Pin every version and re-validate periodically. Judge model and version, the rubric prompt (hashed or versioned), the decoding settings, and the parsing logic are all part of the metric's definition. Change any one and the metric changes. Re-validate on a fresh human-labelled sample each quarter, because a hosted model can change beneath you.

L3 — Depth: the four biases, mechanically

Position bias. In pairwise judging, the position an output occupies affects its chance of winning. The mechanism is straightforward autoregressive conditioning: the first candidate establishes the frame against which the second is read, and the judge's own verdict token is conditioned on the full sequence including presentation order. The consequence is that a judge can prefer "A" over "B" and then prefer "B" over "A" when the same two texts are swapped. Mitigation: dual-order evaluation with agreement required, treating disagreements as ties. The disagreement rate is your bias measurement, and you should report it.

Verbosity bias. Judges systematically favour longer, more elaborate answers, even when the extra length adds nothing. Two plausible mechanisms: length correlates with thoroughness in the preference data that shaped the judge, and a longer answer simply contains more surface features that look like effort. The consequence is severe for optimisation — if you use a verbosity-biased judge to select prompts, you will select for padding, and your product will get wordier without getting better. This is Goodhart's problem (09-05) with a specific direction. Mitigations: include conciseness as an explicit rubric criterion; report answer length as a guardrail metric alongside the judge score and treat a length increase as suspicious; and where the task permits, cap or normalise length before judging.

Self-preference bias. Judges tend to score text from their own model family more favourably. The plausible mechanism is stylistic familiarity — a judge assigns higher quality to phrasing patterns close to its own generation distribution. The consequence is a rigged comparison: using model X as the judge to compare model X against model Y advantages X for reasons unrelated to quality. Mitigations: never use a model to judge its own outputs in a competitive comparison; use a judge from a different family than either candidate; or use a panel of judges from different families and require agreement. If you must use the same family, disclose it as a limitation.

Rubric drift. The criteria the judge actually applies shift over time and across items even though the rubric text is unchanged. Sources: a hosted model version updating under you; a prompt edit that looks cosmetic; stochastic decoding producing different interpretations; and long-context effects where a batch of items in one call influences each other's verdicts. The consequence is the worst kind for a time series — your dashboard moves and you attribute it to the system. Mitigations: pin the model version explicitly; version the rubric prompt and record its hash with every score; judge one item per call rather than batching; include a small set of calibration items with known correct verdicts in every run and alert if the judge's verdict on those changes. That last technique is cheap and catches silent drift better than anything else.

Two further weaknesses that are not usually listed among the four but matter operationally:

• Poor absolute calibration. Pointwise judges cluster their scores (a 1–5 scale that in practice only ever emits 3, 4 and 5) and their thresholds drift. Pairwise comparison is more reliable than absolute scoring, which is why leaderboards use it. If you need an absolute gate, anchor the scale with in-prompt examples and validate the threshold against human labels.
• Non-determinism. Even at temperature 0 a judge's verdict can vary between runs (09-11). That means judge-based metrics have run-to-run variance which must be measured, not assumed away — the same-config-twice check from 09-09.

The stack that follows from this table is a three-tier design: reference-based metrics on every commit (free, deterministic, catch gross regressions); judge-based metrics nightly or per release (rubric-level coverage at scale); human evaluation quarterly on a sample (ground truth, and the recalibration of the judge). Each tier validates the one below it. A team with only the first tier cannot see quality; a team with only the second tier cannot see that its judge has drifted; a team with only the third cannot ship weekly.

One comparison the exam probes directly: a judge and a reward model are the same idea at different points in the pipeline. A reward model is a learned scorer trained on human preference labels and then used as an optimisation target inside RLHF (11-06, 11-07); a judge is a prompted scorer used as a measurement. Both approximate human preference, both inherit the noise in the labels they were validated or trained against, and both can be gamed by the system being optimised. The difference is that optimising against a reward model is intended, which is what makes reward hacking a first-order concern there — whereas optimising against a judge is usually accidental, which is what makes it insidious here.

Constructed example. All counts are invented for the arithmetic.

Part 1 — position bias in a pairwise judge

A judge compares 100 pairs of outputs from systems A and B. Each pair is judged twice: once with A shown first, once with B shown first.

Step 1 — the naive, wrong reading. If you had only run the first row, you would report "A wins 62% of comparisons" and ship A.

Step 2 — the swapped order tells a different story. With B first, A wins only 45 of 100. Averaging the two orders:

A's average win rate = (62 + 45) / 200 = 107 / 200 = 0.535

A wins 53.5% of comparisons after order-balancing — a nearly even split, not a 62/38 rout.

Step 3 — quantify the position bias. The first-presented candidate won 62 times when it was A, and 55 times when it was B:

first-position win rate = (62 + 55) / 200 = 117 / 200 = 0.585

The candidate presented first wins 58.5% of the time regardless of which system it is. That 8.5-point advantage over the 50% baseline is the judge's position bias, and it is bigger than the 3.5-point real difference between the systems. The bias exceeds the effect. Anyone judging in a single fixed order would have measured mostly the bias.

Step 4 — the consistency-required reading. Count a win only when both orders agree.

Suppose the per-pair breakdown is: 41 pairs where both orders picked A, 34 pairs where both picked B, and 25 pairs where the two orders disagreed.

consistent A wins = 41
consistent B wins = 34
inconsistent      = 25  → treated as ties

A's win rate among decided pairs = 41 / 75 = 0.547
inconsistency rate               = 25 / 100 = 0.250

A quarter of all pairs got a different verdict depending on presentation order. That inconsistency rate is the honest headline about your judge, and it should be reported with every judge-based result. A 25% inconsistency rate does not mean the judge is useless — it means 25% of these pairs are genuinely close, and the judge is telling you so, if you let it.

Step 5 — apply the sample-size arithmetic. A's 54.7% win rate among 75 decided pairs, against a 50% null:

SE = sqrt(0.5 × 0.5 / 75) = sqrt(0.003333) = 0.0577
z  = (0.547 - 0.500) / 0.0577 = 0.81

Not significant (09-09). The correct conclusion: A and B are indistinguishable on this rubric with this judge at this sample size. Compare that with the "A wins 62%" that a single-order run would have produced, and you can see how much damage the missing swap does.

Part 2 — validating the judge against humans

Take 100 items and label them with two human annotators plus the judge. All three assign pass/fail against the same rubric.

Human-human agreement. The annotators agree on 84 items. Annotator 1 passed 70, annotator 2 passed 74.

P_o = 0.840
P_e = (0.70 × 0.74) + (0.30 × 0.26) = 0.5180 + 0.0780 = 0.5960
κ_HH = (0.840 - 0.596) / (1 - 0.596) = 0.244 / 0.404 = 0.6040

Human-human κ = 0.604 — "substantial" by the usual bands (09-03). This is the ceiling.

Judge-human agreement. Adjudicate the two humans into one gold label (majority plus a tiebreak), which passes 72 of 100. The judge passes 78 and agrees with the gold label on 81 items.

P_o = 0.810
P_e = (0.72 × 0.78) + (0.28 × 0.22) = 0.5616 + 0.0616 = 0.6232
κ_JH = (0.810 - 0.6232) / (1 - 0.6232) = 0.1868 / 0.3768 = 0.4958

Judge-human κ = 0.496 — "moderate".

Step 6 — interpret the comparison, which is the whole point.

κ_HH = 0.604   (two humans with each other)
κ_JH = 0.496   (judge against adjudicated humans)
ratio = 0.496 / 0.604 = 0.821

The judge captures about 82% of the agreement that two humans achieve with each other. That is genuinely usable: the judge is not as good as a second human annotator, but it is close enough to track direction of travel at scale, especially if you continue to sample human labels for calibration. Had κ_JH come back at 0.25 against a κ_HH of 0.85, the judge would be unusable and the correct decision would be to fix the rubric or keep humans.

Step 7 — read the judge's bias direction. The judge passed 78 items where the gold label passed 72. The judge is systematically more lenient by 6 points. That is a calibration offset, not random error, and it is correctable: tighten the rubric's pass criterion, or record the offset and adjust the threshold. Random disagreement and systematic offset need different fixes, and the marginal counts are what distinguish them — exactly the diagnostic from 09-03.

Step 8 — check verbosity. Compute the mean answer length for judge-passed and judge-failed items:

mean length, judge PASS: 187 words
mean length, judge FAIL: 121 words
mean length, human PASS: 154 words
mean length, human FAIL: 148 words

Humans show almost no length difference between pass and fail (154 versus 148). The judge shows a 66-word gap. That gap is verbosity bias, visible without any special instrumentation — just group your lengths by verdict and compare with the human grouping. If you were to use this judge to select prompts, you would drift toward longer answers, and the human labels say length is not what makes an answer good here.

Two hard rules worth stating separately. Never deploy a judge you have not validated against human labels on your own task — published agreement figures for other tasks do not transfer. And never use a judge as an optimisation target without a guardrail on the dimension you know it is biased about, which in practice means tracking answer length whenever a judge is in the loop.

The Experimentation domain is 22% of the exam, and its suggested-reading list names evaluating RAG applications — where judge-based faithfulness and relevance metrics are the standard implementation. The domain's own scope statement covers "AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)", and the judge sits exactly between those two clauses: it is an evaluation method whose validity is defined by its agreement with human labelling, and it is mechanically the same object as the reward model inside RLHF.

The objective-numbering defect, stated where the objectives are cited. The official study guide prints this domain's objectives as 3.1–3.5, and those lines are a verbatim duplicate of the Data Analysis domain's 2.1–2.5 — data-mining awareness, comparing models using statistical performance metrics, conducting data analysis under supervision, creating graphs, identifying trends. Read literally, 22% of the exam has objectives describing charts and data mining rather than model evaluation and RLHF, contradicting the section's own scope statement. Published candidate reports independently confirm evaluation, hallucination and RLHF content on the exam, so the derived scope governs. The objectives that legitimately apply here are the duplicated pair 2.2 / 3.2 (compare models using statistical performance metrics), 2.5 / 3.5 (identify factors that could affect the results of research — position and verbosity bias are precisely such factors), 4.5 (monitor functioning of experiments and software processes), and from Trustworthy AI 5.4 (minimise bias in AI systems), since a biased judge is a bias-injection mechanism inside your own measurement.

Question phrasings to expect:

• "What is LLM-as-a-judge?" → Using a language model, prompted with a rubric, to score or compare outputs in place of human raters.
• "A judge prefers whichever response is shown first. What is this called?" → Position bias. Mitigation: evaluate both orders and require agreement.
• "A judge consistently rates longer answers higher. What is this and why does it matter?" → Verbosity bias; if used as an optimisation target it selects for padding rather than quality.
• "Why should you not use the same model to judge its own outputs against a competitor's?" → Self-preference bias favours text from the same model family.
• "How do you validate an LLM judge?" → Compare its labels against human labels on the same items, and interpret its agreement relative to human-human agreement.
• "A judge agrees with human labels at kappa 0.50. Is that good?" → It depends entirely on the human-human kappa; 0.50 against a human ceiling of 0.60 is usable, against 0.85 it is not.
• "What is rubric drift?" → The criteria a judge effectively applies shifting over time or across items, typically from an unpinned model version, an edited prompt, or stochastic decoding.
• "Which evaluation approach can assess helpfulness without a reference answer?" → Human evaluation or LLM-as-a-judge; reference-based metrics cannot.
• "Why is an LLM judge not fully reproducible even at temperature 0?" → Generation is not fully deterministic across runs, and hosted model versions can change (09-11).

Distractor families. (1) A judge presented as ground truth — the most common; it is a proxy validated against humans. (2) Judge deployed with no human validation described as best practice. (3) Same-model judging described as a fair comparison — ignores self-preference. (4) Single-order pairwise comparison described as sound — ignores position bias, which in the worked example was larger than the real effect. (5) A judge offered for a deterministic check — exact match, schema validation, or a numeric comparison is cheaper and exact. (6) Judge described as deterministic. (7) Verbosity bias confused with position bias — know which name goes with which mechanism.

Four steps, and the fourth is the one that gives the number meaning.

1. Sample 50–100 items from your frozen evaluation set (09-01), stratified across the slices you care about — a judge can be accurate on easy items and useless on the hard slice.
2. Have two humans label them independently against the same rubric the judge receives, and compute human-human agreement (Cohen's kappa for two raters on nominal labels; Krippendorff's alpha for more raters or ordinal scales). Adjudicate disagreements into a gold label with a defined procedure (09-03).
3. Run the judge on the same items, with the same rubric text, and compute judge-human agreement against the gold label.
4. Compare the two coefficients. The judge is usable when its agreement with humans approaches the agreement humans achieve with each other. In the worked example, κ_JH = 0.496 against κ_HH = 0.604 — about 82% of the human ceiling, which is a reasonable basis for scaled measurement with continued human sampling.

Then look at two things beyond the scalar. The confusion matrix between judge and gold distinguishes a systematic leniency offset (correctable by moving the threshold) from random disagreement (needs a better rubric or a better judge). And length grouped by verdict, compared against the same grouping for humans, exposes verbosity bias in a single glance, as it did in §4 step 8.

Re-validate quarterly, or immediately after any model-version or rubric change. Validation is not a one-time gate; it is the maintenance schedule for a metric that can drift on its own.

Yes for tracking, no for adjudication — and the distinction is the practical answer.

Reliable enough for: direction of travel over time on a frozen set; comparing two configurations when the difference is large and you have order-balanced and paired the comparison; scaled coverage of rubric dimensions that no automatic metric can reach; and narrow evidence-grounded checks like "does this passage support this claim", which is the judge's strongest task.

Not reliable enough for: declaring a small improvement real without the sample-size arithmetic from 09-09; safety, legal, medical or financial sign-off; establishing ground truth for a new rubric; or any comparison where the judge shares a model family with a candidate.

Three practices raise reliability substantially and cost little. Dual-order evaluation with required consistency turns position bias from a hidden distortion into a reported number. A calibration set with known verdicts included in every run detects drift the moment it happens. Continued human sampling — a small labelled batch every quarter — keeps the judge honest and gives you the agreement figure that makes its scores interpretable. A judge maintained that way is a legitimate instrument. An unvalidated, unpinned, single-order judge scoring a vague rubric is a random number generator with good manners.

They are the same idea in different roles, and knowing the mapping is worth exam marks.

The shared property that matters: both are bounded above by the quality of the human labels behind them. A reward model trained on preference labels with 0.55 agreement is fitting substantial noise, and the policy optimisation stage will find and exploit whatever spurious regularity survives. A judge validated against the same labels inherits the same ceiling. This is why 09-03 sits early in this module — the agreement number is the calibration for the entire downstream stack, including alignment training two modules later.

The alignment pipeline order is worth stating precisely while the comparison is in view, because the exam asks for it: supervised fine-tuning first, then a reward model trained from human preference labels, then policy optimisation with PPO. DPO is the simpler alternative that optimises directly against preference pairs without training a separate reward model, and NVIDIA's SteerLM is another approach to steering a model with human feedback. 11-06 and 11-07 develop all of this; the point here is that the judge you build for measurement and the reward model built for training are the same construct, and both live or die by inter-annotator agreement.

1. A judge is a rubric with a model attached, and its whole purpose is to make rubric-based evaluation affordable at scale.
2. It is a proxy, never ground truth. Its quality is its agreement with human labels.
3. Report judge-human agreement relative to human-human agreement. κ_JH = 0.496 against κ_HH = 0.604 is usable; the same 0.496 against 0.85 is not.
4. Position bias can exceed the effect you are measuring. In the worked example, first position won 58.5% of the time while the real gap between systems was 3.5 points.
5. Always judge both orders and require agreement. Report the inconsistency rate — 25% in the example — as part of the result.
6. Verbosity bias is visible for free: group answer length by verdict and compare with the human grouping. A 66-word gap where humans show 6 is the diagnosis.
7. Never let a model judge its own family in a competitive comparison — self-preference bias makes it rigged.
8. Pin the model version, version the rubric prompt, and include calibration items in every run, or rubric drift will move your dashboard without anyone touching the system.
9. Judge one item per call, ask for the reason before the score, and demand structured output.
10. Pairwise beats pointwise for comparisons; pointwise is needed for gates and must have its threshold validated against humans.
11. A judge is not deterministic (09-11), so measure its run-to-run variance before believing small differences (09-09).
12. A judge and a reward model are the same construct in different roles, and both are capped by the inter-annotator agreement of the labels behind them.

Every measurement discipline in this module has quietly assumed that re-running an evaluation gives the same answer. It does not — and not because of anything you did wrong. Setting temperature to zero makes decoding greedy, which people reasonably read as "deterministic", and it is not the same claim. The gap between those two words has consequences for reproducibility, for CI gates that compare against a stored baseline, and for every judge score you just learned to validate.

Next: 09-11 explains exactly why greedy decoding is not reproducible — floating-point non-associativity, batch-dependent kernel reductions, hardware and library differences, and silent provider-side model updates — and what you can and cannot do to pin a generative system down.