Scaling an LLM Evaluation Set to 100 Items: Stratification, Sampling, and Freezing

An evaluation set is a fixed, versioned collection of inputs paired with either a reference output or a gradeable criterion, used to score a system repeatedly over time. At a hundred items, three properties become achievable that were not achievable at twenty:

1. Slice resolution. With a hundred items you can carve out five slices of twenty and still say something about each — coarsely, but honestly. With five slices of four, you cannot.
2. Movement detection. One item is one percentage point. A three-point move is three items, which is at least a plausible signal rather than certain noise. The arithmetic of how much movement is real is the subject of 09-09; for now, take it that a hundred items lets you begin that conversation.
3. Failure-mode coverage. You can afford to reserve quota for rare-but-serious cases — the adversarial prompt, the out-of-scope question, the document with a table in it — instead of hoping they show up by chance.

A hundred-item set is not a statistical instrument in the way a thousand-item benchmark is. It is a decision instrument: enough resolution to choose between two candidate configurations, enough coverage to notice a regression you would otherwise ship, and small enough that a human being will actually re-label it when the rubric changes.

The three components of every item are worth naming precisely, because exam questions about evaluation-set design usually turn on whether you can distinguish them:

The metadata column is the one beginners drop and the one that makes a hundred-item set worth five times a twenty-item set. Slice labels are what convert a single number into a diagnosis.

L1 — Intuition: you are buying resolution, not size

Think of the evaluation set as a camera sensor. Twenty items is twenty pixels: enough to tell light from dark, useless for reading a word. A hundred items is a hundred pixels — still a blurry image, but now you can see that the top-left corner is dark while the rest is bright. The point of going from twenty to a hundred is not that the average gets more accurate (it does, a little); it is that the image acquires structure. You buy the ability to localise the failure.

That reframing tells you how to spend the eighty new items. You do not spend them on more of the same. You spend them where you currently have no visibility.

L2 — Mechanism: quota, sample, label, freeze

The construction procedure has four stages, and they run in this order.

Stage 1 — Define the slices and set quotas. A slice is any partition of inputs you have a reason to care about separately. Common slice dimensions for an LLM feature:

• Input type: short question, long question, multi-part question, document-grounded question, no-context question.
• Source format: clean prose, PDF-with-tables, transcript, code, mixed language.
• Intent: factual lookup, summarisation, comparison, calculation, out-of-scope, adversarial.
• Population: user segment, geography, language, tenant — the slices that matter for fairness as well as quality.
• Expected answerability: answerable from the corpus, partially answerable, unanswerable.

You will not slice on all five dimensions at once; a hundred items cannot support thirty cells. Pick the two dimensions with the most decision value and allocate quota across them. The last bullet is the one most often forgotten and the one that catches the most damage: if a hundred-item set contains zero unanswerable questions, it cannot measure whether your system says "I don't know", and a system that never abstains will confabulate in production. 07-11 is where abstention behaviour is taught; the evaluation set is where it is measured.

Stage 2 — Sample, mostly from reality. Three sources, in descending order of value:

If you have no traffic yet, that ratio inverts and you accept it, while writing down that the set is synthetic and therefore optimistic. Saying so in the set's own README is a discipline that costs nothing and prevents a false confidence you will otherwise carry for months.

Stage 3 — Label once, against a written rubric. For each item, record either a reference answer or the criterion by which a response passes. This is where most evaluation sets rot. A criterion like "answer is good" is not a criterion; two people applying it will agree less often than they expect, and the same person applying it in March will disagree with themselves in June. Write the rubric as observable conditions — "names the correct policy document", "gives a figure within ±2%", "does not assert a date not present in the context" — and record it with the item. Rubric design and how to measure whether two labellers agree is the whole subject of 09-03.

Stage 4 — Freeze and version. Assign the set a version (evalset-v2, or a content hash), store it in the repository next to the code, and stop editing it. Adding an item mid-quarter silently changes the denominator; removing an item you find embarrassing is score laundering. When the set genuinely needs to change, you cut a new version, re-run the current champion configuration on the new version to establish the new baseline, and record both numbers. This is the discipline that makes 10-04 — regression testing in CI — possible at all.

L3 — Depth: the arithmetic that justifies exactly a hundred

Why a hundred rather than fifty or five hundred? Two numbers bracket the answer.

The lower bracket is standard error. For a pass/fail metric, the standard error of an observed proportion p on n items is sqrt(p*(1-p)/n). At the worst case p = 0.5:

At twenty items your 95% interval is ±22 points, which is wider than most real improvements. At a hundred it is ±10 points — still wide, but you can now detect the large effects that early-stage work produces. Note the shape: error falls with the square root of n, so halving your uncertainty costs four times the labelling. Going from 100 to 400 to shrink ±10 into ±5 quadruples the human cost. That diminishing return is exactly why a hundred is a natural resting point for a team that labels by hand.

The upper bracket is labelling throughput. A careful human label on a generative output takes somewhere between thirty seconds (does the answer contain the right figure?) and five minutes (is this summary faithful to a four-page document?). A hundred items at two minutes each is a bit over three hours: one afternoon, one person, repeatable. Four hundred items at two minutes is two and a half days, which in practice means it gets done once and then never again. An evaluation set nobody re-labels is a snapshot, not an instrument. The right size is the largest one your team will genuinely re-run.

There is a third consideration that pushes the same way. A hundred items is small enough that you can read all the failures. Error analysis — the subject of 09-13 — requires a human to look at every wrong answer and name the cause. At a hundred items with an 80% pass rate, that is twenty failures to read, which is an hour. At a thousand items it is two hundred failures, which nobody reads, so the metric becomes a number on a dashboard instead of a source of fixes.

These four artefacts are constantly confused, including in exam distractors. They differ in who made them, what they are for, and what it means when a score on them goes up.

Three consequences follow that are worth memorising as decision rules.

• A test split answers "does the model generalise?"; an evaluation set answers "does the product work?" They are not substitutes. A model with an excellent test-split F1 can produce a terrible product if the split's distribution does not match live traffic. Split hygiene is 01-07's subject; cross-validation, the more careful use of splits, is 09-08's.
• A public benchmark can be contaminated by construction. Once a benchmark's questions and answers are on the public web, they are candidates for inclusion in the next pretraining run, and a model that has memorised the answer key scores well without possessing the capability. That is why your private hundred items has an evidential property no public benchmark can have. 10-01 treats contamination in full.
• A regression suite is monotone by intent. You never expect it to improve; you require it not to degrade. Mixing regression items into your headline evaluation score inflates that score over time, because you keep adding items you have already fixed.

A support-assistant feature answers customer questions over a product-documentation corpus. Six months of logs are available. We want a hundred items.

Step 1 — choose two slice dimensions. From incident history, the two dimensions with the most decision value are intent (what the user is trying to do) and answerability (whether the corpus can support an answer). Format was a third candidate but the corpus is uniform HTML, so it is dropped.

Step 2 — allocate quota. Traffic mix from logs is 62% factual lookup, 18% how-to/procedural, 9% comparison, 6% billing/account (out of scope for this corpus), 5% other. Naively proportional allocation gives 62 factual items and 6 out-of-scope items. We deliberately over-sample the rare-and-serious and under-sample the abundant-and-easy:

Write down the deviation and its reason. A stratified set is not an unbiased estimate of live quality — you have deliberately made it harder than reality. That is a feature, but only if you never quote its aggregate as "our production accuracy". Two numbers, clearly labelled, is the correct output: a stratified score for decisions and a traffic-weighted score for reporting.

Step 3 — recover the traffic-weighted number. Suppose after a run the per-slice pass rates are:

• Unweighted (stratified) score: 74 passes / 100 items = 74.0%.
• Traffic-weighted score: sum of w × rate over the six in-traffic slices = 0.5735 + 0.1350 + 0.0525 + 0.0257 + 0.0333 = 0.8200, and the traffic weights of those five in-traffic slices sum to 0.62 + 0.18 + 0.09 + 0.06 + 0.05 = 1.00, so the weighted estimate is 82.0%.

Eight percentage points separate the two numbers, and both are correct answers to different questions. "How good is the product for a typical user?" → 82%. "How good is the product across the situations we consider risky?" → 74%. A report that quotes one without naming which it is, is not a report.

Step 4 — read the diagnosis, not the score. The interesting content of that table is not 74%; it is that out of scope passes at 42.9% while factual passes at 92.5%. The system is good at answering and bad at declining. That is a single, specific, fixable finding, and it is only visible because fourteen items were spent on a slice that is 6% of traffic. Proportional sampling would have given six items, of which two or three failing would have been indistinguishable from noise.

The general rule: resolution is set by the count inside the slice you are trying to read, not by the total. Every time someone reports "we evaluated on 500 items" and then draws a conclusion about a subgroup of nine, that rule has been broken.

The NCA-GENL blueprint puts 22% of the exam in the Experimentation domain, whose official scope statement reads: "The study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)." Evaluation-set construction is the first practical activity that scope names.

Cite the objective numbering carefully. There is a verified defect in the official study guide: the objectives printed under Experimentation as 3.1–3.5 are a verbatim duplicate of the Data Analysis objectives 2.1–2.5. Read literally, they describe data mining, charts, and data analysis — not model evaluation and not RLHF. The Experimentation section's own scope statement and its suggested-reading list (A/B testing, zero-shot testing, GLUE, evaluating RAG applications, cross-validation, hallucinations, machine-translation evaluation) both describe evaluation work, and published candidate reports independently confirm that BLEU, hallucination mitigation, and RLHF appear on the exam. So the derived scope governs. For this lesson, the objectives that genuinely apply are 2.2 / 3.2 ("compare models using statistical performance metrics, such as loss functions or proportion of explained variance" — the duplicated pair), 2.3 / 3.3 (conduct data analysis under supervision), 2.5 / 3.5 (identify factors that could affect the results of research), and 1.5 (familiarity with ML fundamentals including model comparison and cross-validation). Do not memorise the printed 3.x text as a description of this domain; memorise the scope statement.

Question phrasings you should expect:

• "A team evaluates a summariser on 15 hand-written examples and reports a 7-point improvement. What is the most significant weakness of this conclusion?" → The sample is too small for the claimed effect; the interval around a 15-item proportion is far wider than 7 points.
• "Which of the following most improves the diagnostic value of an evaluation set?" → Stratifying items across input slices and recording slice labels. Distractors: increasing temperature, adding more items of the same kind, using a larger model as a reference.
• "A team adds new items to its evaluation set each sprint and tracks the score over time. What problem does this introduce?" → The denominator changes, so the time series is not comparable; scores must be tied to a frozen, versioned set.
• "Why should an evaluation set include questions the corpus cannot answer?" → To measure abstention/refusal behaviour and detect confabulation.

Distractor families to recognise. The exam's evaluation questions cluster around four wrong instincts: (1) bigger model instead of better measurement — an option that proposes upgrading the model when the question is about measurement validity; (2) more data instead of stratified data — options that add volume without addressing coverage; (3) public benchmark instead of task-specific set — plausible-sounding options that recommend GLUE or MMLU to validate a domain product; (4) automate the judgement before defining it — options that reach for an LLM judge before a rubric exists, which 09-10 shows just moves the ambiguity somewhere less visible.

The last one deserves emphasis because it is the subtlest. Every time you look at a failure, change a prompt, and re-run, you leak a little information from the evaluation set into your design. Over fifty iterations you have effectively fitted your prompt to a hundred specific items. This is the same phenomenon as overfitting a model to a training set, and the defence is the same: hold something back.

Enough that the count inside the slice you want to read supports the size of the effect you want to detect. As a working ladder: 20 items to prove the harness runs, 100 items to choose between configurations with double-digit differences and to localise failures by slice, 400+ items to resolve single-digit differences, and thousands only when the metric is fully automated and the decision is high-stakes. Because standard error shrinks with the square root of n, each halving of uncertainty costs four times the labelling — which is why a hundred, labelled well and stratified deliberately, beats a thousand labelled carelessly. If you can only afford one of "more items" and "better slice coverage", buy slice coverage.

Mostly from logs, with hand-written items filling the gaps logs cannot reach. Real queries carry the distribution you actually serve: the one-word searches, the typos, the questions phrased as statements, the half-finished sentences. Hand-written items are indispensable for three specific cases — features with no traffic yet, adversarial inputs you hope never to see in logs, and quota cells your current users happen not to exercise. Sampling from logs brings obligations, though: strip or pseudonymise personal data before the item enters the repository, respect the consent and retention rules covered in 13-05, and record the sampling date so you can tell later whether the set has drifted away from live traffic.

Refresh on an event, not on a calendar. The events that justify a version cut are: the product's scope changes (a new document type, a new intent, a new language); the traffic distribution shifts enough that your slice quotas no longer resemble reality; the rubric changes, because a rubric change silently re-defines every label; or the set has been iterated against so many times that you suspect you have fitted to it. When you cut a version, always re-run the current champion on the new version and record both scores side by side — otherwise the version boundary looks like a quality change. In between cuts, resist the urge to tidy. A slightly stale set that has held still for a quarter produces a comparable time series; a continuously improved set produces an uninterpretable one.

A stratified score is the plain average over a set whose composition you chose for diagnostic value, deliberately over-representing rare and serious cases. A traffic-weighted score re-weights each slice's pass rate by that slice's real share of production traffic to estimate what a typical user experiences. In the worked example above the same run scored 74.0% stratified and 82.0% traffic-weighted. Neither number is wrong; quoting either without saying which it is, is. Use the stratified score for engineering decisions — it is where the signal is — and the traffic-weighted score for anything anyone outside the team will read.

1. A hundred items buys resolution, not accuracy. The point is to see where the system fails, which twenty items structurally cannot show.
2. Resolution is set by items-in-slice, never by the total. A 500-item set says nothing about a nine-item subgroup.
3. Stratify deliberately and over-sample the rare-and-serious. Then report the stratified score and the traffic-weighted score as two separate, labelled numbers.
4. Sample 50–70% from real traffic. Author-invented inputs test the author, not the product.
5. Reserve 10–15% of quota for unanswerable and out-of-scope inputs, or abstention behaviour goes unmeasured and confabulation goes unpunished.
6. Freeze and version. A growing set produces an uninterpretable time series; deletions are score laundering.
7. Standard error falls as sqrt(n). ±22 points at n=20, ±10 at n=100, ±5 at n=400 — each halving costs four times the labelling.
8. Pick the largest set your team will genuinely re-label. An instrument nobody re-runs is a snapshot.
9. Hold something back. Fifty iterations against the same hundred items is overfitting your prompt to your evaluation set.
10. On the exam, treat the printed 3.1–3.5 objective text as defective and answer from the Experimentation scope statement — model evaluation and RLHF — instead.

You now have a hundred items and a way to score them, but every score so far has come from a human reading an output or from exact match against a reference. Both are expensive, and neither one tells you anything about the model itself in the absence of a reference answer. There is one number you can compute from a model and a piece of text alone, with no labels at all — and it is the number most often quoted and most often misread in the whole field.

Next: 09-02 explains perplexity: exactly what it measures, why a lower number is not automatically a better product, and the specific claims about model quality it cannot support.