Train, Validation, and Test Splits Explained

Typical proportions: 80/10/10 or 70/15/15. There is no official ratio; the driver is having enough validation and test data for the measurement to be stable. On a small dataset, k-fold cross-validation is preferred over a single split precisely because one small validation set produces a noisy estimate.

The rule that explains why three sets and not two: every time you use a set to make a decision, you contaminate it as an estimator. If you tune twenty hyperparameter configurations against your validation set and keep the best, that best score is optimistic — you selected it partly for fitting the validation set's noise. The test set exists to be the one measurement no decision was ever made against.

When it applies: every supervised training run, every fine-tune, every benchmark comparison, and — with an important adaptation covered below — every LLM application evaluation.

The permissions table, stated as rules you can check

The three-way split is really a permissions system, and it is easier to apply as a list of allowed and forbidden operations than as a diagram.

The last row is where discipline actually breaks. Seeing a bad test number and going back to tune is the single most common way a test set silently becomes a validation set. If you must do it — and sometimes the project requires it — the honest move is to say so, and to treat the next reported number as validation-grade until a genuinely fresh test set exists.

How many examples does each split need?

There is no official answer and no rule in the study guide, so reason about it rather than memorising a number. The constraint is measurement stability: the validation and test sets must be large enough that a real difference between two models is distinguishable from noise.

The intuition that transfers: with 100 test examples, one example is a full percentage point of accuracy, so a 2-point difference between two models is within the range that reshuffling the split could produce. With 1,000, one example is a tenth of a point. Small sets do not just give imprecise answers; they give answers that flip when nothing meaningful changed.

Practical shape, and this is reasoning rather than a sourced standard:

L1 — Two curves, four diagnoses

Plot training loss and validation loss against training steps on the same axes. The relationship between the two curves is the diagnosis, and it is one of the highest-yield pictures in machine learning:

The number to watch is the gap. A small gap means the model generalises. A widening gap means it is memorising the training set — its loss improves on data it has seen while getting worse on data it has not.

That fifth row deserves a note because people assume it is impossible. Validation loss below training loss can be benign — regularisation like dropout is active during training and off during evaluation, which can genuinely make the training number look worse — or it can be the signature of a validation set that is easier than the training set, or of leakage. Benign or not, it is worth a look rather than a shrug.

L2 — Why the gap opens, and early stopping

Training loss can always be pushed down: with enough capacity, a model can memorise its training data outright and drive training loss toward zero. Validation loss cannot be memorised into, because that data never enters the update. So the two curves separate at the point where the model stops learning generalisable structure and starts learning the training set's specific noise.

Early stopping is the direct exploitation of this: monitor validation loss, and stop when it stops improving — typically after a patience window of several evaluations with no improvement — keeping the checkpoint with the best validation loss rather than the last one. That is a regularisation technique whose entire mechanism is reading the gap.

The other standard responses to a widening gap, roughly in order of reliability:

That last row is the LLM-specific one and is worth carrying. Full fine-tuning a very large model on a small dataset is an efficient way to overfit; constraining the trainable parameter count with LoRA or adapters is both cheaper (01-06) and often better-behaved on limited data.

L3 — Leakage: the failure that produces flattering numbers

Data leakage is any path by which information from validation or test reaches the training process. It is dangerous specifically because its symptom is good results.

The recurring forms, each with a concrete example, the symptom you would see, and the fix:

Two rules prevent most of it. Split first, then fit every transformation on the training split only and apply it to the others. And split by group, not by row, whenever rows are related — by user, document, patient or time.

Benchmark contamination is specific to LLMs and is not fixable by careful splitting: when a model was pretrained on a web crawl, you generally cannot prove a public benchmark's items were absent from it. That is a live reason to distrust headline benchmark numbers and build your own private evaluation set.

Validation overuse is the subtlest entry and is worth naming because it is invisible in any single experiment. Each individual comparison against the validation set is legitimate. It is the accumulation — hundreds of decisions all made against the same held-out examples — that gradually fits the validation set's noise. This is why a genuinely sealed test set is not bureaucratic ceremony: it is the only measurement in the whole pipeline that has not been selected on.

The confusables table for this lesson is really a boundary table: for each pair of splits, what crosses, and what the crossing costs you.

Read the "what you lose" column as a hierarchy. Train-to-validation leakage costs you your diagnostic. Validation-to-test leakage costs you your report. Losing the diagnostic is worse, because you no longer know whether the model is memorising, so you cannot even choose a checkpoint sensibly.

And the pair most often conflated in language rather than in data: validation and test are not synonyms, even though "holdout" gets used for both. Validation is a working instrument you consult constantly. Test is a sealed one you open once. Any question that asks which split may be used for hyperparameter tuning is testing exactly this, and the answer is always validation.

Stratified k-fold preserves each class's proportion in every fold, which matters when classes are imbalanced; leave-one-out is the extreme case where k equals the number of examples. Cross-validation replaces the validation split, not the test split — a held-out test set is still opened only once. Cross-validation is the right answer for classical ML on limited data; a single held-out split is the right answer for anything involving pretrained LLMs.

The variants you should be able to name and place:

Note that the last two are not refinements for connoisseurs — they are the correct choice whenever grouping or ordering exists, and plain k-fold on grouped or time-ordered data is a leakage bug rather than a stylistic preference.

Why cross-validation is not used for LLM fine-tuning is worth a sentence, because it is a reasoning item rather than a fact to memorise: k-fold multiplies training cost by k, and for a large model each training run may be hours to days of GPU time. The economics only work when training is cheap relative to the value of a more stable estimate. For classical ML on a few thousand rows, that trade is obviously good. For a fine-tune of a 70B model, it is obviously bad.

Split discipline is named in three places at once. Objective 1.5 lists cross validation among the ML fundamentals you must be familiar with. The Experimentation domain — 22% of the blueprint, roughly 13 of 60 questions — is defined as "how to perform, evaluate, and interpret experiments," which is split discipline by another name. And the Data Analysis objectives ask you to "identify relationships and trends or any factors that could affect the results of research," where leakage is the canonical such factor.

One documented oddity worth knowing so it does not confuse you: in the official study guide, the Experimentation domain's printed objectives 3.1–3.5 are a verbatim duplicate of the Data Analysis objectives 2.1–2.5, describing data analysis rather than experimentation. The domain's own scope statement still names model evaluation and RLHF, so the real coverage is broader than the printed list. Do not conclude from the duplicated text that evaluation is untested — it is 22% of the exam.

How the question tends to be phrased

• Reading a curve. "Training loss continues to decrease while validation loss begins to increase. What is happening?" Keyed: overfitting. This is close to a guaranteed appearance in some form.
• Naming the defect from a symptom. "A model scored 99% during evaluation but 60% in production. What is the most likely explanation?" Keyed: data leakage, or a distribution mismatch between evaluation and production.
• Permission questions. "Which dataset should be used to select hyperparameters?" Keyed: validation. "Which should be used only once?" Keyed: test.
• Cross-validation identification. What k-fold is, why it produces a more stable estimate than a single split on limited data, and what stratification preserves.
• Mitigation matching. Overfitting → regularisation, dropout, early stopping, more data, less capacity. Underfitting → more capacity, longer training, better features.
• Split-strategy selection. "The dataset contains multiple records per customer. How should it be split?" Keyed: by customer, so no customer appears in more than one split.
• Temporal reasoning. "The data spans two years and the model will forecast next quarter. How should it be split?" Keyed: chronologically.

What the distractors typically look like

Four families. Permission swaps: "tune hyperparameters on the test set" or "use the training set to decide when to stop." Direction confusion on curves: offering underfitting where the described curves show overfitting, which is defeated by remembering that underfitting has both losses high and close. Plausible-but-wrong mitigations: "train for more epochs" offered as a fix for overfitting, or "add more layers" where regularisation belongs. Split-method traps: a random split offered for time-series or grouped data.

The underfitting-versus-overfitting swap is the highest-value one to be airtight on, because both are described with the word "poor performance" and the discriminator is entirely in the gap. Underfitting: both losses high, gap small. Overfitting: training loss low, validation higher and rising, gap widening. Two sentences, and a whole item family becomes automatic.

Calibration note, and it is [FIELD] calibration rather than official fact: published candidate reports place the exam at a general level and do not report deep statistical derivations. NVIDIA publishes no item-level detail, no official passing score, and its own page is ambiguous about the question count. Plan for recognition-speed fluency on the vocabulary and the diagnostic table rather than for depth.

A team fine-tunes a model for support-ticket classification on 10,000 labelled tickets. They split 80/10/10 at random: 8,000 train, 1,000 validation, 1,000 test. The numbers below are an illustrative construction.

Reading the curves.

epoch     train loss   val loss    gap
  1          0.68        0.71      0.03
  2          0.41        0.48      0.07
  3          0.24        0.39      0.15    ← best validation loss
  4          0.12        0.44      0.32
  5          0.05        0.58      0.53

Validation loss bottoms out at epoch 3 and then climbs while training loss keeps collapsing toward zero. The gap goes from 0.03 at epoch 1 to 0.53 at epoch 5. The correct action is to keep the epoch-3 checkpoint, not the epoch-5 one — the model with the worse training loss is the better model. Epochs 4 and 5 bought memorisation.

Convert the training losses to the perplexity intuition from 01-05 and the memorisation is even starker: a training loss of 0.05 means the model is nearly certain of its answer on data it has already seen, while simultaneously getting less certain on data it has not. That divergence is the definition of the failure.

Then the leak. They report 94% test accuracy and ship. Production accuracy is 71%.

The audit finds two causes. First, many tickets were near-duplicates — the same customer issue filed repeatedly — so a random row-level split put copies of the same ticket in both training and test. Fixing it means splitting by customer, not by row. Second, the tickets were labelled over eighteen months during which the product changed, and a random split let the model train on later tickets and be tested on earlier ones. The honest split is chronological: train on the first fifteen months, test on the last three, which is also what production actually asks of the model.

Re-split correctly, test accuracy comes in at 74% — close to production, and now a number that can be trusted for decisions. The lesson is not that the model got worse. The model never changed. Only the honesty of the measurement did.

The same team, three months later: a second, subtler failure

Worth following the story, because the second failure is the one that catches experienced teams.

With honest splits in place, the team runs experiments. Over three months they try 60 configurations — different learning rates, different LoRA ranks, three base models, several prompt formats — comparing each against the validation set and keeping the best. The best validation accuracy climbs from 74% to 82%.

They open the test set. It reads 76%.

Nothing leaked in the data sense. The splits were clean, grouped by customer, ordered chronologically. What happened is validation overuse: 60 selection decisions made against 1,000 validation examples gradually selected for configurations that suited that specific set's noise. The 82% was a selected maximum, not an estimate, and selected maxima are optimistic by construction.

Two takeaways, both directly exam-relevant. First, this is exactly why the third split exists — the test set caught an 6-point optimism that no amount of care in the data pipeline would have. Second, the correct interpretation is not that the work was wasted: 76% still beats the 74% baseline honestly. The correct interpretation is that the validation number was never the number, and treating it as one would have over-promised to stakeholders by six points.

Split strategy is a judgement, and the exam tests it as recognition. Four cases, with the reasoning made explicit.

Case 1 — 50,000 independent product reviews, one per customer, no time trend. Random 80/10/10 is correct. Rows are independent, nothing is grouped, nothing is ordered. Stratify by rating if the classes are imbalanced.

Case 2 — 8,000 medical records from 900 patients. Group split by patient. A random row split would place different visits from the same patient on both sides, and the model would learn patient-specific quirks that inflate the score. Note the second-order consequence: with 900 groups rather than 8,000 rows, your effective sample size for the split is 900, so the splits are noisier than the row count suggests.

Case 3 — Two years of daily sales, forecasting next month. Chronological split, and cross-validation must be the time-series variant where each fold trains on a prefix and validates on the following period. A random split lets the model see the future, which is both leakage and a task the model will never actually face.

Case 4 — 300 hand-written prompts and expected answers for an LLM assistant. No training is happening, so there is no train split. What you need is an evaluation set held stable over time, ideally in two parts: a working set you inspect while iterating on prompts, and a sealed set you check before shipping. That is the 01-08 pattern, and it is the shape most LLM application work actually takes.

The unifying question, which is worth carrying into any scenario item: what does the model face in production, and does my split simulate it? Production shows the model unseen customers, so split by customer. Production shows it future dates, so split by time. Production shows it novel prompts, so hold prompts back. Every split rule in this lesson is that question applied to a specific structure in the data.

No. It means the model is consistent, which is a different claim.

A small gap tells you one specific thing: performance on unseen data resembles performance on seen data. It is entirely compatible with both losses being terrible, which is underfitting — a model that predicts the majority class for everything has an essentially zero gap and no value.

It is also compatible with both losses being good and the model still being unfit for production, in three ways worth naming. The splits may share a bias the real world does not: if your labelled data over-represents one customer segment, both splits inherit that and agree with each other while disagreeing with reality. The metric may not be the thing you care about: low cross-entropy is not the same as helpful, safe, correctly formatted output (01-05). And the world may move: a model consistent across splits drawn from last year is not thereby consistent with next quarter.

The correct reading of the gap is therefore diagnostic rather than evaluative. The gap tells you about the fit. The absolute numbers tell you about the quality. Production tells you about the value. Three separate questions, and only the first is answered by the gap.

Practically, yes — projects run for years and eventually every set gets examined. The professional question is not whether it happens but what you owe the reader afterwards.

Once a test set has informed a decision, it stops being an unbiased estimator. The score it produces afterwards is a selected score, like a validation score. Nothing catches fire; you have simply spent the instrument. What is not acceptable is continuing to report the number as if it were unbiased, because that is the point at which a methodological compromise becomes a misleading claim.

Three practices keep this honest, and they are what a senior reviewer will look for:

Report the number of times. "Test accuracy 76%, third look after two rounds of tuning" is a defensible sentence. "Test accuracy 76%" alone implies something stronger than you have.

Reserve capacity up front. If the project will plainly need several honest checkpoints, carve out several sealed sets at the start rather than re-using one. Fresh data is the only thing that genuinely restores unbiasedness.

Escalate to a real holdout for the decision that matters. A staged rollout or an online A/B test measures the thing offline splits only approximate, and it is not subject to any of this — which is why the Experimentation domain covers both offline and online evaluation rather than treating splits as the end of the story.

They produce similar-looking disappointment in production and have opposite locations, which is why they get conflated.

Overfitting is a property of the model relative to the training data. The model learned the training set's noise as if it were signal. The diagnostic is internal and visible: the train-validation gap widens. Nothing is wrong with your data pipeline. The fixes are all model-side — early stopping, regularisation, less capacity, more data.

Leakage is a property of the data pipeline. Information that should have been unavailable crossed a boundary. The diagnostic is not visible in the gap — leakage typically makes the gap look reassuringly small, because the validation set is no longer measuring generalisation. The fixes are all pipeline-side: deduplicate, split by group, split by time, fit transformations on train only.

The decisive discriminator: a large test-to-production gap with a small train-to-validation gap points at leakage, not overfitting. That single sentence resolves the most common diagnostic scenario item on this topic, and it is worth memorising verbatim.

• Three disjoint sets, three permissions: train updates weights, validation guides decisions, test is opened once.
• Typical ratios are 80/10/10 or 70/15/15, driven by needing stable measurements, not by any official rule.
• The train/validation loss gap is the overfitting read-out. Falling training loss with rising validation loss is the signature; both losses high and close is underfitting.
• Early stopping keeps the best-validation checkpoint, not the last one — the model with the worse training loss usually generalises better.
• Leakage produces good numbers, not errors. Split first, then fit transformations; split by group and by time, never by row alone.
• Leakage narrows the train/validation gap while widening the test-to-production gap — the opposite signature to overfitting, and the fastest way to tell them apart.
• Validation overuse is real leakage even when the data pipeline is clean: many decisions against one set make its best score optimistic.
• k-fold cross-validation stabilises estimates when data is scarce and training is cheap; a single split is standard for LLMs because k training runs are not affordable.
• Use stratified folds for imbalanced classes, group folds for related rows, time-series splits for ordered data.
• Benchmark contamination is the LLM-specific form of leakage and is why a private evaluation set matters.
• The unifying test for any split strategy: does it simulate what production will ask of the model?

Splits assume you have labelled data with correct answers already. For most LLM applications you do not — nobody handed you the right output for "summarise this contract." Before you can measure anything at all, you have to write the ground truth yourself, and doing that early and crudely is worth more than doing it late and elaborately.

Next: 01-08 How to build an evaluation set for an LLM project — starting with roughly twenty hand-written examples you can grade by eye, and why that beats waiting for a proper benchmark.