Early Stopping and Measuring Fine-Tuning Impact Against a Baseline

Early stopping is a training-control rule: monitor a model's performance on a held-out validation set at regular intervals during training, and halt training at the point where that validation performance stops improving — rather than training for a fixed, predetermined number of epochs regardless of what the validation curve is doing. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the mechanism and its purpose together: early stopping "halts training when validation performance stops improving — a simple, effective guard against overfitting during fine-tuning."

The shape of the curve early stopping is watching for is consistent across most fine-tuning runs, and it has three phases:

• Early training: both training loss and validation loss (or its corresponding metric) improve together. The model is learning genuinely useful, generalizable patterns from the data.
• The validation optimum: validation performance reaches its best point. Training loss is typically still improving at this point, or about to keep improving, but validation performance has caught up to what the model's current capacity and the data actually support.
• Overfitting: training loss continues to fall — the model is fitting the training data ever more closely — but validation performance stalls and then gets worse. The model has started memorizing specifics of the training set (its particular phrasing, its particular examples' quirks) rather than continuing to learn anything that generalizes.

[GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) names the trap this curve produces directly: "more epochs isn't better — beyond the validation optimum you overfit. Early stopping captures the best generalizing checkpoint." The word "beyond" is doing real work in that sentence: it is not that more training is never good — the early phase of the curve is a real, useful improvement — it is that the benefit is bounded, and the exam's standing misconception is treating "more training" as a monotonically increasing good with no ceiling.

L1 — Intuition: studying versus memorizing the practice exam

Imagine a student preparing for an exam using one specific practice test as their only study material. Early in their preparation, working through the practice test teaches them the underlying subject — the concepts the practice test happens to be testing. If they stop there, they walk into the real exam able to apply what they learned to new questions they have never seen.

But if the student keeps re-taking the exact same practice test past the point where they have learned the underlying subject, something different starts happening: they begin memorizing that specific test's particular answers, its particular wording, its particular quirks — getting a perfect score on the practice test itself while learning nothing further that transfers to the real exam. Their practice-test score keeps climbing (that is the training loss, still falling), but their actual exam-day performance (validation performance) has stopped improving and may even get worse, because time spent memorizing test-specific quirks is time not spent reinforcing generalizable understanding, and some memorized quirks can actively mislead them on a real question that looks similar but is not identical.

Early stopping is the rule "stop practicing on this specific test once your real-exam performance, measured independently, stops improving" — not "stop after a fixed number of practice sessions," and not "keep practicing forever because more practice sounds like it should help."

L2 — Mechanism: checkpointing, a monitored metric, and a patience window

A practical early-stopping setup has three moving pieces. First, a monitored metric — typically validation loss, or a task-specific metric evaluated on a held-out set — measured at regular intervals during training (after every epoch, or after every fixed number of training steps). Second, checkpointing — saving the model's weights at each of these measurement points, so that if training continues past the best point, there is a saved copy of the model at that best point to return to. Third, a patience window — because validation metrics are noisy from one measurement to the next, a single-step plateau or a small dip is not necessarily the true optimum; a patience window says "if the monitored metric has not improved for N consecutive measurement points, stop training and roll back to the best checkpoint," which tolerates a small amount of noise without either stopping prematurely on a false plateau or training for a very long time past the real optimum trying to confirm it.

The output of this procedure is not "the model that exists when training loop finishes" — it is specifically the checkpoint saved at the validation optimum, which is very often an earlier checkpoint than the training run's final one. This is worth being explicit about because it means "training completed" and "training produced the best available model" are different claims, and only early stopping's checkpoint-selection step reconciles them.

L3 — The exam-relevant edge case: a falling training loss is not evidence of anything past the optimum

The professional-level distinction this domain rewards is narrower than "watch for overfitting." It is: once training loss and validation performance have diverged — training loss still falling, validation performance flat or worsening — the training loss curve has stopped being informative about the question that actually matters, which is how the model performs on data it was not trained on. A candidate who reports "training loss kept improving throughout the run" as evidence the fine-tune succeeded has reported a fact about the wrong curve. Training loss falling only tells you the model is fitting whatever it is currently being shown; it says nothing about whether that fit generalizes, and past the validation optimum, it specifically does not.

This is also why "train for more epochs" is never, by itself, a valid fix for a fine-tune that underperformed. If the run stopped before the validation optimum, more epochs may genuinely help. If the run already passed the validation optimum, more epochs make things worse, and the correct fix is rolling back to the best checkpoint, not continuing to train. Distinguishing which side of the optimum a given run is on is exactly what the monitored validation curve — not the training loss curve — is for.

Early stopping answers "when should this specific training run stop." Impact assessment answers a different, later question: "did this fine-tune, once complete, actually make the model better for the purpose it was fine-tuned for." [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) frames this as choosing "phase-appropriate metrics (e.g., loss during training, task metrics on validation) and assess the fine-tuning impact against a baseline before/after." Two distinct comparisons are named in that sentence, and conflating them is a common mistake: loss during training is a training-time signal, appropriate for early stopping's own monitoring; task metrics measured before and after, on a fixed baseline, are what actually answers the impact question.

A defensible impact assessment needs three pieces, all fixed before the comparison is trusted:

• A stated baseline, captured on the pre-fine-tune model, using the same evaluation set and the same metric that will be used post-fine-tune. Without this, "the fine-tuned model scores 82%" is a number with nothing to compare it against — 82% could be a large improvement or a regression, and there is no way to tell without the baseline.
• A fixed evaluation set, held constant across the before and after measurement, and ideally never touched during training or hyperparameter selection — using the same set the model was tuned against for its own impact assessment inflates the apparent improvement, because the model has had an opportunity to fit that specific set's quirks.
• Phase-appropriate metrics, per the source's own framing: training loss is the right signal for monitoring a training run in progress, but it is the wrong signal for the impact question, which needs a task-specific metric — accuracy, a preference win rate, a domain-specific score — measured identically before and after.

All figures below are a constructed scenario, not a measured benchmark of any specific fine-tuning run. A team fine-tunes a model for 10 epochs, checkpointing and measuring validation accuracy after each one.

Step 1 — the recorded curve.

Epoch:               1     2     3     4     5     6     7     8     9    10
Training loss:      1.20  0.95  0.78  0.65  0.55  0.47  0.40  0.34  0.29  0.25
Validation accuracy: 71%   78%   83%   86%   87%   87%   86%   85%   84%   83%

Step 2 — locate the validation optimum. Validation accuracy climbs through epoch 5, reaches its best value (87%) at epochs 5 and 6, and then declines steadily from epoch 7 onward, even as training loss keeps falling through epoch 10.

Best validation accuracy: 87%, first reached at epoch 5
Training loss at epoch 5:  0.55  (still falling, not yet at its lowest)
Training loss at epoch 10: 0.25  (the lowest of the run — and the worst
                                   validation accuracy of the run, at 83%)

Step 3 — apply a patience window to decide exactly when to stop. With a patience of 2 (stop if validation accuracy has not improved for 2 consecutive epochs), the run would halt after epoch 7, since epoch 6 tied epoch 5's 87% and epoch 7 is the second consecutive epoch without improvement — training then rolls back to the epoch 5 or epoch 6 checkpoint, whichever was saved as best, rather than keeping epoch 7's already-declining weights.

Step 4 — the checkpoint actually kept. The model shipped is the epoch 5 (or 6) checkpoint at 87% validation accuracy — not the epoch 10 checkpoint, despite epoch 10 having the lowest training loss of the entire run. A team that shipped "whatever the training loop produced when it finished" would have shipped the epoch 10 checkpoint: the worst-performing checkpoint on validation data of the whole run, dressed up in the most impressive-looking training loss number.

Step 5 — now assess impact, using a baseline captured before this run started.

Baseline (pre-fine-tune model), same evaluation set:  74% task accuracy
Fine-tuned model (epoch 5 checkpoint), same eval set:  87% task accuracy

Impact: +13 percentage points, measured on the same fixed evaluation set
before and after — this is the number an impact assessment reports, not
the raw 87% alone, and not the training loss at any epoch.

⭐ THE EARNED INSIGHT

The exam's likely framing treats early stopping as a straightforward overfitting-prevention rule, which it is. The professional-depth trap underneath that surface fact is conflating two different curves that happen to run during the same training job: the training loss, which can fall monotonically for the entire run and tells you only about fit to the training batch, and the validation metric, which is the one curve that actually has an optimum worth stopping at. A checkpoint chosen by "lowest training loss" and a checkpoint chosen by "best validation performance" are frequently two different checkpoints entirely — and section 5's worked example shows they can be at opposite ends of the same ten-epoch run.

All figures below are a constructed scenario, not a measurement of any real system. A team fine-tunes a customer-support model to answer product questions in a more concise style, using early stopping correctly: they monitor a validation set of held-out conciseness-labeled examples, and they stop at the true validation optimum.

Step 1 — the target metric, measured correctly, looks like a clear win.

Baseline (pre-fine-tune), conciseness score, held-out eval set:  58/100
Fine-tuned model, same conciseness metric, same eval set:        91/100

By the letter of the impact-assessment discipline this lesson has built — a stated baseline, a fixed evaluation set, a before/after comparison — this looks like an unambiguous success, and a team that stops here would report it as one.

Step 2 — the same before/after discipline, applied to a metric nobody thought to re-check.

Baseline (pre-fine-tune), factual-accuracy score, same eval set:  89/100
Fine-tuned model, same factual-accuracy metric, same eval set:    76/100

Step 3 — the mechanism behind the regression. The conciseness-focused training data, in this constructed scenario, disproportionately rewarded short, confident-sounding answers during training — and confident brevity is exactly the failure mode that drops supporting detail and caveats, some of which were carrying real information about edge cases the longer, pre-fine-tune answers used to include. Early stopping did its job correctly here: it stopped at the validation optimum for the metric it was told to monitor, conciseness. It was never monitoring factual accuracy, and it has no mechanism to protect a metric nobody pointed it at.

Step 4 — the discipline this scenario is built to demonstrate. Early stopping protects the monitored metric from overfitting. It makes no claim about any metric outside the one it is watching. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md)'s framing of impact assessment as comparing "phase-appropriate metrics ... against a baseline before/after" is the discipline that catches exactly this gap — but only if the before/after comparison is run on more than the single metric the fine-tune was optimized for. A team that assessed impact using only the conciseness score would have shipped a model with a real, measurable regression on factual accuracy, entirely undetected, despite having done early stopping and impact assessment both technically "correctly" on the metric they chose to look at.

Fine-Tuning is 13% of the NCP-GENL blueprint, tied for third-largest, and this lesson's subject is the domain's explicitly named guardrail against the whole module's central risk: every method covered elsewhere in this module changes weights, and every weight-changing method can be run for too long. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the trap in the same breath as the fix — "more epochs isn't better ... early stopping captures the best generalizing checkpoint" — which signals that a professional-level question is likely to present a training curve or a scenario describing "the training loss kept falling" and ask what that fact does and does not establish.

Expect this material in a few recurring shapes: a direct identification question ("what technique halts training when validation performance stops improving?"), a scenario naming a curve shape (training loss still falling, validation performance declining) and asking what to do, and a question testing whether "more training" or "a bigger model" is being offered as a fix for a problem actually caused by having already passed the validation optimum. The distractors in this domain's house style tend to offer a real, plausible-sounding lever — more epochs, a lower learning rate, more data — attached to a scenario where the actual, stated cause is that training already ran past its useful point, and the correct response is stopping and rolling back rather than pushing further.

What the distractors typically look like

Common wrong answers include: "increase the learning rate" or "add more attention heads" for a scenario whose real cause is overfitting past the validation optimum (real levers, wrong problem); "the training loss is still falling, so training should continue" as a description of legitimate progress when the described validation curve has already turned over; and reporting a single post-fine-tune metric as evidence of impact with no mention of what the baseline was, which the source's before/after framing directly rules out as a sufficient answer.

Because the model has a fixed amount of capacity and a fixed, finite training set, and past a certain point, continuing to reduce training loss requires fitting increasingly specific, non-generalizable quirks of that particular training set — patterns that exist in the training data by chance rather than reflecting anything true about the broader task. Fitting those quirks pulls the model's behavior away from what actually generalizes, which shows up as validation performance declining rather than merely leveling off. A plateau alone would mean the model simply stopped learning anything new; an actual decline means the model is actively trading generalization for training-set-specific fit, which is the mechanism overfitting describes.

No — the two serve different jobs and one does not substitute for the other. Early stopping guarantees you kept the best-generalizing checkpoint relative to the training run you actually did; it says nothing about whether that best checkpoint is actually better than the model you started with, on the metric that matters for deployment. A fine-tuning run could be early-stopped perfectly and still underperform its own pre-fine-tune baseline, if the training data or the chosen method was a poor fit for the task — which is exactly why [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) names impact assessment as its own, separate discipline, not a conclusion early stopping already proves.

• More epochs is not unconditionally better. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md): beyond the validation optimum, continued training overfits, and early stopping is the guard against exactly that.
• Training loss and validation performance can diverge, and once they do, training loss stops being informative about generalization — the validation curve is the one that has a real optimum to stop at.
• Early stopping's output is a checkpoint, not a training-loop completion — specifically the checkpoint saved at the validation optimum, which is frequently an earlier checkpoint than the run's final one.
• A patience window absorbs measurement noise, preventing both premature stopping on a false dip and prolonged training chasing an already-passed optimum.
• Impact assessment is a separate discipline from early stopping, requiring a stated before/after baseline on a fixed evaluation set — a single post-training number, with nothing to compare it against, does not answer whether a fine-tune helped.
• Phase-appropriate metrics matter: training loss is the right signal for monitoring a run in progress; a task-specific metric measured before and after is the right signal for the impact question.
• "More training" is never, by itself, a valid fix for a fine-tune that underperformed once the run has already passed its validation optimum — the fix is rolling back to the best checkpoint, not training further.

Early stopping and impact assessment answer how to know whether a specific fine-tuning run went well. They do not answer the earlier, larger question this module has been building toward across every method it has covered: whether a fine-tune was ever the right tool for the job in the first place. Next: M5-05 closes the module with that decision — when a new skill, style, or behavior genuinely needs to be baked into the weights permanently, versus when the same requirement is better served by prompting or retrieval instead.