Reading Loss Curves: Diagnosing a Training Run from Its Graph

A loss function returns a single number measuring how wrong the model's predictions are on a batch of data. For language models that number is almost always cross-entropy — the negative log probability the model assigned to the tokens that actually appeared, averaged per token. 01-05 derives it. Lower is better, zero is perfect, and the value is on a log scale in disguise, which is why a drop from 3.0 to 2.0 is an enormous improvement and a drop from 0.30 to 0.29 is often noise.

Two lines get plotted:

• Training loss is computed on the batches the optimizer is currently learning from. It is measured while the weights are being updated, so it is a running commentary on data the model is actively memorizing. It is expected to fall.
• Validation loss is computed periodically on a held-out split the optimizer never sees, with the model in evaluation mode and gradients off. It is the estimate of generalization. It is the line that matters.

The distinction is the whole reason 01-07 insists on three splits. Training loss measures fit; validation loss measures fit to unseen data of the same kind; the test set, touched once at the end, measures whether your validation-driven decisions themselves overfitted. A curve plotting only training loss is close to uninformative for diagnosis — it will descend beautifully while the model rots.

Three related quantities show up on the same dashboards and should not be confused:

That last row is the most important caveat in the lesson. Loss and task quality usually agree, and sometimes they diverge — a model can have slightly worse validation loss and better downstream behaviour, or vice versa. Loss is a proxy. It is a fast, free, high-resolution proxy that you read every run, and it is not the acceptance criterion. The acceptance criterion is your eval set.

L1 — Intuition: four questions, asked in order

Every loss-curve diagnosis reduces to four questions, and asking them in this order stops you from misreading the graph:

1. Is there a NaN or an explosion? If loss went to nan, inf, or jumped by orders of magnitude, stop. Nothing else in the graph means anything. This is a numerical failure, not a learning failure.
2. Is the training loss going down at all? If not, the problem is upstream of generalization — a learning rate near zero, a frozen graph, a broken data pipeline, a label bug.
3. Are the two lines converging or diverging? Converging and both falling: healthy. Diverging with validation rising: overfitting. Both flat at a high value: underfitting.
4. Where is the validation minimum? That step, not the last step, is the checkpoint you want.

L2 — Mechanism: the canonical shapes and what causes each

Why overfitting produces that shape. Early in training the model learns structure that is genuinely present in the language — grammar, common facts, the shape of the task. Those patterns generalize, so both curves fall together. Later, once the general structure is exhausted, the cheapest remaining way to reduce training loss is to memorize idiosyncrasies of the specific training examples. Those idiosyncrasies do not appear in the validation split, so memorizing them lowers training loss and raises validation loss. The minimum of the validation curve is the moment the marginal pattern being learned stops generalizing.

The train–validation gap as a number. Report it explicitly: gap = validation_loss - training_loss. A small, stable gap is healthy. A gap that grows monotonically is overfitting in progress even if validation loss has not yet turned upward — a growing gap is the early warning and the upturn is the late one. The gap's absolute size is domain-dependent and not comparable across tasks; its trend is what you read.

Early stopping is the operational response: track validation loss, keep the best-so-far checkpoint, and stop when it has failed to improve for a set number of evaluations (the patience). Two implementation details cause real bugs. First, you must save the best checkpoint, not the last one — stopping without checkpointing throws away the model you were trying to keep. Second, patience must be measured in evaluations, and if you evaluate rarely you will overshoot the minimum by a lot.

L3 — Fine-tuning curves, LLM-specific reading, and what loss cannot tell you

Fine-tuning curves look different from pretraining curves and get misread constantly.

Pretraining runs one pass, or barely more, over an enormous corpus. Training loss and validation loss stay close because the model never sees an example twice — there is little to memorize. The curve is a long, smooth, slowly flattening descent, and the interesting features are learning-rate warmup at the start, occasional loss spikes, and the shape of the decay schedule.

Supervised fine-tuning runs several passes over a small dataset. Overfitting is the default outcome, not an exception. It is entirely normal for validation loss to reach its minimum inside the first epoch and rise thereafter, and the practical consequence is that fine-tuning is usually a 1-to-3-epoch activity, not a 20-epoch one. If your fine-tune's validation minimum is at step 200 of 5,000, the correct read is not "training failed" — it is "you needed 200 steps, and you now have the checkpoint."

Catastrophic forgetting is the failure that loss curves conceal. Fine-tuning loss on your new task can fall beautifully while the model quietly loses general capability it had before, because your validation split contains only the new task. The loss curve is blind to this by construction. The only detection is an eval slice that covers the old capabilities — the argument 11-03 makes at length.

LoRA and PEFT curves. With only adapter parameters trainable, the effective capacity is much smaller, so the curve typically descends less far and overfits more slowly. Loss values are not comparable between a LoRA run and a full fine-tune of the same model; compare each against its own baseline and against the eval set.

Mixed-precision artifacts. A BF16 run often shows a marginally noisier curve than FP32, which is expected from the reduced mantissa and is not a defect — 12-01 explains why. An FP16 run that produces repeated skipped steps or a sudden NaN is a loss-scaling problem, and BF16 is the standard remedy. When comparing two runs, precision mode belongs in the experiment record alongside seed and library version, because a curve difference caused by TF32 being enabled in one run is an infuriating thing to debug.

Distributed-training artifacts. In data-parallel training the loss you log is typically the loss on one rank's local batch unless you explicitly reduce across ranks; a curve that looks noisier than expected may simply be under-averaged. And the effective batch size is per_device_batch × devices × gradient_accumulation_steps, so a run that "changed nothing but the GPU count" changed the effective batch size and therefore the appropriate learning rate. 12-04 covers the synchronization that makes this true.

What a loss curve cannot tell you: whether the model is factually accurate, whether it hallucinates, whether it is safe, whether it produces valid JSON, whether it forgot something, whether it is biased, or whether it will be fast enough. Every one of those needs a separate instrument. Loss tells you whether optimization worked. It is necessary and nowhere near sufficient.

And the two adjacent shapes that are frequently mistaken for pathologies:

A constructed scenario. You are fine-tuning a 7B instruction model on 4,000 support-ticket summarization examples. Configuration: 3 epochs, per-device batch 4, gradient accumulation 4, 2 GPUs, BF16, LR 2e-5 with 30 warmup steps and cosine decay. Validation split: 400 held-out examples, evaluated every 50 steps.

Step 1 — how many steps is this run?

effective batch = 4 (per device) × 4 (accumulation) × 2 (GPUs) = 32 examples per step
steps per epoch = 4000 / 32                                    = 125 steps
total steps     = 125 × 3                                      = 375 steps
evaluations     = 375 / 50                                     = 7 evaluations (plus step 0)

Seven validation points across the whole run. That is the first finding, and it is a design flaw: with a validation minimum somewhere in a 375-step run, evaluating every 50 steps localizes it only to ±50 steps. For a short run, evaluate every 25 steps or every 10% of total steps, whichever is finer.

Step 2 — the logged numbers. Constructed, illustrative:

Step 3 — the diagnosis, using the four questions.

1. No NaN, no explosion. The run is numerically sound.
2. Training loss fell from 2.41 to 0.26. Optimization is working; the learning rate is not broken.
3. The lines diverge from step 100 onward and the gap grows monotonically from +0.03 to +1.37. Textbook overfitting.
4. Validation minimum is at step 150, value 1.24 — barely into epoch 2 of a 3-epoch run.

Step 4 — what to do about it. Deploy the step-150 checkpoint, not the step-375 one. If checkpointing was set to "last only," you have thrown the good model away and must re-run — which is why save_best and a validation-loss-driven checkpoint selection are non-negotiable configuration, not niceties.

Step 5 — quantify the mistake you almost made. The final-step model has training loss 0.26 and validation loss 1.63. The best model has validation loss 1.24. In perplexity terms:

best checkpoint:   exp(1.24) ≈ 3.46
final checkpoint:  exp(1.63) ≈ 5.10

The final checkpoint is roughly 47% worse in perplexity than the one you had at step 150, despite its training loss looking six times better. Anyone reading only training loss would have concluded the opposite. This is the single most valuable habit the lesson teaches.

Step 6 — the change for the next run. Three epochs was 2.5 times too many for 4,000 examples at this learning rate. Set epochs to 1–2, evaluate every 25 steps, enable early stopping with patience 2 evaluations, and keep the best checkpoint by validation loss. If the minimum lands even earlier next time, the dataset is smaller than the model's appetite and the fix is more or better data, or a lower-capacity adaptation such as a smaller LoRA rank.

Step 7 — the step loss does not close the case. Validation loss says step 150. Your eval set (01-08, 09-01) may disagree slightly, and where they disagree the eval set wins, because it measures the behaviour you promised and loss measures a proxy. Run the eval on the two or three best checkpoints and pick on the metric you shipped against.

Loss curves sit at the intersection of several official objectives: 4.5 (monitor the functioning of data collection, experiments, and other software processes), 1.5 and 1.1 (ML fundamentals and assisting with evaluation of model performance and reliability), and the data-analysis objectives on creating and interpreting visualizations to convey results (2.4, 2.5). Learning curves are one of the named chart types a candidate is expected to be able to read, and the train-versus-validation gap is the canonical over/underfitting diagnostic in the ML-fundamentals material.

The exam asks this as symptom-to-diagnosis matching, which is exactly how this lesson is built. Typical phrasings:

• "A model's training loss continues to decrease while its validation loss increases. What is happening?" — overfitting.
• "Which technique directly addresses the situation above?" — early stopping (also: more data, regularization, fewer epochs).
• "Both training and validation loss plateau at a high value. What does this indicate?" — underfitting / insufficient capacity or training.
• "Which chart type shows model performance against training progress?" — a line chart / learning curve, per 08-04.
• "A training run's loss becomes NaN. Which is the most likely cause?" — exploding gradients or numerical overflow; remedy is gradient clipping or a lower learning rate.
• "Why must validation loss, not training loss, drive checkpoint selection?" — training loss measures fit to data the model is memorizing and cannot estimate generalization.
• "What does the gap between training and validation loss measure?" — the degree of overfitting / the variance component of error.

Distractor families:

It means the model is overfitting: it is still reducing error on the examples it is being updated on, but the patterns it is now learning are specific to those examples and do not transfer to held-out data. The step at which validation loss reached its minimum is the point at which the last generalizing pattern was learned; everything after that is memorization. The fixes, in order of how often they are the right one: stop at the minimum and deploy that checkpoint (early stopping), reduce the number of epochs on the next run, get more or more diverse training data, add regularization such as dropout or weight decay, or reduce the trainable capacity — for LoRA, a smaller rank. Training longer is not a fix; training longer is the mechanism producing the problem.

Training loss is computed on the batches the optimizer is currently learning from, while the weights are being updated and with regularization such as dropout active. Validation loss is computed periodically on a held-out split the optimizer never touches, in evaluation mode with regularization disabled and gradients off. The consequence is that training loss measures fit and validation loss estimates generalization. Two practical corollaries: training loss is noisy because it is a per-batch number while validation loss is smooth because it is averaged over a whole split; and validation loss can legitimately sit below training loss, because dropout makes the training measurement harder than the evaluation measurement. Checkpoint selection, early stopping, and hyperparameter decisions must all be driven by validation loss. The test set stays sealed until the end so that it can audit the decisions validation loss drove.

Look at the gap and the level. Overfitting shows a large and growing gap: training loss low and still falling, validation loss having bottomed out and turned upward. Underfitting shows a small gap at a high level: both curves flat, close together, and disappointing. The mnemonic is that overfitting is a divergence problem and underfitting is an altitude problem. In bias–variance language, underfitting is high bias and overfitting is high variance, and the remedies are opposite — underfitting wants more capacity, more steps, a higher learning rate, and less regularization, while overfitting wants fewer steps, more data, and more regularization. Applying an overfitting remedy to an underfitting curve makes it strictly worse, which is why naming the shape correctly before acting is the entire skill.

Because a number in the forward or backward pass exceeded the representable range of its dtype, or a division or logarithm hit an invalid input, and the resulting inf or nan propagated through the graph. The usual triggers, in rough order of frequency: a learning rate high enough that a single step blows up the weights; exploding gradients in a deep stack with no clipping; FP16's narrow exponent range overflowing where loss scaling is misconfigured or absent; and a genuinely pathological training example — an empty target, a corrupted record, a label out of vocabulary range. The remedies map one to one: clip gradient norms, lower the learning rate and add warmup, switch from FP16 to BF16 so the exponent range matches FP32's (12-01), and locate the offending batch by logging batch indices and re-running deterministically. Once loss is nan, weights are usually corrupted beyond recovery, so restart from the last good checkpoint rather than hoping the run heals.

Early stopping monitors validation loss and terminates training when it has not improved for a set number of consecutive evaluations — that count is the patience. Its purpose is to select the checkpoint at the validation minimum rather than the one at the end of a fixed epoch budget, which makes it both a regularization technique and a compute saving. Choosing patience is a trade-off between two failure modes: too small and normal evaluation-to-evaluation noise stops a run that was still improving; too large and you burn compute past the minimum. Two or three evaluations is a common starting point, with the crucial condition that your evaluation interval is fine enough that a few evaluations is a small fraction of the run. And the detail that causes real incidents: early stopping is useless unless the best checkpoint was actually saved. "Stop when validation stops improving" and "keep the model from before it stopped improving" are two separate configuration settings, and only the second one gives you an artifact.

No, and treating it as if it can is the most consequential misreading in this lesson. A loss curve certifies that optimization behaved: the numbers were stable, the model fit the data, and generalization did not collapse. It says nothing about factual accuracy, hallucination rate, output-format compliance, latency, safety, bias, or whether the model retained capabilities it had before fine-tuning. Catastrophic forgetting is invisible to a validation split that contains only the new task. Format regressions are invisible to cross-entropy. The correct pipeline is: read the loss curve to select candidate checkpoints, then run the frozen eval set with per-slice reporting on those candidates, then decide. The loss curve narrows the field; the eval set makes the call. 12-14 extends the same logic past deployment, where the instrument that validated the build becomes the instrument that detects its decay.

• Read two lines, not one. A single training-loss curve cannot diagnose anything about generalization.
• Ask four questions in order: NaN? Is training loss falling? Are the lines converging or diverging? Where is the validation minimum?
• Overfitting = growing gap plus a validation upturn. Underfitting = small gap at a high level. Divergence = a spike to NaN.
• The gap's trend is the early warning; the validation upturn is the late one.
• The validation minimum is the checkpoint you deploy. Save best-by-validation, not last.
• Fine-tuning on small data overfits by default. Expect the minimum inside the first or second epoch and plan for 1–3 epochs.
• Validation loss below training loss is usually dropout, not a bug.
• Cross-entropy values are not comparable across tokenizers or model families. Compare each run to its own baseline.
• Effective batch size changes when you add GPUs or gradient accumulation, and the learning rate has to follow.
• Loss is a proxy. It cannot see forgetting, hallucination, format compliance, safety, or latency. The frozen eval set is the acceptance criterion.

Next: 12-04 takes up the machinery hiding behind that "× devices" term in the effective-batch formula. When a model trains on more than one GPU, every device computes gradients on different data — and unless those gradients are synchronized into a single averaged update, the replicas silently drift into different models. AllReduce and NCCL are how that synchronization happens.