Loss Functions and Cross-Entropy Loss Explained

Identity statement: cross-entropy measures how far a predicted probability distribution is from the true distribution. When exactly one answer is correct — which is the case for both classification and next-token prediction — it collapses to a single term:

loss = −log(p_correct)

where p_correct is the probability the model assigned to the true label.

When to use it: any task whose output is a probability distribution over discrete classes. Language modelling, sentiment classification, named-entity tagging, toxicity detection. Not for predicting a continuous number — that is what MSE and MAE are for.

The behaviour to internalise, from the shape of the logarithm:

Loss is zero only at perfect confidence in the right answer, and it grows without bound as the model approaches certainty in a wrong one. This is the property that makes cross-entropy the right loss: it punishes confident errors far harder than hesitant ones. A model that says "60% sure" and is wrong pays about 0.9; a model that says "99.9% sure" and is wrong pays about 6.9 — roughly eight times more for the same incorrect top choice.

Why the general formula collapses to one term

You will sometimes see cross-entropy written in its full form, summed over all classes:

loss = −Σ  y_i · log(p_i)        over all classes i

where y_i is the true distribution and p_i the predicted one. That looks like more work than −log(p_correct), and it is worth seeing why it is not.

In ordinary classification and in next-token prediction the true label is one-hot: exactly one class is correct, so y is 1 for that class and 0 for every other. Every term in the sum where y_i = 0 contributes 0 × log(p_i) = 0. Only the correct class survives, and the whole sum reduces to −1 × log(p_correct).

Two consequences fall out of that reduction, and both are testable as reasoning.

First, only the probability on the correct class enters the loss. How the model distributed the rest of its mass does not change the number. That said, the mass elsewhere is not irrelevant to learning, because probabilities must sum to 1 — pushing up the correct one necessarily pushes down the others, which is what the gradient does.

Second, the full form matters when the target is not one-hot. Label smoothing (spreading a little probability across wrong classes deliberately, to discourage overconfidence) and knowledge distillation (training a small model against a large model's full output distribution) both use soft targets, and there the full sum is doing real work. You should recognise the terms; you are not asked to implement either.

L1 — One number per token, averaged

For each position, the model outputs a distribution over the vocabulary. Look up the probability it gave to the token that actually appeared. Take the negative log. Do that for every position and average. That average is the sequence's cross-entropy loss, and it is the number that gradient descent drives down.

Stop here and you can answer most items on this topic. Everything below either makes the arithmetic concrete or connects the number to perplexity.

L2 — From logits to loss, in three steps

1. logits      = model(context)             # raw scores, shape (batch, seq, vocab)
2. probs       = softmax(logits)            # normalised to sum to 1 per position
3. loss        = −mean( log probs[correct_token] )

Step 2 matters for vocabulary: softmax is the function that converts arbitrary real-valued scores into a probability distribution, and cross-entropy is the loss applied to its output. They are two different things that appear together so often they get conflated. In practice frameworks fuse them into one numerically stable operation — PyTorch's CrossEntropyLoss expects raw logits, not softmax output, and applying softmax yourself first is a well-known bug that quietly degrades training.

The shapes tie back to 01-03. Logits arrive as (batch, sequence, vocabulary). The correct-token lookup selects one value per (batch, sequence) position, giving (batch, sequence). The mean over both axes gives a single scalar — rank 0, one number for the whole batch. That scalar is what the optimiser sees.

Note also which positions are scored: every position except the last has a known next token, so a sequence of n tokens contributes about n − 1 loss terms. This is the "one document is thousands of training examples" property from 01-01, now visible as arithmetic.

One practical detail with a real failure mode attached: padding positions must be excluded. A batch is rectangular (01-03), so short sequences carry filler tokens with no meaningful target. If those positions enter the average, the loss is diluted by however much padding happens to be in the batch, and the number becomes a function of your batching strategy rather than of your model. Frameworks provide an ignore-index mechanism for exactly this; a loss that moves when you change batch composition and nothing else is the symptom that it was not used.

L3 — Cross-entropy and perplexity are the same measurement

Perplexity is the exponential of the mean cross-entropy loss:

perplexity = e^(cross-entropy loss)        [when loss is in nats, i.e. natural log]

So a loss of 2.303 is a perplexity of 10.0, and a loss of 0.693 is a perplexity of 2.0. They are strictly monotonic in each other, which means they always rank models identically. The difference is interpretability: perplexity reads as "the model is as uncertain as if it were choosing uniformly among this many equally likely tokens." Perplexity 10 means "effectively picking among 10 options"; perplexity 2 means "effectively a coin flip."

That gives you an immediate sanity check. A model with a 32,000-token vocabulary that has learned nothing predicts uniformly, so p_correct = 1/32,000, giving a loss of ln(32,000) ≈ 10.4 and a perplexity of 32,000. Any real trained model reports far less. If you ever see a training loss hovering near ln(vocab_size), the model is not learning.

The distinctive relationship to carry: perplexity's floor is 1 (perfect prediction) and its ceiling for an untrained model is the vocabulary size. Cross-entropy loss and perplexity are one measurement expressed in two units — like Celsius and Fahrenheit, never in conflict.

A units footnote that occasionally matters when reading papers. Cross-entropy in nats uses the natural log and exponentiates with e; in bits it uses log base 2 and exponentiates with 2. Both give the same perplexity, because the base cancels: 2^(log₂-loss) = e^(ln-loss). Bits-per-character and bits-per-byte figures you sometimes see quoted are the same measurement in the base-2 convention. Mixing bases without converting is the only way to get this wrong.

The lesson stops at L3 by design. The full treatment of loss functions as model comparison tools belongs to the Experimentation module (09-05 owns it in this course's numbering; the blueprint lists it at 3.04 and 2.2/3.2). Here you need the identity, the direction, and the perplexity relationship.

Objective 2.2 and objective 3.2 are printed identically in the official study guide: "compare models using statistical performance metrics, such as loss functions or proportion of explained variance." That wording names two things — a loss function and R² — so the table below is close to the objective's literal text. This is the highest-value asset on the page, because metric-to-task matching is the single most likely item form.

Four distinctions in that table generate almost every distractor you will meet.

Discrete versus continuous output. Cross-entropy needs a probability distribution over classes. MSE, RMSE and MAE need a continuous number. Offering cross-entropy for a house-price prediction, or MSE for a sentiment classifier, is the standard trap.

Loss versus metric. Cross-entropy and MSE are differentiable, so they can be optimised — that is their job during training. Accuracy, F1, R², BLEU and ROUGE are not differentiable (or not usefully so) and cannot be trained against directly; they are reports. A model can improve its loss while its accuracy stays flat, because loss registers a rise in confidence on answers that were already correct.

Direction of goodness. Losses and perplexity go down. R², accuracy, F1, BLEU and ROUGE go up. Mixed-direction lists are exactly what distractors are built from, and the fastest way to lose a point here is to read "the model with the highest loss performed best."

Loss versus similarity. Cosine similarity from 01-04 compares two vectors. Cross-entropy compares a predicted distribution to a truth. Different objects, different purposes; a distractor that offers cosine similarity as a loss function for classification is testing exactly this.

MSE vs MAE, since they are the pair inside the pair

Both are regression losses and both go down, so the discriminator has to be their behaviour on outliers.

Illustrative arithmetic on four errors of 1, 1, 1 and 10:

MSE = (1 + 1 + 1 + 100) / 4 = 25.75      ← the single 10 supplies 97% of it
MAE = (1 + 1 + 1 + 10)  / 4 = 3.25       ← the single 10 supplies 77% of it

Same errors, radically different emphasis. That is the whole reason both exist, and it is the level of detail the exam asks for. Note the structural parallel with cross-entropy: MSE is to outliers what cross-entropy is to confident errors — a loss that deliberately makes the worst cases hurt most.

R², since the objective names it in words rather than symbols

"Proportion of explained variance" is R², and the phrase-to-term mapping is worth memorising in the objective's own wording because that is how it appears.

R² = 1 − (sum of squared residuals) / (total sum of squares)

Read as: of all the variation in the target, what fraction did the model account for? R² = 1 means every point predicted exactly. R² = 0 means the model did no better than always predicting the mean. R² can be negative, which means it did worse than predicting the mean — a genuine and instructive result, and one people wrongly assume is impossible because they remember the name "R-squared" and infer a squared quantity cannot be negative.

Two further points of precision: R² is a regression quantity and has nothing to do with classification or language modelling, and higher is better, which makes it the odd one out among the metrics that most often appear next to it.

A single position. The context is The capital of France is, and the true next token is Paris. Treat the probabilities as an illustrative construction rather than measurements from any named model.

Model 1 assigns Paris a probability of 0.89:

loss = −ln(0.89) = 0.117

Model 2, less well trained, assigns Paris a probability of 0.20:

loss = −ln(0.20) = 1.609

Model 3 is confidently wrong — it gives Paris only 0.001 and puts 0.95 on Berlin:

loss = −ln(0.001) = 6.908

Only the probability on the correct token enters the calculation. What the model did with the rest of its 32,000 tokens is irrelevant to the loss at this position — though it is not irrelevant to the gradient, which pushes down on the mass wrongly assigned elsewhere.

Note the ratio between Model 1 and Model 3: a 59-fold gap in loss between a good prediction and a confidently wrong one, from probabilities that differ by less than one unit. The logarithm is what produces that leverage, and it is why training moves fastest on the examples the model is most wrong about.

A short sequence. Four scored positions, with these probabilities on the correct tokens:

position 1:  p = 0.90  →  −ln(0.90) = 0.105
position 2:  p = 0.60  →  −ln(0.60) = 0.511
position 3:  p = 0.30  →  −ln(0.30) = 1.204
position 4:  p = 0.80  →  −ln(0.80) = 0.223

mean loss  = (0.105 + 0.511 + 1.204 + 0.223) / 4 = 0.511
perplexity = e^0.511 = 1.67

Perplexity 1.67 on a 32,000-token vocabulary is very strong — the model is behaving as though it had fewer than two plausible options at each step, against a possible 32,000. And note position 3 contributing more than the other three combined: in an averaged loss, the hardest tokens dominate the number, which is why loss curves are noisy and why one pathological batch can spike a run.

The same sequence with one catastrophic token

Change position 3 alone to p = 0.001 and leave the other three untouched:

position 1:  p = 0.90   →  0.105
position 2:  p = 0.60   →  0.511
position 3:  p = 0.001  →  6.908
position 4:  p = 0.80   →  0.223

mean loss  = (0.105 + 0.511 + 6.908 + 0.223) / 4 = 1.937
perplexity = e^1.937 = 6.94

One token out of four moved the mean loss from 0.511 to 1.937 — nearly a four-fold increase — and the perplexity from 1.67 to 6.94. Three quarters of the tokens were predicted just as well as before.

This is the most useful diagnostic intuition in the lesson. A spiking loss curve does not mean the model got broadly worse; it usually means a few very badly predicted tokens entered the batch. Data corruption, an encoding problem, a document in an unexpected language, or a very long rare token can all produce it. The fix is to look at which examples are producing the high per-token losses, not to reach immediately for the learning rate.

You are handed a training log and told the loss is 4.2. Is that good?

Unanswerable without the vocabulary size, which is exactly the point. The reference point is always the uniform baseline, ln(number of classes):

Now the 4.2 reads instantly:

On a 10-class problem:    4.2 > 2.30  →  worse than guessing. Something is broken.
On a 32k-vocab LLM:       4.2 << 10.37 →  perplexity e^4.2 ≈ 67, a real trained model.
On a binary problem:      4.2 >> 0.69  →  catastrophically wrong, likely a label bug.

Two habits to take from this. First, always compute ln(number of classes) before interpreting a loss. Second, if a loss sits stubbornly at the uniform baseline the model is predicting nothing — check that labels are aligned with inputs, that the learning rate is not zero, and that gradients are actually reaching the parameters (01-06).

And the corollary that catches teams out: loss values are not comparable across models with different vocabularies. A 128k-vocabulary model faces a harder prediction problem per token than a 32k one, all else equal, so its loss and perplexity will tend to be higher without being worse. Perplexity comparisons are only meaningful between models sharing a tokenizer and evaluated on identical text.

Loss functions are named explicitly in the official objective text for two domains — Data Analysis at 14% and Experimentation at 22% — through the duplicated "compare models using statistical performance metrics, such as loss functions or proportion of explained variance." They also underpin Core ML at 30% as the mechanism of training. That duplicated objective is itself a documented defect in the source PDF, where the Experimentation objectives repeat the Data Analysis ones verbatim; the practical upshot is that this exact phrase is printed twice, which raises rather than lowers the odds of seeing it.

Candidate reports place loss functions in a mid-to-lower frequency tier rather than the top, so calibrate accordingly: the return here is in clean, fast recognition, not depth. That tiering is [FIELD] calibration from published candidate reports, not an official statement — NVIDIA publishes no item-level detail.

How the question tends to be phrased

• Metric-to-task matching. "Which metric is most appropriate for evaluating a model that predicts a continuous value?" or the same question for a classifier or a language model. The single most likely item form on this topic.
• Phrase-to-term mapping. "Proportion of explained variance" appears in the objective's exact words; the answer is R². Know it cold, in both directions.
• Direction of goodness. An item that asks which of several reported numbers indicates the best model, mixing losses (lower better) with accuracy or R² (higher better).
• Perplexity's definition. "Perplexity is best described as…" with the keyed answer naming the exponential of cross-entropy, or the effective number of equally likely choices.
• Loss versus metric. "Why is accuracy not used as a training objective?" with the keyed answer naming non-differentiability.

What the distractors typically look like

Four reliable families. Task swaps: MSE offered for classification, cross-entropy for regression. Direction inversions: "the model with the highest perplexity generalises best." Category errors: cosine similarity, BLEU or a vector distance offered as a classification loss. Near-miss definitions of perplexity: "the probability of the correct token", "the number of parameters", "the inverse of accuracy" — all wrong, all plausible-sounding, and all defeated by knowing it is e^loss.

Notice the pattern across all four: each distractor is a real, correct concept placed in the wrong slot. The defence is the metric-to-task table in section 3, held as a shape rather than as a list.

The bottom row is the one to carry furthest. Loss is the training signal. It is not the acceptance criterion. Every serious LLM project ends up with an evaluation set that measures the thing users care about, precisely because a good loss is necessary and nowhere near sufficient.

No, and the three ways it fails are all exam-relevant.

Lower training loss can mean memorisation. A model can drive training loss toward zero by memorising its training examples while getting worse on anything new. The diagnostic is the gap between training and validation loss, which is the subject of 01-07. A falling training loss with a rising validation loss is the canonical overfitting signature.

Lower loss can mean an easier evaluation. Loss is computed on a specific set of text. Move to easier text and the number falls without the model changing. This is why loss comparisons require identical evaluation data, and why benchmark contamination — evaluation text that leaked into training — produces impressively low numbers that mean nothing.

Lower loss does not mean more useful. Cross-entropy measures next-token uncertainty. It does not measure whether answers are helpful, correctly formatted, safe, well-cited or factually right. A model can have excellent perplexity and refuse to follow instructions, because instruction following came from a later stage (01-01) that perplexity on generic text does not probe.

The honest summary: loss is the right thing to minimise during training and the wrong thing to make a shipping decision on. Both halves of that sentence matter.

Because accuracy is not differentiable, and training needs a gradient.

Accuracy asks a yes-or-no question: was the top prediction correct? Nudge a weight by a tiny amount and the answer almost always stays exactly the same, then occasionally flips. A function that is flat almost everywhere and jumps at thresholds gives a gradient of zero almost everywhere — nothing to descend. 01-06 explains why zero gradients mean no learning.

Cross-entropy is smooth in the model's outputs. Any small improvement in the probability assigned to the correct answer produces a small decrease in the loss, so there is always a direction to move in. That is the technical requirement a training objective must satisfy, and accuracy fails it.

The secondary reason is that cross-entropy carries more information per example. Accuracy sees only whether the argmax was right. Cross-entropy sees how confident the model was, so it can distinguish "correct but barely" from "correct and certain", and it can distinguish "wrong but nearly right" from "wrong and confident". That extra signal is what makes learning efficient.

Which is why real projects use both, at different points: cross-entropy to train, accuracy and F1 to report. They are not competing answers to the same question.

Only under conditions that are usually not met, and knowing the conditions is the useful part.

Perplexity is comparable when the two models share a tokenizer and are evaluated on identical held-out text. Then the comparison is apples to apples: the same prediction problems, scored the same way.

It becomes unreliable the moment either condition breaks. Different tokenizers mean different numbers of tokens for the same text and different vocabulary sizes, so the per-token difficulty differs — a model with a coarser tokenizer faces fewer, harder predictions and can look worse for reasons that have nothing to do with quality. Different evaluation text means different difficulty. And if either model saw the evaluation text during training, its perplexity is meaningless.

Beyond the mechanics, there is a scope limit: perplexity measures the pretraining objective, and much of what makes a modern LLM useful is added afterwards. Instruction tuning and alignment can make a model dramatically more useful while leaving generic-text perplexity roughly unchanged, or even slightly worse. So a perplexity comparison is silent about exactly the properties you most likely care about.

The practical position: use perplexity for tracking one model's progress during training on a fixed held-out set, where it is excellent. Use task metrics and your own evaluation set for choosing between models, which is what 01-08 builds and what the generation-quality metrics lesson refines.

• A loss function compresses prediction error into one differentiable number that training minimises.
• Cross-entropy = −log(probability assigned to the correct answer), averaged over positions. The general summed form collapses to this because one-hot targets zero out every other term.
• It is the loss for classification and language modelling because next-token prediction is classification over the vocabulary.
• It punishes confident wrong answers disproportionately: about 0.9 at 60% wrong confidence versus about 6.9 at 99.9%.
• Perplexity = e^(cross-entropy loss). Same measurement, different units. Floor of 1; an untrained model sits at roughly the vocabulary size.
• A loss value means nothing without the class count. Compare against ln(number of classes).
• Match metric to task: cross-entropy for classification, MSE/RMSE/MAE for regression, R² for proportion of explained variance, accuracy/F1 for final labels, BLEU/ROUGE for generated text.
• Direction matters: losses and perplexity down, R²/accuracy/F1/BLEU up.
• Loss is trainable; accuracy is only reportable — because accuracy is not differentiable.
• Lower loss is not automatically a better model: check for memorisation, easier evaluation data, and the gap between loss and usefulness.

You can now score a prediction. What you cannot yet explain is how that single number turns into changed weights across billions of parameters — which is the step that connects the loss you just computed to the knowledge stored in the parameters from 01-02.

Next: 01-06 Gradient descent and backpropagation — how the loss propagates backwards to produce a gradient for every parameter, and how the optimizer takes the step. Taught to the depth the exam asks for and deliberately no further.