Gradient Descent and Backpropagation Explained

Backpropagation is an algorithm for computing gradients efficiently. A gradient is a derivative: for one parameter, it answers "if I nudge this number up slightly, does the loss go up or down, and how sharply?" Backpropagation obtains that answer for every parameter in the model by applying the chain rule from the loss backwards, layer by layer, reusing intermediate results so the whole sweep costs about the same as one forward pass.

Gradient descent is the update rule that consumes those gradients:

new_value = old_value − learning_rate × gradient

The minus sign is the algorithm's entire idea. The gradient points uphill on the loss surface; you want to go downhill; so you subtract.

When each applies: both are training-time only. Neither happens during inference. A served model's parameters are frozen (01-02), so no gradient is computed and no step is taken when you send a prompt.

The division of labour, which is the most commonly muddled point on this topic:

They are not synonyms and they are not alternatives. Backpropagation without an optimizer computes numbers nobody uses; gradient descent without backpropagation has no numbers to use. That last row is the clarifying one: swapping SGD for Adam changes the optimizer and leaves backpropagation completely untouched, which is why "Adam replaces backpropagation" is a wrong statement and a good distractor.

What a gradient is, in one paragraph without calculus

For one parameter, the gradient is a signed number with two pieces of information. Its sign says which direction increases the loss: positive means raising the parameter would make things worse, negative means lowering it would. Its magnitude says how sharply — a large gradient means the loss is very sensitive to this parameter right now, a near-zero gradient means the parameter barely matters at this point in training.

For the whole model, "the gradient" is the collection of all of those numbers: one per parameter, so a 7-billion-parameter model has a 7-billion-entry gradient. That is why gradients cost as much memory as the weights themselves, and it is the first term in the training-memory arithmetic that section 2's L3 tier finishes.

L1 — The four-step cycle

Every training step, for one batch of data, does the same four things in the same order:

1. Forward pass — run the batch through the model to get predictions.
2. Loss — score the predictions against the truth with cross-entropy (01-05).
3. Backward pass — backpropagate to get a gradient for every parameter.
4. Update — the optimizer steps each parameter against its gradient.

Then repeat with the next batch. That loop is the whole of training. Everything else — schedules, mixed precision, distributed parallelism — is engineering around those four steps.

If you retain only the order of those four words, you can answer the most common item form on this topic. The order is never shuffled: you cannot score a prediction you have not made, and you cannot step in a direction you have not computed.

The vocabulary that goes with the loop, and that the exam does test:

An epoch therefore contains many steps: dataset size ÷ batch size. Confusing an epoch with a step is a standard distractor.

Worth doing the arithmetic once so the relationship is concrete. Illustrative numbers:

dataset      = 1,000,000 examples
batch size   = 32
steps/epoch  = 1,000,000 / 32 = 31,250 steps
3 epochs     = 93,750 weight updates

Almost a hundred thousand updates from three passes over the data. Now double the batch size to 64 and hold the epoch count: you get half as many updates, each computed from twice as much data. That trade is the reason batch size and learning rate are tuned together — fewer, better-estimated steps often want a larger learning rate to cover the same ground.

L2 — What the learning rate controls, and what goes wrong

The learning rate is the single most consequential hyperparameter in training, and its failure modes are symmetric:

Because the ideal value changes over the course of a run, real training uses a schedule — commonly a short warmup, then a decay. Warmup exists because the very first steps of training happen at a random initialisation where gradients are large and unreliable; taking full-size steps there destabilises the run. Decay exists because large steps are useful for covering ground early and harmful for settling into a minimum late.

The three schedule shapes worth recognising by name:

You are not asked to choose between them on the exam. You are asked to know that a schedule exists, that warmup is at the start, and that the learning rate is not a single fixed number for a whole run.

L3 — Optimizer variants and gradient pathologies, at recognition depth

Three names to recognise, in the order they were built on each other:

The distinctive fact this page owns: Adam's adaptivity is not free in memory. It stores two extra values per parameter — a first-moment and a second-moment estimate — so the optimizer state alone is about twice the size of the model's weights, on top of the weights and the gradients. That is the concrete reason training needs several times the memory of serving, and the reason parameter-efficient methods like LoRA are so effective: freezing the base weights removes optimizer state for all of them, not just gradient computation.

Two pathologies you should be able to name from a symptom:

• Vanishing gradients — gradients shrink toward zero as they propagate back through many layers, so early layers barely learn. Mitigated by ReLU-family activations, residual connections and normalisation layers, which is why transformer blocks contain residual connections and layer norm.
• Exploding gradients — gradients grow without bound, producing NaN loss. Mitigated by gradient clipping, which caps gradient magnitude before the update.

The mechanism behind both, in one sentence, because it makes them memorable rather than arbitrary: the chain rule multiplies a factor per layer, so if those factors are consistently below 1 the product shrinks toward zero over many layers, and if they are consistently above 1 it grows without bound. Depth is what turns a small bias in either direction into a catastrophe, which is precisely why architectural fixes — residual connections that give the gradient a shortcut path, normalisation that keeps activations in a sane range — were the enabling ingredients for very deep networks.

This is the depth ceiling, stated explicitly. You will not implement backpropagation and you will not be asked to; published candidate reports consistently describe the exam as general-level, favouring "what is it and when do you use it" over derivation. That is [FIELD] calibration rather than an official statement — NVIDIA publishes no item-level detail — but it is consistent enough across reports to plan study around.

Five terms sit close together and get swapped. The table is the fastest defence.

Two clarifications the table earns.

"Optimizer" and "gradient descent" are not quite the same word. Gradient descent is the basic rule; an optimizer is whatever object implements a rule, and every optimizer you will meet is a gradient-descent variant. Saying "the optimizer takes the step" is always safe.

Backpropagation is not a training method, it is a differentiation method. It would still be the right way to compute those derivatives even if you used some completely different update rule. Holding that distinction is what makes the "which one computes and which one applies" item trivially easy.

Take a single weight w with a current value of 0.500 and a learning rate of 0.1. Backpropagation reports its gradient at each step. The gradient values are an illustrative construction chosen to show the behaviour, not measurements.

step 1:  gradient = +2.0   →  w = 0.500 − 0.1(+2.0) = 0.300
step 2:  gradient = +1.0   →  w = 0.300 − 0.1(+1.0) = 0.200
step 3:  gradient = −0.4   →  w = 0.200 − 0.1(−0.4) = 0.240

Read the behaviour. In step 1 the gradient is positive and large: raising w would raise the loss a lot, so w drops sharply. By step 2 the gradient has halved — the parameter is nearer a minimum — so the step is smaller. In step 3 the gradient has gone negative, meaning w overshot slightly, and subtracting a negative moves it back up. The parameter is converging by oscillating into a valley, and the shrinking gradient magnitude is doing the step-size reduction automatically.

Now the same sequence with a learning rate of 1.0:

step 1:  gradient = +2.0   →  w = 0.500 − 1.0(+2.0) = −1.500

One step has thrown the parameter far past any plausible minimum. Do that across billions of parameters simultaneously and the loss diverges to NaN within a handful of steps. This is the entire content of "the learning rate was too high," and it is worth being able to picture, because the exam tests the symptom rather than the arithmetic.

And the opposite extreme, learning rate 0.001 on the same gradients:

step 1:  gradient = +2.0   →  w = 0.500 − 0.001(+2.0) = 0.498
step 2:  gradient = +2.0   →  w = 0.498 − 0.001(+2.0) = 0.496
step 3:  gradient = +2.0   →  w = 0.496 − 0.001(+2.0) = 0.494

Three steps have moved the parameter by 0.006. At this rate it would take hundreds of steps to reach the value one step achieved earlier, and the gradient has not even begun to shrink because the parameter has barely moved. Multiply that waste by the cost of a GPU cluster and "too low" stops sounding like the safe option.

Finally, scale it. An LLM does not do this to one parameter; it does it to every parameter at once, per step, for hundreds of thousands of steps. That is what a GPU cluster is for, and it is why the collective-communication topics later in the course exist: with the model split across devices, the gradients must be summed across all of them before any step can be taken.

This is the arithmetic that turns "training is expensive" into a number you can defend, and it is the single most useful thing to carry out of the optimizer discussion. Illustrative construction: a 7-billion-parameter model, full fine-tuning, Adam, weights in BF16 at 2 bytes and optimizer state in FP32 at 4 bytes.

weights           7e9 × 2 bytes  =  14 GB
gradients         7e9 × 2 bytes  =  14 GB      one number per parameter
Adam moment 1     7e9 × 4 bytes  =  28 GB
Adam moment 2     7e9 × 4 bytes  =  28 GB
--------------------------------------------------
subtotal                            84 GB
plus activations saved for the backward pass, which scale with
batch × sequence × hidden × layers  →  substantial and batch-dependent

Against 01-02's serving figure of about 14 GB for the same weights, that is roughly a six-fold increase before activations. The proportions, not the exact bytes, are the point: weights are one share, gradients another equal share, and Adam's two moments the largest share of all.

Now the same model under LoRA, where the base weights are frozen and only a small added set is trainable. Suppose the trainable set is 0.5% of the parameters:

weights (frozen)         14 GB      still resident, but no gradient or state
gradients      0.005 × 7e9 × 2  =  0.07 GB
Adam moments   0.005 × 7e9 × 8  =  0.28 GB
--------------------------------------------------
subtotal                          ~14.4 GB

From roughly 84 GB to roughly 14 GB, and the reason is visible in the arithmetic: the three per-parameter terms that dominated have shrunk by 200×, while the one term that cannot shrink — the frozen weights themselves — is unchanged. That is why parameter-efficient fine-tuning is not a small optimisation but a change in which hardware the job can run on at all.

Two honest caveats. Activation memory is omitted because it depends on batch size, sequence length and whether activation checkpointing is used, and inventing a figure would be fabrication. And real frameworks differ in which tensors they keep at which precision, so treat the numbers as a structural illustration rather than a specification.

Objective 1.5 asks for familiarity with the fundamentals of machine learning, and the official study guide's suggested-reading list names Back Propagation as its own entry — a direct signal that the term is expected vocabulary. Core ML is 30% of the blueprint, the largest domain.

Reported frequency places optimizers and loss functions in a middle tier rather than the top, so the efficient posture is recognition, not mastery.

How the question tends to be phrased

• Term-to-role matching. "Which of the following computes the gradient of the loss with respect to each weight?" Keyed: backpropagation. Distractors: gradient descent, the loss function, the forward pass.
• Order of operations. "Place the following in the order a single training step executes them." Keyed order: forward pass, loss, backward pass, weight update.
• Learning-rate symptoms. "During training the loss oscillates wildly and then becomes NaN. What is the most likely cause?" Keyed: the learning rate is too high.
• Unit definitions. "A dataset of 10,000 examples is trained with a batch size of 100. How many steps are in one epoch?" Keyed: 100.
• Optimizer identity. Adam as the adaptive default for transformers; SGD as the plain rule; momentum as the accumulated-direction variant.
• Pathology-to-mitigation matching. Exploding gradients → gradient clipping. Vanishing gradients → ReLU-family activations, residual connections, normalisation.
• Training versus inference. "Which of these occurs during inference?" Keyed answer includes the forward pass and excludes backpropagation and the weight update.

What the distractors typically look like

Four families recur. Role swaps: "gradient descent computes the gradient" or "backpropagation updates the weights." Unit swaps: epoch where step belongs, or batch size where step count belongs. Direction errors: an update rule written with a plus sign instead of a minus, or "a higher learning rate always converges faster." Category errors: naming an activation function or a loss function where an optimizer belongs, since ReLU, softmax, cross-entropy and Adam all live in the same neighbourhood of a study guide and are easy to shuffle.

The defence against all four is the section 3 table, which fixes each term's category. Once "backpropagation is a differentiation algorithm" and "Adam is an optimizer" are firmly categorised, role-swap distractors stop being tempting.

The seventh row is worth pausing on, because it is the most commonly misdiagnosed situation in practice. A rising validation loss is not a gradient-descent failure — the optimiser is doing its job perfectly, driving the training loss down. The problem is what "down" is being measured on, which is exactly why 01-07 exists and why it is the root of the whole Experimentation domain.

No. It finds a local minimum that works well enough, and the honest framing of why is more interesting than the disclaimer.

A billion-parameter loss surface is not a bowl with one bottom. It has an enormous number of minima, saddle points and flat regions. Gradient descent follows the local slope from wherever initialisation happened to place it, so a different random initialisation, a different data order, or a different learning-rate schedule can land in a different minimum. Nobody verifies global optimality at this scale; nobody could.

What makes this acceptable in practice is an empirical observation rather than a proof: in large overparameterised networks, most of the minima gradient descent actually reaches turn out to be roughly comparable in quality. So the practical question shifts from "did we find the best minimum" to "does this checkpoint pass evaluation" — which is a measurement question, not an optimisation one.

Two consequences you can use. First, training runs are not exactly reproducible unless every source of randomness is pinned, which is why serious experiments record seeds alongside results. Second, the right response to a disappointing trained model is usually not a better optimiser. It is better data, a different objective, or a corrected evaluation. Optimiser choice is rarely the binding constraint.

Because both directions have real costs, and only one of them is dramatic enough to be noticed.

Too high fails loudly: NaN loss, wild oscillation, an obviously broken run. You cannot miss it, so you fix it.

Too low fails quietly and expensively. The loss does go down, the curve looks plausible, and nothing errors. What you lose is invisible: compute spent covering ground a larger step would have covered for free, and a real risk of settling into a mediocre region because the steps were never large enough to escape it. A run that finishes on schedule with a disappointing final loss and a perfectly smooth curve is the classic signature, and it is much harder to diagnose than a NaN.

There is also an interaction that makes "just go lower" actively wrong at scale. Larger batches give better-estimated gradients, so they generally want a larger learning rate; pairing a large batch with a timidly small rate wastes the batch's main benefit. Learning rate and batch size are tuned together, not independently.

The practical posture, which is what the exam actually rewards: the learning rate is a hyperparameter with symmetric failure modes, real training uses a schedule rather than one fixed value, and warmup exists because the earliest steps are the least trustworthy.

• Backpropagation computes the gradient for every parameter via the chain rule, backwards from the loss. It is a differentiation algorithm.
• Gradient descent applies it: new = old − learning_rate × gradient. The minus sign is the algorithm.
• The training loop is four steps in fixed order: forward → loss → backward → update.
• Learning rate too high → divergence or NaN. Too low → stalled, expensive progress. Schedules with warmup manage this, and no direction is risk-free.
• Adam/AdamW is the transformer default; its state costs roughly 2× the weights in memory, which with gradients puts full fine-tuning at several times serving cost.
• Freezing base weights (PEFT/LoRA) removes gradients and optimizer state for those parameters, which is why it changes which hardware a job can run on.
• Vanishing gradients are mitigated by ReLU-family activations, residual connections and normalisation; exploding gradients by clipping. Both come from per-layer factors compounding through depth.
• Epoch = full pass over data. Step = one weight update. They are not the same unit.
• Gradient descent finds a local minimum that works, not a proven global one.
• All of this is training-time only. Inference is frozen.

You can now describe how a model's parameters change. What you cannot yet do is tell whether the change was an improvement or merely memorisation — and a loss that falls on the training data proves nothing on its own. Answering that requires holding data back, which is the root of the entire Experimentation domain.

Next: 01-07 Train, validation, and test splits — the three disjoint datasets, what each is allowed to be used for, and how to read overfitting straight off the gap between training and validation loss.