How GPUs work and what happens during a training run

A GPU (graphics processing unit) is a co-processor designed for throughput: getting an enormous volume of identical arithmetic done per second. It sits alongside the CPU, has its own dedicated memory, and only works on data that has been explicitly copied into that memory. The CPU is designed for latency: finishing any one unpredictable task as fast as possible.

The contrast in one table:

The term you will see for the GPU execution model is SIMT — single instruction, multiple threads. Thousands of threads execute the same instruction stream over different array elements. Because a matrix multiplication computes millions of independent dot products, and each dot product is the same multiply-and-add sequence, the mapping is near perfect. This is the entire reason the modern LLM era is a GPU era.

NVIDIA's ecosystem terms that go with this, at the identity level the exam expects:

That last row matters more than it looks: it is a memory feature. Partitioning a GPU gives each tenant a fixed memory slice, which tells you what the scarce resource really is.

L1 — Intuition: the factory floor

Think of a CPU as one extremely skilled craftsman who can build anything, one item at a time, adapting as they go. A GPU is a factory floor with thousands of workers who can each do exactly one simple motion, all at the same time, on a conveyor belt of identical parts. Ask the craftsman to tighten ten million identical bolts and he will be slow, however skilled. Ask the factory to hand-build a bespoke item and it will be useless.

Neural network training is ten million identical bolts.

Two consequences follow from the factory picture and they are the two facts that dominate practice:

1. Parts must be delivered to the floor. The GPU can only work on data in its own memory. Copying between system RAM and GPU memory is comparatively slow, so real pipelines try to move data over once and keep it there. A workload that spends its life copying is called transfer-bound, and it will be slower than doing the work on a CPU.
2. The floor has a fixed footprint. VRAM is finite, and much smaller than system RAM. Everything the training step needs must be on the floor simultaneously. When it does not fit, nothing degrades gracefully — the job raises an out-of-memory error and stops.

L2 — Mechanism: one training iteration in five stages

Here is what actually happens when you call a training loop. Every stage has a memory consequence, listed alongside.

Then repeat, thousands of times. The loss should trend downward; that trend is the only evidence that anything is working.

The four persistent-or-peak memory consumers, which is the list to memorise:

This table is the reason inference fits comfortably on hardware that cannot train the same model at all. Drop gradients and optimizer state and you have removed the majority of the training footprint.

Precision is the multiplier on all of it. A number stored in fp32 takes 4 bytes; in fp16 or bf16, 2 bytes; in int8, 1 byte; in fp4/int4 schemes, half a byte. So the same model's weights occupy half as much memory in bf16 as in fp32, which is why mixed precision — keeping most arithmetic in 16-bit while holding a few sensitive accumulations in 32-bit — became the default for training. You will meet quantization properly in the deployment module; here it is enough to know that precision is a straight multiplier on the memory bill.

Optimizer state is the consumer that surprises people. Plain SGD stores essentially nothing extra. Adam and AdamW, which are what LLM work actually uses, store two additional values per parameter — a running average of gradients and a running average of squared gradients. So Adam's state alone is roughly twice the size of the model's trainable parameters, in whatever precision it keeps them. That single fact explains most "why can't I fine-tune a model that loads fine" confusion.

L3 — Where the time actually goes: compute-bound vs memory-bound

A useful mental model, and one that recurs throughout this course: any GPU operation is limited either by how fast the cores can do arithmetic (compute-bound) or by how fast numbers can be moved between memory and cores (memory-bandwidth-bound).

Training with a healthy batch size is usually compute-bound, which is the happy case: the hardware is doing what it is for. Single-stream text generation is famously memory-bound, because each new token requires reading the whole set of weights out of memory to produce a handful of numbers. That asymmetry is the root of most modern serving optimisation — batching many requests together, caching, and reducing the bytes each token has to read — and it is the subject of 12-05 much later.

Two further mechanisms worth naming now because they explain error messages you will see:

• Asynchronous execution. GPU work is queued, not executed inline. Your Python line returns before the GPU has finished. A consequence: a CUDA error's stack trace can point at an innocent line, because the failure surfaced whenever the queue was next synchronised. If a traceback makes no sense, this is usually why.
• The caching allocator. Frameworks do not return freed GPU memory to the driver immediately; they keep a pool to avoid slow allocation calls. This means "memory in use" as your framework reports it and "memory the driver thinks is allocated" differ, and it means memory can become fragmented — you can have 3 GB free in total and still fail to allocate a contiguous 2 GB block. Hence the genuinely infuriating error that says memory is available and then refuses to give it to you.

Two comparison tables. The first is the processor choice.

Note the row that most candidates skip: the CPU is not a failure case. Tokenising a corpus, chunking documents, computing BLEU, running a similarity search over a few thousand embeddings — all of these are perfectly good CPU work, and reaching for a GPU adds transfer cost for no benefit. Knowing when GPU acceleration does not pay off is an explicitly examinable judgement, and it comes up again in the RAPIDS material.

The second table is the memory distinction, which is the one that causes the most confusion in a first lab.

The two most common confusables in this area, stated flatly:

• "My machine has 32 GB, so a 20 GB model fits." No. 32 GB of system RAM has almost nothing to do with whether a model fits in VRAM. These are separate pools, and only one of them the GPU can compute in.
• "The GPU is too slow." Usually it is not slow, it is out of memory, or it is memory-bandwidth-bound, or your data loader is starving it. Speed complaints in this field are overwhelmingly memory complaints in disguise.

Everything in this section is a constructed illustrative example, not a measured benchmark. The purpose is to show the shape of the arithmetic; the real number for any specific model and framework differs, and you should always confirm against your own nvidia-smi reading rather than trusting an estimate. Framework overhead, kernel workspaces and allocator behaviour add real amounts that a back-of-envelope calculation deliberately ignores.

Suppose a model with 7 billion parameters. Take the four consumers in turn.

Step 1 — weights. Bytes per parameter depends entirely on precision:

fp32 (4 bytes):  7e9 × 4 = 28e9 bytes  ≈ 28 GB
bf16 (2 bytes):  7e9 × 2 = 14e9 bytes  ≈ 14 GB
int8 (1 byte):   7e9 × 1 =  7e9 bytes  ≈  7 GB
4-bit (0.5):     7e9 × 0.5 = 3.5e9     ≈  3.5 GB

Already the headline result: the same model is an 8× different memory problem depending on precision alone. Note these are GB in the decimal sense — M0.4 is entirely about why that distinction is not pedantry.

Step 2 — gradients. One gradient value per trainable parameter, typically in the same precision as the compute. Training all 7B parameters in bf16:

gradients ≈ 7e9 × 2 = 14e9 bytes ≈ 14 GB

Step 3 — optimizer state. Adam/AdamW keeps two values per parameter. Many implementations keep them in fp32 for numerical stability even when compute is bf16, so assume 4 bytes each and state that assumption:

optimizer state ≈ 7e9 × 2 values × 4 bytes = 56e9 bytes ≈ 56 GB

Step 4 — running total before activations.

weights (bf16)       14 GB
gradients (bf16)     14 GB
optimizer (fp32 ×2)  56 GB
                   ------
subtotal             84 GB     ← and no activations yet, and no framework overhead

Step 5 — activations, order of magnitude. Activation memory scales with batch × sequence × hidden × layers and depends heavily on implementation details such as whether activation checkpointing is enabled. Rather than invent a figure, note the scaling and the lever: at a large batch and long sequence, activations can be the dominant term, and activation checkpointing trades recomputation time to shrink it substantially. Treat activations as "somewhere between a few GB and the largest single item on the list, depending on batch size", and measure rather than guess.

The conclusion this arithmetic forces: full fine-tuning of a 7B model with Adam is a job for many tens of GB of VRAM — well beyond a single consumer card and beyond a free notebook tier. This is not a marginal shortfall you can squeeze past with a smaller batch. It is roughly an order of magnitude.

Step 6 — now do the same arithmetic for a LoRA-style adapter. Suppose you freeze all 7B base weights and train only a small set of added parameters — say 0.1% of the model, which is 7 million trainable parameters. Constructed figure; real adapter sizes vary with rank and which layers you target.

base weights, frozen, 4-bit    ≈  3.5 GB
gradients: 7e6 × 2 bytes       ≈ 0.014 GB
optimizer: 7e6 × 2 × 4 bytes   ≈ 0.056 GB
                                ---------
persistent subtotal            ≈  3.6 GB   + activations + overhead

The gradient and optimizer terms have collapsed from 70 GB to well under a tenth of a GB, because they scale with trainable parameters, not total parameters. Activations do not shrink as dramatically, since data still flows through the whole frozen network. But the run has gone from impossible-on-one-GPU to plausible-on-one-modest-GPU.

That is the whole reason parameter-efficient fine-tuning exists, and you can now derive it from four lines of arithmetic rather than take it on faith. When the course reaches LoRA in the customization module, this is the calculation underneath it.

Step 7 — inference, for contrast. Drop gradients and optimizer state entirely:

weights (bf16)  14 GB   +  KV cache  +  the current step's activations
weights (4-bit)  3.5 GB +  KV cache  +  the current step's activations

The KV cache grows with how many tokens you are holding in context and how many requests you are serving concurrently, and it is the term that makes serving-capacity planning interesting. Again: 12-05.

The single most useful hour you can spend on GPU literacy is deliberately causing an out-of-memory error while you are calm, so that when one happens under time pressure you recognise it in seconds. This is why this lesson sits before the first lab rather than inside it.

The provocation. In a notebook with any GPU attached, allocate progressively larger tensors until it fails:

import torch
assert torch.cuda.is_available()

blocks = []
try:
    while True:
        # 1 GiB of fp32 = 268,435,456 elements of 4 bytes
        blocks.append(torch.empty(268_435_456, dtype=torch.float32, device='cuda'))
        print(f"allocated {len(blocks)} GiB   "
              f"reserved={torch.cuda.memory_reserved()/2**30:.2f} GiB")
except torch.cuda.OutOfMemoryError as e:
    print("OOM after", len(blocks), "GiB")
    print(e)

Three things to observe, and they are the lesson:

1. The count at which it fails is lower than the card's advertised capacity. The driver, the CUDA context, and framework workspaces all take a cut before your first tensor. Advertised VRAM is a gross figure, not a budget you get to spend.
2. The error text names the numbers. A CUDA out of memory message typically reports how much was requested, how much is free, and how much the allocator has reserved. Reading those three numbers is the diagnosis. If free memory exceeds the requested amount and it still failed, you are looking at fragmentation, not exhaustion.
3. Freeing is not immediate. del blocks followed by torch.cuda.empty_cache() returns memory to the driver; deleting alone returns it to the framework's pool. In a notebook, the surest reset is restarting the runtime, because a dead reference held by an output cell or a traceback object will pin gigabytes indefinitely.

The second provocation, which is more realistic: take a working training loop and increase the batch size until it fails. This teaches the more useful lesson, that batch size is the fastest lever you have, and that the failure boundary depends on sequence length too — a batch of 8 at 512 tokens and a batch of 2 at 2,048 tokens are similar activation loads.

The fix ladder, in the order to try it:

Steps 1 through 4 are free and cost nothing but time. Reaching for step 9 before trying steps 1 through 4 is the classic beginner move.

Reading the message properly. Three distinct failures wear similar-looking errors:

That fourth row is worth internalising: not every CUDA error is a memory error, and a device-side assert about indices is a data bug that happens to surface on the GPU. Misdiagnosing it as a memory problem sends you tuning batch sizes for an hour on a problem batch size cannot touch.

Two reasons, and they pull in different directions, so it is worth being precise about how deep to go.

Reason one: it is examinable at identity level. The Software Development domain is 24% of the blueprint and includes official objective 4.4 — identify system data, hardware, or software components required to meet user needs. That objective is asking exactly the question this lesson answers: given a described need, what hardware does it require, and why. Elsewhere the blueprint covers GPU architecture generations, MIG partitioning, quantization and mixed precision, distributed training and collectives, and latency/throughput capacity planning. All of those are downstream of "what is a GPU and what fills its memory".

Reason two: it is explicitly not examinable at spec-sheet level. Candidate reports are consistent that extensive dives into GPU hardware specifications were "overkill" and did not appear [FIELD], and the course index carries that as a deliberate depth ceiling. So the calibration is: know what a GPU does, why memory is the binding constraint, and which lever fixes which symptom. Do not memorise HBM bandwidth figures, SM counts, or clock speeds. If you catch yourself learning a spec table, you are spending exam-prep time on something reported as absent.

The practical consequence is that this lesson's payload is the four-consumer memory model and the fix ladder — both of which are decision rules, which is the form the questions take.

Where it feeds forward:

If you understand only one thing from this lesson, make it this: capability in this field is bounded by memory far more often than by speed. Almost every architectural choice you will study — quantization, LoRA, activation checkpointing, paged attention, MIG, model parallelism — exists because something did not fit.

The pattern behind six of these eight is the same: the symptom appears on the GPU, but the cause is somewhere else — in the host, in the data, in the allocator, in your assumptions. Building the habit of asking "which of the four consumers grew, or is this even a memory problem" is worth more than any amount of hardware trivia.

No. The exam is a proctored multiple-choice paper; nothing in it requires you to have trained anything. For the labs in this course, a free hosted notebook tier with an attached accelerator is sufficient for everything designed to run, and the course's own design keeps GPU-dependent work modest for exactly this reason. What you cannot do on free or modest hardware is a full fine-tune of a multi-billion-parameter model — section 4 shows why in four lines of arithmetic — and the course does not ask you to. Understanding the constraint is the examinable skill; owning the hardware is not.

Because a neural network's core operation is matrix multiplication, which is millions of independent dot products, and a GPU has thousands of cores that execute the same multiply-and-add instruction on different data simultaneously. A CPU has a few dozen very clever cores optimised for running different instructions in sequence, which is the wrong shape for this work. The gap is a throughput gap, not a cleverness gap. It disappears for branchy, unpredictable, or small workloads — where the CPU is genuinely the better choice, and where the cost of copying data to the GPU exceeds the compute saved.

The sum of four things exceeded the GPU's available VRAM: model weights, activations from the forward pass, gradients, and optimizer state. Which one dominated depends on the run. If it fails immediately on loading, weights are too big. If it loads and then fails at the first step, gradients and optimizer state are the likely culprits — Adam alone stores roughly two extra values per trainable parameter. If it fails a few steps in, or only at longer sequence lengths, activations are the term that grew. A fifth possibility is that nothing exceeded anything and the allocator is fragmented, which the error message reveals by reporting more free memory than was requested.

Far more than you would guess from the weight size alone, if you fine-tune every parameter with Adam. The constructed estimate in section 4 puts weights at about 14 GB in bf16, gradients at another 14 GB, and Adam state at roughly 56 GB if it is kept in fp32 — around 84 GB before activations or framework overhead, which is many tens of GB beyond a single consumer card. With a parameter-efficient adapter over a 4-bit base, the same constructed arithmetic falls to roughly 3.6 GB persistent plus activations. Those are illustrative figures for showing the shape of the trade, not measurements; confirm against your own environment. The stable takeaway is the ratio, not the digits: gradients and optimizer state scale with trainable parameters, so freezing the base collapses them.

VRAM is the GPU's own memory and is the only place the GPU can compute. System RAM belongs to the CPU and holds your Python process, your dataset, and anything not explicitly moved to the device. VRAM is much smaller and much higher bandwidth. Exhausting VRAM produces an immediate, clean CUDA out of memory exception; exhausting system RAM usually gets your process killed by the operating system with no CUDA message at all — which is how you tell the two apart from the failure alone. Offloading can park some tensors in system RAM, but at a substantial speed cost, because they have to travel back over a comparatively slow link every time they are needed.

Primarily memory and speed, but not only those. Batch size does not change what each example is, and no operation mixes one example with another during a forward pass, so results are equivalent up to floating-point ordering effects. However, batch size interacts with the optimizer: the gradient of a larger batch is an average over more examples, which changes the effective noise in each update and therefore interacts with the learning rate. Gradient accumulation exists precisely to decouple these — it lets you keep a large effective batch for optimisation purposes while only holding a small one in memory at a time. For the purposes of this exam, the decision rule is what matters: batch size is your fastest memory lever, and gradient accumulation is how you use it without changing the optimisation you intended.

Five stages. A batch is copied into GPU memory. The forward pass sends it layer by layer through the model, keeping each layer's activations because the backward pass will need them — memory climbs throughout. The loss compares the final output with the true next tokens, producing one scalar. The backward pass walks in reverse computing a gradient for every parameter, consuming activations as it goes; peak memory usually lands here. Finally the optimizer combines each gradient with its stored state and updates every weight. Then it repeats, thousands of times, and the loss curve trending downward is your only evidence anything is working.

1. A GPU is a throughput machine. Thousands of simple cores doing the same arithmetic on different numbers. Matrix multiplication is precisely that shape, which is why deep learning lives here.
2. A CPU is not a failure case. Tokenisation, chunking, evaluation scripts and small-corpus retrieval are good CPU work; moving them to a GPU can be slower. Knowing when acceleration does not pay is an examinable judgement.
3. Four things fill GPU memory during training: weights, activations, gradients, optimizer state. Memorise this list; it answers most capacity questions you will be asked.
4. Inference drops two of the four. No gradients, no optimizer state — which is why serving fits on hardware that could never train the same model.
5. Precision is a straight multiplier. fp32 is 4 bytes per number, bf16 is 2, int8 is 1. The same model can be an 8× different memory problem before you change anything else.
6. Adam's state is roughly twice the trainable parameter count, often kept in fp32. This is the term that turns "the model loaded fine" into an OOM one step later.
7. Gradients and optimizer state scale with trainable parameters, not total parameters. That single sentence is the entire argument for parameter-efficient fine-tuning, and section 4 derives it.
8. Advertised VRAM is gross, not net. Driver, context and workspaces take a cut. Budget with headroom.
9. Provoke an OOM on purpose, once. Then read the three numbers in the message — requested, free, reserved — and you will diagnose real ones in seconds. If free exceeds requested, it is fragmentation, and a runtime restart is the fix.
10. Work the fix ladder in order: restart, batch size, sequence length, gradient accumulation, mixed precision, activation checkpointing, PEFT, quantization, bigger hardware. The first four are free.
11. Not every CUDA error is a memory error. A device-side assert about indices is a data bug; a "no kernel image" error is an environment bug. Classify before you tune.
12. Depth ceiling, deliberately. Know what a GPU does and why memory binds you. Candidate reports say spec-sheet detail did not appear [FIELD]; spend that time on tool identity and decision rules instead.

You now know what a GPU does, what fills its memory, and how to read the error it raises when you ask for too much. What you do not have yet is somewhere to run any of it. That is a fifteen-minute problem with a few genuinely non-obvious traps — accelerator runtimes that are not attached by default, sessions that reset and take your files with them, quotas that change without notice, and dependency installs that vanish the moment the runtime restarts.

Next: M0.2a Setting up Google Colab and Jupyter notebooks — choosing between a hosted notebook and a local install, attaching and verifying an accelerator, the verification cell to run before you trust anything, what survives a restart and what does not, and the cost and quota realities to plan around.