Mixed-Precision Training for Multimodal Models: FP16, FP32, Loss Scaling, and Tensor Cores

Identity statement: mixed-precision training is running a neural network's forward pass, loss computation, and backward pass primarily in FP16 (half precision, 16 bits per value) instead of FP32 (single precision, 32 bits per value), while keeping a small number of specific quantities — the weight update, the loss scale, and certain accumulations — in FP32, so that the reduced-precision run reaches the same accuracy as a full-FP32 run.

When it matters: any time a scenario describes a multimodal model that does not fit in GPU memory at full precision, a training run that wants a larger batch size or a bigger model on the same hardware, or a question that asks why an FP16 run diverged.

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "Mixed-precision training uses lower-precision FP16 (half precision) arithmetic together with FP32 (single precision), cutting memory use and speeding training with minimal accuracy loss." Unpack the two effects separately, because they come from different places.

The memory effect. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "FP16 uses 16 bits vs. 32, roughly halving memory and enabling larger models/minibatches." A weight, activation, or gradient stored at FP16 occupies half the bytes it would at FP32. For a multimodal model with two or three encoders and a fusion stage, that halving applies across the whole graph — the vision tower's activations, the text tower's activations, and the fused representation all shrink together. The freed memory converts directly into either a larger batch (more images and captions per step, which stabilizes gradient estimates) or a larger model (more layers or a wider hidden dimension) on the same GPU.

The speed effect. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "NVIDIA GPUs deliver much higher half-precision throughput on math-limited layers." This is not a side effect of using fewer bytes — it is a property of the hardware itself, covered in full in section 2. A matrix multiply that runs at FP16 on a Tensor Core–equipped GPU completes in a fraction of the time an equivalent FP32 multiply would take, independent of the memory savings.

What does not change. Mixed-precision training does not change the model's architecture, its parameter count, or the mathematical function it is trying to learn. It changes only the numeric format the arithmetic runs in during training. This distinction matters because the exam's most common wrong-framing question conflates a training-time precision choice with an inference-time one — see section 6 for exactly why that conflation is a named trap.

L1 — Intuition: three specific leaks, three specific patches

FP16 has a much narrower representable range than FP32 — its smallest and largest finite magnitudes are both far closer to zero and to overflow than FP32's. Running training naively in FP16, with no other change, produces one of two failure signatures: gradients that quietly round to exactly zero and stop updating their weights, or a weight update that gets swallowed because it is too small relative to the weight it is added to. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) names three specific techniques that patch these leaks, and each technique targets exactly one failure mode rather than being a general-purpose fix.

L2 — Mechanism: master weights, loss scaling, and FP32 accumulation

FP32 master copy of weights. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "FP32 master copy of weights — updates accumulate in full precision." The model keeps two copies of every weight during training: an FP16 copy used for the forward and backward pass, and an FP32 "master" copy that the optimizer actually updates. Without this, a small gradient-derived update added to an FP16 weight can round to nothing — FP16's 10 mantissa bits give it far less resolution near a large value than FP32's 23, so a genuinely useful update simply disappears into rounding error, and the weight never moves. Keeping the master copy at FP32 means the update is applied where it has room to register, and only the resulting value is rounded back down to FP16 for the next forward pass.

Loss scaling. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "Loss scaling — multiply the loss so small activation gradients don't underflow to zero in FP16's narrower range." Gradients late in a deep network, and especially gradients flowing back through a multimodal fusion layer into an upstream encoder, are often tiny by the time backpropagation reaches them — well within FP32's range but below FP16's smallest representable nonzero magnitude. Multiplying the loss by a large constant (commonly a power of two, so the multiplication and its later reversal are numerically clean) before the backward pass shifts every gradient in the graph up by that same factor, moving them into FP16's representable band. After the backward pass computes gradients at this scaled-up magnitude, the optimizer divides them back down by the same constant before applying the update, so the final weight change is mathematically identical to what an unscaled FP32 run would have produced — the scaling only protects the values in transit through FP16's narrow range.

FP32 accumulation. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "FP32 accumulation — multiply FP16 values but accumulate products into FP32." A matrix multiply is a long sum of many products. If both the multiplication and the running sum happen in FP16, rounding error compounds across every term added, and for a wide layer — the kind a vision or text encoder's hidden dimension routinely is — that compounding is large enough to measurably corrupt the result. Multiplying two FP16 values but adding each product into an FP32 running total keeps the individual multiplications fast and low-memory while keeping the part of the computation most sensitive to compounding error at full precision. This is not a training-loop-level technique the way the other two are; it happens inside the hardware unit that performs the multiply, which is exactly what a Tensor Core is built to do.

L3 — The exam-relevant edge case: why all three are needed together, not any one alone

A question that offers "just use loss scaling" or "just keep an FP32 master copy" as a complete fix for FP16 training is offering a partial answer dressed as a whole one. Each technique closes a different, independent failure point: the master copy protects the weight update from vanishing into a large weight's rounding error; loss scaling protects the gradient from underflowing to zero before it ever reaches the optimizer; FP32 accumulation protects the matrix multiply itself from compounding rounding error across a long sum. A run using loss scaling and FP32 accumulation but no FP32 master copy can still stall, because tiny-relative-to-the-weight updates round away even though the gradient that produced them was healthy. A run with a master copy and FP32 accumulation but no loss scaling can still produce dead gradients in a deep or multimodal fusion path, because the gradients underflowed before the master copy ever got a chance to receive them. The three are a matched set, not a menu.

⭐ THE EARNED INSIGHT

FP16 training does not fail all at once — it fails at whichever one of the three specific points you left unprotected, and the failure signature tells you exactly which point that was. A loss that goes to NaN or inf points at an unscaled overflow; a loss that plateaus with gradients that look suspiciously small or zero points at underflow; a model that trains but converges to a measurably worse accuracy than its FP32 twin, with no NaNs anywhere, points at accumulation error compounding quietly rather than crashing loudly.

Halving a value's byte count is only half of why mixed precision is worth the trouble. The other half is a specific piece of NVIDIA hardware. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "Tensor Cores (introduced with the Volta architecture) multiply half-precision matrices and accumulate into FP32, giving speed with good accuracy." A Tensor Core is a specialized arithmetic unit, distinct from an ordinary CUDA core, purpose-built to perform a small fused-multiply-add matrix operation — commonly described as computing D = A×B + C in one hardware step — at much higher throughput than the same operation would achieve running through general-purpose FP32 arithmetic units. [VENDOR SPEC] (Sources/nca-genm/domain-5-performance-optimization.md): Tensor Cores take FP16 (or another supported reduced-precision format) as the multiplication inputs A and B, and accumulate the result into FP32 — which is precisely the FP32-accumulation technique from section 2, implemented in silicon rather than in a training script.

This is why "mixed precision" and "Tensor Cores" are taught as one topic rather than two adjacent ones: the software techniques in section 2 exist to make a model's numbers safe to run through this specific hardware path, and the hardware path is what turns "smaller numbers" into "faster training" rather than just "training that uses less RAM." A GPU generation without Tensor Cores can still technically run FP16 arithmetic, but it gets only the memory benefit, not the multiplied-throughput benefit that makes mixed precision the default choice for training large multimodal models today.

Treat the following as a constructed scenario, with every figure derived from stated assumptions rather than measured from a real training run. Consider a multimodal model with 1.2 billion trainable parameters, trained with the Adam optimizer, which keeps two additional per-parameter states (a running mean and a running variance of the gradient) beyond the weights and gradients themselves.

Per-parameter memory at pure FP32:
  weights                4 bytes
  gradients              4 bytes
  Adam state (2 values)  8 bytes
  total per parameter   16 bytes

Total at 1.2e9 parameters: 1.2e9 x 16 bytes = 19.2 GB

Per-parameter memory at mixed precision (FP16 forward/backward, FP32 master + Adam state):
  FP16 weight copy        2 bytes   (used in forward/backward)
  FP16 gradient           2 bytes   (produced by backward pass)
  FP32 master weight      4 bytes   (the "master copy" from section 2)
  Adam state (2 values)   8 bytes   (kept at FP32, since optimizer state is small relative to activations)
  total per parameter    16 bytes

Total at 1.2e9 parameters: 1.2e9 x 16 bytes = 19.2 GB

Notice what the arithmetic actually shows: the weight and optimizer state memory footprint does not shrink in this accounting, because the FP32 master copy and Adam state are kept at full precision regardless — this is a deliberate, honest correction to the common oversimplification that mixed precision "halves everything." What mixed precision actually halves is the activation memory, which for a multimodal model with two encoders and a fusion stage is frequently the larger term at real batch sizes, because activations scale with batch size and sequence/patch count in a way that weights do not:

Activation memory per training step at batch size 64, illustrative:
  FP32 activations across a moderately deep vision+text stack: 22 GB
  FP16 activations, same architecture, same batch:              11 GB

Freed activation memory: 11 GB, which becomes available for either
a larger batch size (more image-caption pairs per step) or a larger
model (more layers, wider hidden dimension) on the same GPU.

This is inference, not a citation to a real run — the point is the shape of the accounting, not the specific numbers. The lesson to generalize: mixed precision's memory win is concentrated in activations, not in the weights and optimizer state a naive "everything is now 2 bytes" assumption would expect, because sections 2's three safeguards deliberately keep several of the smaller-but-precision-critical quantities at FP32.

A team trains the same multimodal model twice, both times switching from FP32 to FP16 for speed, and compares against the FP32 baseline's loss curve.

FP32 baseline:      loss falls smoothly from 4.10 to 1.35 over 10,000 steps.

Run A (FP16, no loss scaling): loss falls from 4.10 to 3.95 over
  2,000 steps, then plateaus and does not move for the remaining
  8,000 steps.

Run B (FP16, loss scaling enabled, FP32 master weights, FP32
  accumulation): loss falls from 4.10 to 1.34 over 10,000 steps,
  matching the FP32 baseline within normal run-to-run variance.

Run A's signature — an early partial drop followed by a hard plateau, with no NaN or inf anywhere in the log — is the specific fingerprint of gradient underflow: gradients that were never large enough to represent in FP16 rounded to exactly zero early in training, once the loss landscape flattened out enough that gradient magnitudes shrank below FP16's representable floor, and every parameter downstream of that point simply stopped receiving a nonzero update. Run B, with all three section-2 safeguards active, tracks the FP32 baseline because none of the three failure points was left open. Constructed scenario — the exact loss values are illustrative, not measured — but the qualitative shape (early progress, then a flat plateau with no error, versus a curve that tracks the FP32 baseline) is the pattern the exam's scenario questions describe when they ask you to diagnose a stalled FP16 run.

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) names two specific misconceptions directly, and both are worth holding as separate, precise claims rather than a vague sense that "FP16 is risky."

Mixed precision and Tensor Cores are not unique to multimodal training — the concept also appears on the NVIDIA generative-AI LLM-operations track, in a lesson called "Mixed precision and Tensor Cores," and it is worth being explicit about why the two lessons cover genuinely different ground rather than the same facts twice. That LLM-serving lesson sits inside a domain about deploying and operating large language models at scale, and its framing is comparative-hardware and comparative-format: it is concerned with FP16 versus BF16 as the two 16-bit choices available on current NVIDIA GPU generations, with BF16's wider exponent range usually removing the need for loss scaling at all, and with which of the two formats a given hardware generation supports well.

This lesson's framing is different because Domain 5's own objectives (5.1 through 5.4) are about optimizing an already-designed multimodal training pipeline for accuracy per unit of compute, memory, and energy — the question is not "FP16 or BF16, and which GPU generation," it is "what specific mechanism keeps FP16 safe at all," which is why sections 2 and 3 above go to the level of naming the master-copy, loss-scaling, and accumulation mechanisms individually rather than treating "use 16-bit precision" as a single settled decision. A multimodal model's extra structure — two or more encoders trained under one composite loss, as M1-08 and M1-09 cover — is also where this lesson's edge cases live: which branch of a fused model is most likely to produce an underflowing gradient first is a multimodal-specific question that a single-tower LLM-training lesson has no occasion to raise. Read both lessons and the shared vocabulary (FP16, loss scaling, Tensor Cores) reinforces rather than repeats, because each is answering a different question with it.

Performance Optimization is Domain 5 of the NCA-GENM blueprint at 10% weight, and the domain's own scope note frames the whole module at foundational depth: recognize each technique, what it trades off, and when to reach for it, not derive the numerics by hand. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) opens the domain by naming mixed-precision training as one of the concrete techniques — alongside quantization, pruning, hyperparameter tuning, and transfer learning — through which performance optimization "refines multimodal AI models for energy efficiency, trustworthiness, and accuracy." Mixed precision sits first in that list because it is the technique with the fewest moving parts and the most immediate payoff: a training run that fits at FP16 and did not fit at FP32 is deployable purely from a dtype change, no architecture work required.

Questions in this family tend to arrive in a small number of recognizable shapes: a direct identification item naming which of the three safeguards prevents a specific failure (the self-check material's own example asks which technique prevents small FP16 gradients from underflowing, keyed to loss scaling); a scenario item describing a training run's failure signature and asking you to diagnose the missing safeguard, in the shape of section 5's worked example; and a "which statement is true" item offering one of section 6's traps as a plausible-sounding but incorrect claim. Distractors typically substitute a technique from a neighboring lesson — pruning, or grid search — as if it were one of the three FP16 safeguards, trading on the fact that all six of this module's techniques are, in the broadest sense, "ways to make training or inference more efficient," which is exactly specific enough to sound right and exactly general enough to be wrong.

Mixed-precision training reduces numeric precision during training, specifically to speed up and shrink the training run itself, while keeping an FP32 master weight copy so the final trained model is just as accurate as if it had trained purely at FP32. Quantization, covered fully in M5-02, reduces numeric precision after training is complete (or, in one variant, during a training-aware pass) specifically to shrink and speed up inference on an already-trained model. The two are easy to conflate because both involve running arithmetic at a lower bit width, but they operate at different stages of a model's lifecycle, solve different problems, and are evaluated with different metrics — a mixed-precision training run is judged by whether its final accuracy matches an FP32 baseline, while a quantized model is judged by how much accuracy it lost relative to the full-precision model it was quantized from.

Because the weight update computed by the optimizer is frequently much smaller in magnitude than the weight it is being added to, and FP16's limited mantissa resolution means a sufficiently small update, added to a sufficiently large weight, rounds to no change at all when both live in FP16. Keeping an FP32 master copy gives every update room to register at full resolution; only after the update is applied is the resulting value rounded back down to an FP16 copy for the next forward and backward pass. This is why the model still trains in FP16 for speed — the master copy is bookkeeping the optimizer maintains alongside the fast path, not a parallel FP32 training run.

Not identically, though the same three safeguards apply throughout. A multimodal model's vision encoder, text encoder, and fusion layer are all subject to the same underflow and overflow risks, but they do not necessarily hit them at the same point in training or to the same degree — a fusion layer combining two already-compressed representations, or a loss term weighted very differently from the others (the composite-loss weighting problem M1-09 covers), can produce gradients at unusual magnitudes relative to a single-modality model. Dynamic loss scaling, which adjusts its constant automatically rather than using one fixed value for the whole run, is the practical answer to exactly this variability: it does not need to know in advance which part of a multimodal graph will underflow first, only how to detect and correct for it as training proceeds. ⚠️ UNVERIFIED: the source material does not specify whether NVIDIA's own study guide treats per-branch loss-scaling behavior in multimodal models as a distinct testable point beyond the single-model case, so treat this as a reasonable extension of the general mechanism rather than a separately cited fact.

Work through each item before checking the answer key. Every option is a real claim about some training run somewhere — the task is matching it to the described scenario, not spotting an obviously fabricated distractor.

1. A team casts an entire multimodal model to FP16, changes nothing else, and training loss goes to NaN within a few hundred steps. What is the most likely cause?
   • A. The learning rate is too low.
   • B. Gradients or activations overflowed FP16's representable range.
   • C. The batch size is too large.
   • D. The evaluation set leaked into training.
2. Which technique specifically prevents small activation gradients from underflowing to zero in FP16?
   • A. FP32 master weight copy.
   • B. Loss scaling.
   • C. Structured pruning.
   • D. Grid search.
3. Why does a mixed-precision run keep an FP32 copy of the weights rather than updating the FP16 copy directly?
   • A. FP16 cannot be stored in GPU memory at all.
   • B. A small update can round to no change when added to an FP16 weight, so the update is applied to an FP32 copy instead.
   • C. FP32 weights train faster than FP16 weights.
   • D. The optimizer only supports FP32 inputs.
4. What does a Tensor Core accumulate its FP16 matrix-multiply products into?
   • A. FP16, to save memory.
   • B. FP32, even though the inputs are FP16.
   • C. INT8, for the fastest possible throughput.
   • D. BF16, on every NVIDIA GPU generation.
5. A multimodal model trains successfully at FP16 with all three safeguards in place, then is quantized to INT8 for deployment. What is true about that quantization step?
   • A. It is the same technique as mixed-precision training, applied again.
   • B. It is a separate, inference-time precision reduction that does not use loss scaling.
   • C. It requires an FP32 master weight copy at inference time.
   • D. It removes the need for the eval set to be re-run.
6. A training run plateaus early with no NaN or inf anywhere in the log, at a loss level clearly worse than the FP32 baseline. Which missing safeguard best explains this signature?
   • A. Missing FP32 accumulation, causing an early crash.
   • B. Missing loss scaling, causing gradients to underflow to zero.
   • C. Too large a batch size.
   • D. An incorrectly configured evaluation set.
7. Which statement about FP16 accuracy is correct?
   • A. FP16 always loses accuracy relative to FP32, regardless of technique.
   • B. FP16 accuracy loss can be avoided by using a larger batch size alone.
   • C. With FP32 master weights, loss scaling, and FP32 accumulation, FP16 training reaches accuracy on par with FP32.
   • D. FP16 accuracy loss can only be recovered by quantization-aware training.

Answers

1. B. No safeguards were added, so FP16's narrow range is the first place to look; NaN within a few hundred steps is the classic overflow signature, distinct from the underflow-and-plateau signature covered in section 5.
2. B. Loss scaling is the technique named specifically for underflow; the FP32 master copy protects the update step, not the gradient's representability.
3. B. This is the master-copy mechanism from section 2 — the update is computed and applied at full precision so it does not round away against a large FP16 weight.
4. B. This is the defining property of a Tensor Core named in the source material: FP16 inputs, FP32 accumulation, which is also FP32 accumulation from section 2 implemented in hardware.
5. B. Quantization is a separate, inference-time technique (the subject of M5-02); it does not use loss scaling, which is specific to training-time gradient underflow.
6. B. An early partial drop followed by a flat plateau with no error is the underflow-to-zero signature from section 5's worked example, not an overflow crash.
7. C. This restates the source material's own correction to the "FP16 always loses accuracy" trap named in section 6.

• Mixed-precision training runs FP16 for speed and memory while keeping three specific quantities at FP32: the master weight copy, the loss scale, and the accumulation inside each matrix multiply.
• Each of the three safeguards protects a different, independent failure point — an update rounding away, a gradient underflowing to zero, and compounding error inside a long sum — and all three are needed together.
• Tensor Cores are the hardware reason mixed precision is a speed technique and not just a memory technique: they multiply FP16 inputs and accumulate into FP32, at throughput an ordinary FP32 unit cannot match.
• "FP16 always loses accuracy" and "loss scaling is optional" are the two named exam traps; both are false once the three safeguards are in place.
• Mixed precision is a training-time technique. It changes nothing about how the finished model is served, which is the separate subject M5-02 picks up next.

This module now turns from making training itself cheaper to making the trained model cheaper to run. Next: M5-02 covers quantization — reducing a trained multimodal model's precision for inference, why it does not improve accuracy the way this lesson's techniques preserve it, and why quantization-aware training recovers more of that accuracy than post-training quantization by simulating the precision cut during training rather than applying it afterward.