Numeric Precision Explained: FP32, TF32, FP16, BF16, INT8, and FP8

Numeric precision is the encoding contract for every number in the model. A transformer holds three families of numbers at serving time: weights (fixed after training), activations (computed per token, transient), and the KV cache (activations deliberately kept around, covered in 12-05). Each family has its own dtype, and they need not match. A common production configuration stores weights at INT8 or FP8, runs activations at FP16 or BF16, and accumulates matrix-multiply partial sums in FP32 — three precisions inside one forward pass.

Two independent properties are encoded in those bits, and confusing them is the most common conceptual error on this topic:

• Dynamic range — how large and how small a magnitude the format can represent before it overflows to infinity or underflows to zero. Range is controlled by the exponent bits.
• Resolution (precision proper) — how finely the format can distinguish two nearby values. Resolution is controlled by the mantissa (significand) bits.

A 16-bit format has 16 bits to spend. It can buy range or it can buy resolution. It cannot buy both, and the two 16-bit formats you must know — FP16 and BF16 — made opposite purchases. That single sentence answers most exam questions on this topic.

Integer formats such as INT8 work differently again. An INT8 value has no exponent; it is a plain signed integer in [-128, 127]. To represent a real-valued weight you must attach a scale factor (and sometimes a zero-point) that maps the integer grid onto the real range. The scale is metadata carried alongside the tensor, which is why integer quantization is a procedure with calibration data attached, not merely a cast. That procedure is 12-02.

L1 — Intuition: a fixed bit budget spent on two competing goods

Think of a floating-point number as scientific notation in binary: a sign, an exponent, and a set of significant digits. 1.0110 × 2^13 has four significant binary digits and an exponent of 13. Give the exponent more bits and you can write astronomically large and vanishingly small numbers. Give the significand more bits and you can write numbers close together. With 32 bits you can afford to be generous with both. With 16 you must choose, and with 8 you are choosing under real duress.

Neural networks turn out to care far more about range than about resolution during training. Gradients late in training are tiny; if the format cannot represent 1e-8, that gradient becomes exactly zero and the corresponding weight stops learning. Losing a couple of significant digits, by contrast, is absorbed by the fact that the network is an averaging machine trained with noisy stochastic gradients anyway. This asymmetry is the whole argument for BF16 and the whole reason TF32 exists.

L2 — Mechanism: the bit layouts side by side

Read the exponent column top to bottom and the story is immediate. FP32, TF32, and BF16 all have 8 exponent bits, so all three cover approximately the same magnitude span. FP16 has 5, so its representable range is dramatically narrower — its largest finite value is 65,504, a number an unscaled loss or an attention logit sum can genuinely exceed. FP16's overflow risk is not theoretical; it is the reason mixed-precision training needed an extra mechanism.

That mechanism is loss scaling. In FP16 mixed-precision training you multiply the loss by a large constant before the backward pass so that small gradients land inside FP16's representable band, then divide the gradients by the same constant before the optimizer step. Dynamic loss scaling adjusts the constant automatically, backing off when it detects an overflow. BF16 mixed-precision training typically does not need loss scaling at all, because BF16's 8 exponent bits already reach where the gradients live. "BF16 removes the need for loss scaling" is a compact, high-value fact.

The second mechanism that makes 16-bit training work is the FP32 master copy. Weights are kept in FP32; a 16-bit copy is used for the forward and backward passes; the optimizer applies updates to the FP32 master. Without this, tiny updates repeatedly round to nothing against a large weight value and training stalls. Automatic mixed precision (AMP) in PyTorch and the equivalent in other frameworks is exactly this bookkeeping, automated.

L3 — Tensor cores, accumulation, and why TF32 is a compute mode

Tensor cores are fixed-function matrix-multiply units. They consume low-precision inputs and, critically, accumulate in a higher precision — FP16 or BF16 inputs with FP32 accumulation is the standard arrangement. This is why a 16-bit matmul is not as inaccurate as you would fear: the multiply happens at 16 bits but the long summation across the reduction dimension, which is where error compounds, happens at 32. When you see "FP16 with FP32 accumulate," that is what it means.

TF32 is the format that confuses people most, so state it precisely: TF32 is not a storage dtype. Tensors stay in memory as FP32. When a matmul is dispatched to the tensor cores in TF32 mode, the inputs are internally rounded to a 19-bit form — sign, 8 exponent bits, 10 mantissa bits — multiplied, and accumulated back into FP32 outputs. Your code still declares FP32; your memory footprint is still FP32; only the inner multiply loses mantissa bits. TF32 arrived with the Ampere generation of tensor cores and is enabled or disabled by a framework flag rather than by changing your model's dtype. If an exam option describes TF32 as "a 32-bit storage format that halves memory," that is wrong on both counts — TF32 does not reduce memory, because it is not how the tensor is stored.

FP8 arrived with the Hopper generation and comes in the two encodings named above, E4M3 and E5M2, so that a workload can pick range-leaning or resolution-leaning behaviour per tensor. The conventional split — E4M3 for forward-pass weights and activations, E5M2 for gradients, which need the wider range — is worth remembering as a pattern rather than as a rule. Availability of any given precision depends on the GPU generation and on the software stack version, so treat "which GPU supports which format" as version-sensitive and confirm against current documentation rather than memorizing a matrix that will age. What does not age is the identity of each format.

Three distinctions carry most of the exam value:

FP16 vs BF16. Same size, opposite trade. FP16 keeps resolution and sacrifices range, so it needs loss scaling and is prone to overflow. BF16 keeps range and sacrifices resolution, so it trains more robustly out of the box. Large-model pretraining converged on BF16 for exactly this reason. If a question asks which 16-bit format has "the same dynamic range as FP32," the answer is BF16.

TF32 vs FP32. Compute mode versus storage format. TF32 speeds up matmuls without changing your dtypes or your memory bill. It is the only entry in this table that gives you throughput without giving up bytes.

INT8 vs FP8. Both are one byte. INT8 is a uniform integer grid that requires a scale factor and, for post-training quantization, a calibration pass over representative data. FP8 is a floating-point format with an exponent, so it handles a wide magnitude spread natively and needs less careful per-tensor range fitting. INT8 has broader hardware and toolchain support; FP8 is the newer path and is generation-dependent.

The following is a constructed scenario, not a measured benchmark. Assume a decoder-only model with exactly 7.0 × 10⁹ parameters. Weight memory is parameter count times bytes per parameter. Using decimal GB (10⁹ bytes) throughout, as M0.4 establishes:

FP32:  7.0e9 params × 4 bytes = 28.0e9 bytes = 28.0 GB
BF16:  7.0e9 params × 2 bytes = 14.0e9 bytes = 14.0 GB
FP16:  7.0e9 params × 2 bytes = 14.0e9 bytes = 14.0 GB
FP8:   7.0e9 params × 1 byte  =  7.0e9 bytes =  7.0 GB
INT8:  7.0e9 params × 1 byte  =  7.0e9 bytes =  7.0 GB
INT4:  7.0e9 params × 0.5 byte=  3.5e9 bytes =  3.5 GB

Now add the overheads that decide whether it actually fits.

Inference at BF16. Weights are 14.0 GB. You also need the KV cache, activation scratch space, and the CUDA context. Suppose the deployment target has 24 GB of usable device memory. Weights leave 10.0 GB, minus perhaps 1–2 GB of context and workspace, so roughly 8–9 GB for the KV cache. Whether that is generous or fatal depends entirely on the per-token cache cost, which 12-05 computes. At FP32 the same model needs 28.0 GB of weights alone and does not fit at all on a 24 GB card — the model becomes deployable purely by choosing BF16.

Inference at INT8 weight-only. Weights drop to 7.0 GB. On the same 24 GB card you now have roughly 15–16 GB for KV cache and workspace, close to doubling the concurrency you can serve. Note the phrase weight-only: activations and the KV cache are still typically 16-bit, so total memory does not fall by 4×, only the weight term does. Overstating the saving is a classic estimation error.

Training at mixed precision. Training memory is not just weights. A rough accounting for FP16 mixed precision with the Adam optimizer, per parameter:

FP16 working copy of weights          2 bytes
FP32 master copy of weights           4 bytes
FP32 Adam first moment  (momentum)    4 bytes
FP32 Adam second moment (variance)    4 bytes
FP16 gradient                         2 bytes
                                    ---------
                                     16 bytes per parameter

For 7.0 × 10⁹ parameters that is 112 GB before a single activation is stored. This is the arithmetic that makes full fine-tuning of a 7B model impossible on a single consumer GPU and makes LoRA the default — the same calculation 11-04 and 11-05 build on. Notice the shape of the result: switching the working copy to 16 bits saved 2 bytes out of 16, about 12%, not 50%. Precision reduction is a much bigger win at inference than at training, because training keeps FP32 state per parameter regardless.

The last row is the one to internalize. Reducing precision without an eval set is not an optimization; it is an unmeasured change to model behaviour.

Numeric precision sits directly under official objective 4.1 (assist in deployment and evaluation of model scalability, performance, and reliability) and 4.4 (identify system data, hardware, or software components required to meet user needs), and it is a named prerequisite for the quantization and mixed-precision material the study guide lists explicitly. The blueprint's suggested readings include TensorRT and INT8 quantization-aware training, which cannot be discussed without the format table above.

Calibration matters here. Field reports converge on the finding that the exam asks general-level identity and when-to-use questions, not spec-sheet recall. You are far more likely to be asked "which 16-bit format preserves FP32's dynamic range" than "what is the maximum representable FP16 value." Learn the identity of each format and the one property that distinguishes it; do not memorize hardware specification sheets, which candidate reports describe as overkill.

Question phrasings that appear in this family:

• "A team's FP16 training run produces NaN losses after several thousand steps. What is the most likely cause and the standard remedy?" — narrow FP16 exponent range causing overflow; dynamic loss scaling.
• "Which precision offers the same dynamic range as FP32 while halving memory?" — BF16.
• "Enabling TF32 on an existing FP32 workload will primarily…" — increase matmul throughput without changing memory footprint.
• "A 13B-parameter model must be served on a GPU that cannot hold it at FP16. Which change addresses this most directly?" — reduce weight precision (INT8/FP8), not batch size, not context length.
• "Which of the following requires a calibration dataset?" — post-training INT8 quantization.

Distractor families to recognize:

FP16 and BF16 both occupy 16 bits but split them differently. FP16 uses 5 exponent bits and 10 mantissa bits; BF16 uses 8 exponent bits and 7 mantissa bits. Because BF16's exponent field is the same width as FP32's, BF16 covers approximately the same range of magnitudes as FP32 and therefore rarely overflows or underflows in practice — which is why BF16 mixed-precision training usually needs no loss scaling. FP16 has three more mantissa bits, so it distinguishes nearby values more finely, but its largest finite value is 65,504 and its underflow threshold is comparatively high, so gradients and intermediate sums can fall off either end. Practical rule: for training stability prefer BF16; for pure inference either is usually fine, and the deciding factor is what your hardware and runtime support best.

No. FP32 is an IEEE storage format: 4 bytes per value, 8 exponent bits, 23 mantissa bits. TF32 is an NVIDIA tensor-core compute mode introduced with the Ampere architecture. Tensors remain stored as FP32; when a matrix multiply runs in TF32 mode, the inputs are internally rounded to a 19-bit representation with 8 exponent bits and 10 mantissa bits, multiplied on the tensor cores, and accumulated back into FP32. The consequences are: matmuls get faster, memory footprint is unchanged, and results differ slightly from a strict FP32 matmul. TF32 is toggled by a framework or library flag, not by changing your model's dtype. Any statement that TF32 "reduces model size" or "is a 19-bit storage format" is incorrect.

It can, and the honest answer is that you have to measure it on your own eval set. INT8 replaces a floating-point value with an integer on a uniform grid plus a scale factor, so values that sit between grid points are rounded and outlier activations can be clipped. Well-executed INT8 for many models lands close enough to the FP32 or FP16 baseline to be shipped, which is exactly why the technique is standard — but "close enough" is a property of your model, your calibration data, and your task, not a property of INT8. Two things make the difference: choosing per-channel rather than per-tensor scales where the toolchain supports it, and calibrating on data that matches production traffic. If post-training quantization does not recover the accuracy you need, quantization-aware training is the next step, and that is the subject of 12-02.

Loss scaling multiplies the loss by a large constant before backpropagation so the resulting gradients land inside FP16's representable band, then divides the gradients by that same constant before the optimizer applies them. It is needed because FP16's 5 exponent bits cannot represent the very small gradient magnitudes that appear later in training; without scaling, those gradients round to exactly zero and the corresponding weights stop updating. Dynamic loss scaling raises the constant while steps succeed and backs it off when an overflow is detected, skipping the affected step. BF16 does not need this machinery because its 8 exponent bits already reach the relevant magnitudes — one of the strongest practical arguments for BF16 over FP16 in training.

The KV cache is usually held at the same precision as the activations — FP16 or BF16 — because it is written and read every decoding step and any conversion cost is paid per token. Quantizing the KV cache to 8 bits is a real technique and a real lever, because on long-context workloads the cache can rival or exceed the weights in size. It is also a distinct decision from quantizing weights: the cache carries per-request state, so cache quantization error interacts with generation length in a way weight quantization does not. Treat KV-cache precision as a separate experiment with its own eval run. The sizing arithmetic that tells you whether it is worth doing is in 12-05.

Because single-request LLM decoding is limited by memory bandwidth, not by arithmetic. To generate one token the GPU must read essentially every weight in the model out of HBM, do a small amount of math per weight, and move on. Halving the bytes per weight halves the bytes that must be streamed, which nearly halves the time floor for that step. This is why weight-only quantization improves latency even when the arithmetic still happens at 16 bits, and it is the mechanism that makes 12-05 the pivotal lesson of this module. In the prefill phase, where a whole prompt is processed at once and the arithmetic-to-bytes ratio is much higher, the picture flips and compute throughput becomes the limit — which is why prefill and decode need separate performance reasoning, taken up in 12-10.

• Precision sets memory, speed, and numerical error simultaneously. It is the cheapest deployment lever you have.
• Exponent bits buy range; mantissa bits buy resolution. A 16-bit format must choose.
• FP32, TF32, and BF16 all have 8 exponent bits. FP16 has 5, which is why FP16 training needs loss scaling and BF16 does not.
• TF32 is a compute mode, not a storage format. It accelerates matmuls and does not reduce memory.
• INT8 and FP8 are both one byte, but INT8 needs a scale factor and calibration data while FP8 has a native exponent.
• Weight memory is parameters × bytes per parameter. Total serving memory is weights plus KV cache plus activations plus runtime overhead — never quote the weight term alone.
• Training memory is dominated by FP32 optimizer state, so precision reduction saves far less during training than during inference.
• Every precision reduction is a quality intervention. Gate it on a frozen eval set and re-run that set, or you have shipped an unmeasured behaviour change.

Next: 12-02 turns the last takeaway into a procedure. Reducing precision to INT8 is not a cast — it is quantization, with a calibration step, two competing methods (post-training quantization versus quantization-aware training), a symmetric-versus-affine choice, and a mandatory eval re-run to prove the accuracy you did and did not keep.