Quantization: PTQ vs QAT, Calibration, and Recovering Lost Accuracy

Quantization replaces each real-valued number x with an integer q plus a small amount of shared metadata, such that x can be approximately reconstructed. The standard affine form is:

q = round(x / scale) + zero_point        # quantize
x̂ = (q - zero_point) × scale             # dequantize
error = x̂ - x                            # the rounding error you accepted

scale is a floating-point number that sets how much real-valued magnitude one integer step represents. zero_point is an integer offset that lets the grid be positioned asymmetrically around zero. Two variants matter:

• Symmetric quantization sets zero_point = 0. The grid is centred on zero, so scale = max(|x|) / 127 for signed INT8. Cheaper arithmetic, natural fit for weights, which are roughly zero-centred.
• Affine (asymmetric) quantization allows a nonzero zero_point, so the grid can cover a lopsided range such as a post-ReLU activation that is never negative. Slightly more arithmetic, better utilization of the 256 available levels when the distribution is skewed.

The second axis is granularity — how many values share one scale:

Finer granularity is the single most effective, lowest-effort accuracy recovery available, because it stops one outlier value from stretching the scale for thousands of well-behaved neighbours. If a question asks how to improve INT8 accuracy without retraining, "use per-channel instead of per-tensor scales" is a correct and often-keyed answer alongside "improve the calibration data."

The third axis is what gets quantized:

• Weight-only quantization stores weights at low precision and dequantizes them on the fly into 16-bit arithmetic. Saves memory and memory bandwidth; the math still happens at 16 bits. This is the dominant approach for LLM decoding, because decoding is bandwidth-bound.
• Weight-and-activation quantization (sometimes called full integer quantization) also quantizes the activations so the matmul itself runs on integer tensor cores. Bigger throughput win, harder to do without accuracy loss, because activation distributions vary per input and contain outliers.
• KV-cache quantization compresses the per-request attention cache. On long-context workloads this can matter more than the weights, for reasons 12-05 computes.

L1 — Intuition: fit the ruler afterwards, or teach the model to expect a coarse ruler

PTQ is measuring. You take a trained model, push a few hundred representative examples through it, watch how large the activations get at each layer, and choose scale factors that cover the observed range without wasting levels on magnitudes that never occur. Nothing is retrained. The whole procedure takes minutes and needs no labels — only inputs.

QAT is rehearsing. You put the rounding operation into the training graph, so during the forward pass every quantized tensor is rounded to the integer grid and then dequantized before the next layer. The loss the model sees is therefore the loss it will actually suffer at deployment, and gradient descent adjusts the weights to minimize it. The model learns to place its weights where rounding hurts least — for example, away from decision boundaries that a coarse grid would straddle.

The metaphor that sticks: PTQ finds the best possible ruler for a house that is already built. QAT builds the house knowing the ruler will be coarse.

L2 — Mechanism: calibration, clipping, and the straight-through estimator

PTQ's calibration pass. For each tensor that will be quantized, the toolchain records a statistic over the calibration data and derives a scale. The main strategies:

Notice what all four are doing: choosing where to clip. Quantization error has two components — rounding error inside the range, and clipping error outside it — and they trade against each other. A wider range means less clipping and coarser rounding; a narrower range means finer rounding and more clipping. Calibration is the search for that balance, and it is why calibration data must resemble production traffic. Calibrate a customer-support model on Wikipedia text and your activation ranges are fitted to the wrong distribution.

Calibration typically needs only a few hundred to a couple of thousand unlabelled examples. It does not need labels, does not need gradients, and must not include your test set — using eval data for calibration is a leakage pattern in the same family as the ones 01-07 names.

QAT's fake quantization and the gradient problem. QAT inserts "fake quant" nodes: quantize then immediately dequantize, in floating point. The forward pass therefore experiences the exact rounding the deployed model will experience, while everything stays differentiable-shaped. But round() has a derivative of zero almost everywhere, so a naive backward pass would deliver no gradient at all. The fix is the straight-through estimator (STE): in the backward pass, pretend the rounding operation was the identity function and pass the gradient through unchanged (usually zeroed outside the clipping range). STE is a deliberate approximation — it is wrong about the true derivative and it works anyway, which is why it is worth knowing by name.

QAT is normally run as a short fine-tune from the trained FP checkpoint, not as training from scratch: a small number of steps at a reduced learning rate, often with the scale factors themselves learnable. Starting from a converged model is what makes QAT affordable. Learned scale factors (learned step size) are a refinement worth recognizing but not worth memorizing in detail.

L3 — Why LLMs are harder to quantize than vision models, and the accuracy-recovery ladder

Transformer activations contain systematic outliers: a small number of hidden dimensions carry values far larger than the rest, consistently, across inputs. Per-tensor activation quantization is devastated by this, because the scale is stretched to cover the outlier channels and every ordinary channel is left with a handful of usable levels. Recognizing that this is a structured problem rather than random noise explains the family of techniques built around it: keeping outlier channels in higher precision, mathematically migrating activation scale into the weight matrix so the activation range shrinks, and per-channel or per-group weight scaling that lets the well-behaved channels keep their resolution.

Which parts of an LLM are most and least quantization-tolerant is a useful hierarchy:

The accuracy-recovery ladder, in the order you should climb it — cheapest and least invasive first:

1. Re-run the eval set and quantify the loss. You cannot recover what you have not measured. Report per-slice, not just the aggregate.
2. Improve the calibration data. Make it representative, and large enough. Frequently the whole problem.
3. Change the calibration method. Min-max to percentile or entropy-based clipping.
4. Go finer-grained. Per-tensor to per-channel to per-group.
5. Exclude the sensitive layers. Keep the LM head, embeddings, and any layer your sensitivity analysis flags at 16-bit. Mixed-precision quantization is normal, not a compromise.
6. Back off the bit width. INT4 to INT8, or INT8 to FP8 where supported.
7. Do QAT. Now you are spending a training run, so this is where the ladder gets expensive.
8. Accept it and choose a different base model. Sometimes a smaller model at 16-bit beats a larger model at INT4 on your task, and that comparison is a legitimate experiment.

Steps 1–5 are cheap and resolve most cases. Reaching straight for QAT when the real problem is 128 calibration samples of the wrong distribution is the most common waste of effort in this area.

Two sub-distinctions inside PTQ are worth carrying:

"Dynamic quantization needs no calibration data" is a clean fact that distractors like to invert.

A constructed scenario, with every figure derived from stated assumptions rather than measured. A decoder-only model: 7.0 × 10⁹ parameters, 32 layers, hidden size 4096, 32 attention heads, currently served at BF16 on a device with 24 GB usable memory.

Step 1 — the starting budget.

Weights at BF16:  7.0e9 × 2 bytes            = 14.0 GB
Runtime + workspace (assumed)                =  1.5 GB
Remaining for KV cache                       = 24.0 - 15.5 = 8.5 GB

Step 2 — the INT8 weight-only budget.

Weights at INT8:  7.0e9 × 1 byte             =  7.0 GB
Scale metadata, per-channel:
  assume ~7 large matrices per layer × 32 layers × 4096 channels
  ≈ 917,504 scales × 4 bytes (FP32)          ≈ 0.004 GB   (negligible)
Runtime + workspace                          =  1.5 GB
Remaining for KV cache                       = 24.0 - 8.5 = 15.5 GB

The KV-cache budget went from 8.5 GB to 15.5 GB — an 82% increase in the memory available for concurrent requests, from a change that touched no model logic. Note how small the scale metadata is: fewer than a million FP32 scales against seven billion weights. This is why per-channel granularity is essentially free and why "the scale factors cost too much memory" is not a real objection.

Step 3 — what did NOT change. The KV cache is still BF16. Activations are still BF16. If you told your capacity planner "quantizing to INT8 cut our memory by 4×," you would be wrong: total memory went from 15.5 GB of non-cache footprint to 8.5 GB, a 45% reduction in that term and 0% in the cache term. Always quote the term you changed.

Step 4 — the eval re-run, and how to read it. Suppose your frozen 100-item eval set (built as in 01-08, scaled as in 09-01) is scored on a faithfulness rubric and on exact-match for a structured-output slice. Constructed results:

Two lessons are embedded in those numbers. First, the aggregate score moved 7 points in the naive configuration but the structured-output slice moved 23 — the damage was concentrated, and an aggregate-only report would have understated it badly. This is why per-slice evaluation is the rule, a point 09-13 and 13-04 both press. Second, the entire gap was closed by better granularity and better calibration data. No QAT was required. That is the typical outcome, and the reason QAT sits at step 7 of the ladder rather than step 2.

Step 5 — the decision. The per-channel configuration loses 0.01 overall and 0.02 on structured output while freeing 7 GB. Whether that is acceptable is a product decision, not an engineering one, and the right artifact to produce is the table above rather than a verbal reassurance.

Quantization sits under objective 4.1 (deployment and evaluation of model scalability, performance, and reliability) and the study guide's suggested readings name TensorRT and INT8 quantization-aware training explicitly — QAT is called out by name in the source material, which is a strong signal that the PTQ/QAT distinction is testable. It also touches 4.4 (identifying the hardware and software components needed to meet user needs), because bit width determines what hardware a model can run on.

The exam tests this at identity-and-when-to-use depth, consistent with field reports that deep numeric detail did not appear. You should be able to answer instantly: which method needs retraining, which needs calibration data, which recovers more accuracy, and which you try first.

Common phrasings:

• "A team must deploy an INT8 model but cannot retrain. Which technique applies?" — post-training quantization with calibration.
• "PTQ at INT8 costs more accuracy than the product can accept and retraining is available. What next?" — quantization-aware training.
• "What is the purpose of the calibration dataset in INT8 quantization?" — to determine activation ranges / scale factors, not to fine-tune weights.
• "Which of these reduces model memory footprint by roughly 4× relative to FP32?" — INT8.
• "After quantizing a model to INT8, what must the team do before release?" — re-run the evaluation suite; quantization changes model outputs.
• "What is the primary trade-off introduced by quantization?" — accuracy against throughput and memory.

Distractor families:

PTQ quantizes an already-trained model without any gradient updates. It runs a calibration pass over a few hundred to a few thousand unlabelled inputs, records the range of values at each tensor, derives scale factors, and writes out a low-bit model — minutes of work, no training data required beyond representative inputs. QAT inserts simulated ("fake") quantization into the training graph so that the forward pass suffers the real rounding error, uses the straight-through estimator to push gradients back through the non-differentiable rounding operation, and fine-tunes the weights so they tolerate the coarse grid. QAT typically recovers more accuracy, especially at aggressive bit widths like INT4, but it costs a training run and requires labelled training data and access to the training code. The operating rule: always try PTQ first, and escalate to QAT only when PTQ's measured accuracy loss is unacceptable and you have exhausted the cheap recovery options.

Because an INT8 value carries no information about its own magnitude. To reconstruct a real number from an integer in [-128, 127] you need a scale factor, and to choose a good scale factor you must know how large the values in that tensor actually get. Weights are available for inspection directly — you can compute their range statically. Activations are not: they depend on the input, so the only way to learn their range is to run representative inputs through the model and observe. That observation pass is calibration. It needs inputs but not labels, it needs data that resembles production traffic, and it must not reuse your evaluation set. The calibration method then decides where to clip: min-max never clips but wastes resolution on outliers, while percentile and entropy-based methods clip deliberately to buy finer resolution for the bulk of the distribution.

It always introduces rounding error, but the measured effect on task accuracy ranges from undetectable to severe depending on bit width, granularity, calibration quality, and which layers you touched. Well-executed INT8 on many models lands close enough to the FP32 or FP16 baseline that it ships — NVIDIA's own material on INT8 and QAT frames the goal as reaching accuracy comparable to FP32, which is a claim about what good practice achieves, not a guarantee for an arbitrary model. INT4 is a different regime and normally requires per-group scales and often QAT. The unavoidable discipline is the same in every case: quantization is a behaviour change, you re-run the eval set, and you report per-slice results because the damage concentrates rather than spreading evenly.

Exactly the memory of the tensors you quantized, and nothing else. Weight-only INT8 from FP32 cuts the weight term by 4× and the activation, KV-cache, and runtime terms by 0×. In the constructed 7B example above, moving weights from BF16 to INT8 took the non-cache footprint from 15.5 GB to 8.5 GB — a 45% cut in that term — while the KV cache was untouched and simply gained the freed headroom. The scale metadata is negligible: under a million FP32 scales against seven billion weights. To predict a real deployment, write the four terms out separately — weights, KV cache, activation workspace, runtime overhead — and apply the reduction only to the term you changed. 12-05 supplies the KV-cache formula and 12-09 turns the whole budget into a cost figure.

Yes, and on long-context workloads it can be the more valuable intervention, because cache size grows with batch size and sequence length while weight size does not. It is a genuinely separate decision, though, for two reasons. First, the error mechanism differs: cache quantization error accumulates over the generation as each step attends to previously quantized keys and values, so the sensitivity can depend on output length in a way weight quantization does not. Second, the eval design differs: you need a long-context and long-generation slice to see the effect at all, and a short-answer eval set will report no problem where a real one exists. Treat it as its own experiment with its own before-and-after table.

It is a precision reduction, and in practice it is handled by the same toolchains, but it is mechanically simpler than integer quantization because FP8 has an exponent field. A format with an exponent adapts to magnitude natively, so it is far less dependent on a carefully fitted per-tensor range than INT8 is — the structured activation outliers that wreck per-tensor INT8 are much better tolerated. FP8 still typically uses a per-tensor scaling factor to centre the values in the format's sweet spot, so calibration-like machinery does not vanish entirely. The practical distinction for the exam: INT8 is an integer grid plus a scale factor requiring calibration; FP8 is a floating-point format with a native exponent, available on newer GPU generations. Which generations support which format is version-sensitive and should be confirmed against current documentation rather than memorized.

• Quantization = integer grid + scale factor. Three axes: symmetric vs affine, granularity, and which tensors you touch.
• PTQ needs calibration data and no gradients. QAT needs training data and gradients. That one sentence answers most exam questions here.
• Calibration fits activation ranges, which cannot be known statically because they depend on the input. Weights can be inspected directly.
• Calibration is a choice about where to clip, trading clipping error against rounding error. Min-max, percentile, and entropy-based methods differ only in that choice.
• QAT works via fake-quant nodes plus the straight-through estimator, usually as a short fine-tune from a converged checkpoint.
• Transformers have structured activation outliers, which is why per-tensor activation quantization fails on LLMs and per-channel or per-group scaling is the standard remedy.
• Climb the recovery ladder in order: measure → calibration data → calibration method → granularity → exclude sensitive layers → bit width → QAT. QAT is step 7, not step 2.
• Report per-slice eval results. Quantization damage concentrates in specific capabilities such as structured output and long context.
• Quote only the memory term you changed. Weight-only INT8 does nothing for the KV cache.
• A quantization is a quality intervention. Re-run the eval set, version the artifact with its calibration set and method, and keep the FP baseline available for rollback.

Next: 12-03 turns to the other place numerical choices show up as visible curves. Before you can trust any quantized or fine-tuned checkpoint you have to be able to read the training run that produced it — what a healthy loss curve looks like, what a diverging one looks like, and what the widening gap between training and validation loss is telling you about the model you are about to serve.