Quantization for Multimodal Models: PTQ vs. QAT Across Separate Encoders

Identity statement: quantization is the process of reducing the numeric precision used to store a trained model's weights and, optionally, its activations — commonly from FP32 or FP16 down to INT8 — in order to shrink the model's memory footprint and speed up inference, at the cost of some accuracy that the two methods in section 2 recover to different degrees.

When it matters: any time a scenario describes a trained multimodal model that needs to run faster, fit on smaller hardware, or serve more requests per GPU, and offers quantization as one of several possible interventions.

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "Quantization reduces numeric precision of weights/activations (e.g., FP32 → INT8 or FP16) to shrink memory and speed up inference — it does not improve accuracy." That last clause is not a hedge, it is the load-bearing fact this lesson exists to fix in place. Quantization's entire value proposition is smaller and faster; whatever accuracy the full-precision model had is a ceiling quantization approaches from below, never a floor it can exceed. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "Lower precision → smaller, faster, more energy-efficient models, at some risk to accuracy (mitigated by QAT)." Read "mitigated," not "eliminated" — even the stronger of the two methods in section 2 is recovering accuracy that quantization already put at risk, not adding accuracy the full-precision model never had.

This is also where quantization is cleanly separated from M5-01's subject, even though both involve "using fewer bits." [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) names this directly as a common exam trap: "confusing mixed-precision training (a training technique) with quantization (mainly an inference optimization)." Mixed-precision training changes the arithmetic during training so the trained model is unaffected. Quantization changes the trained model's own stored numbers, after the model's accuracy has already been fixed by training — which is exactly why quantization has a genuine accuracy cost that mixed-precision training does not.

L1 — Intuition: measuring after the fact versus rehearsing in advance

Post-training quantization takes a model that finished training with no idea it would ever be quantized, and squeezes its numbers down after the fact. Quantization-aware training tells the model, during training, that a precision cut is coming, so the model's own weights settle into values that survive the cut better. One approach adapts the numbers to a fixed model; the other adapts the model to fixed, anticipated numbers.

L2 — Mechanism: when each happens and what each costs

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) gives the identity of each method directly:

PTQ takes the finished, full-precision weights and maps them onto a lower-precision grid — commonly by observing the range of values a layer actually produces and choosing a scale factor that covers that range — with no further training. It requires no labeled data, no gradient computation, and typically completes in minutes to hours. Because the model's weights were never adjusted with the coming precision cut in mind, PTQ simply accepts whatever accuracy loss the rounding introduces.

QAT inserts the quantization operation into the training graph itself, so that during training the model repeatedly experiences the rounding error it will face at deployment, and gradient descent nudges the weights toward values that tolerate that rounding — for instance, away from a decision boundary that a coarse grid would blur. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) frames the payoff plainly: QAT "simulates quantization so the model adapts" and "usually recovers more accuracy." The cost is a genuine training run — labeled data, gradient computation, and GPU time — rather than PTQ's after-the-fact pass.

L3 — The exam-relevant edge case: applying PTQ or QAT per encoder, not per model

A single-modality model gives you one obvious unit to quantize: the whole network. A multimodal model gives you a genuine choice, because its vision encoder, text encoder, and fusion layer are architecturally distinct components that were often pretrained on different data at different scales — a vision transformer trained on hundreds of millions of images behaves numerically differently from a text encoder trained on web-scale token sequences, and neither behaves like the fusion layer that was trained last, on top of both. This is inference, reasoned from the general mechanism of quantization rather than a claim traceable to a specific NVIDIA study-guide sentence naming per-encoder quantization directly: because PTQ and QAT operate on whatever tensors you point them at, nothing prevents applying PTQ to one encoder and QAT to another inside the same multimodal model, and doing so can be the more accuracy-efficient choice than treating the whole model as one quantization decision.

The practical case for this split follows directly from where accuracy risk concentrates. A vision encoder producing dense, well-distributed activations across many channels is often comparatively forgiving of a precision cut — PTQ alone recovers most of its accuracy. A component whose output feeds directly into a decision boundary that the rest of the system depends on heavily — commonly the fusion layer, or an encoder whose representations are consumed by very few downstream parameters, leaving little redundancy to absorb rounding error — is a better candidate for QAT's extra cost, because that is exactly where a coarse grid does the most damage. Reaching for QAT uniformly across every component of a multimodal model spends a training run on parts that PTQ alone would have handled adequately; reaching for PTQ uniformly risks under-protecting the one component that actually needed the stronger method. Treating the choice as one decision per encoder, rather than one decision for the whole model, is the version of "know when to reach for it" that Domain 5's foundational-level scope note is asking a multimodal-specific course to teach.

⭐ THE EARNED INSIGHT

The PTQ-versus-QAT decision is not a single global setting on a multimodal model — it is a per-component question, because "the model" is actually several separately-trained encoders joined by a fusion layer, and the component most sensitive to a precision cut is rarely the same component that is cheapest to leave at full precision. Asking "which encoder can least afford PTQ alone" is a more useful question than asking "PTQ or QAT for this model," and it is a question a single-modality model never gets to ask in the same way.

Constructed scenario, with every figure derived from stated assumptions rather than measured on a real checkpoint. A multimodal classifier pairs a vision encoder (340 million parameters) with a text encoder (110 million parameters) and a small fusion head (6 million parameters) that combines their outputs into a final prediction.

Step 1 — full-precision (FP32) baseline, weights only:
  Vision encoder:  340e6 params x 4 bytes = 1.36 GB
  Text encoder:    110e6 params x 4 bytes = 0.44 GB
  Fusion head:       6e6 params x 4 bytes = 0.024 GB
  Total weight memory:                    ~1.82 GB
  Baseline task accuracy (held-out eval set): 88.0%

Step 2 — uniform PTQ (INT8) across every component:
  Vision encoder:  340e6 params x 1 byte  = 0.34 GB
  Text encoder:    110e6 params x 1 byte  = 0.11 GB
  Fusion head:       6e6 params x 1 byte  = 0.006 GB
  Total weight memory:                    ~0.46 GB   (a 4x reduction)
  Measured accuracy after uniform PTQ:      82.5%     (5.5-point drop)

A 4x memory reduction is a genuine win, but a 5.5-point accuracy drop from one uniform pass is large enough to be a real product concern. Re-measuring per component — the diagnostic step this scenario is built to demonstrate — shows the drop is not evenly distributed:

Step 3 — per-component accuracy attribution (constructed):
  Vision encoder INT8 alone, everything else FP32: accuracy 87.6%  (0.4-point drop)
  Text encoder INT8 alone, everything else FP32:   accuracy 87.1%  (0.9-point drop)
  Fusion head INT8 alone, everything else FP32:    accuracy 83.9%  (4.1-point drop)

The fusion head — 6 million parameters, roughly 0.3% of the model's total parameter count — accounts for the large majority of the accuracy loss. This is the pattern section 2's L3 discussion predicted: a small component sitting at the narrowest point of the whole pipeline, with the least redundant capacity to absorb rounding error, is disproportionately sensitive to a precision cut that a much larger encoder tolerates easily.

Step 4 — the mixed decision: PTQ on both encoders, QAT on the fusion head only:
  Vision encoder:  INT8 via PTQ    -> 0.34 GB, ~87.6% (matches step 3's isolated result)
  Text encoder:    INT8 via PTQ    -> 0.11 GB, ~87.1% (matches step 3's isolated result)
  Fusion head:     INT8 via QAT    -> 0.006 GB, ~87.5% (recovered from 83.9%, close to FP32's contribution)
  Total weight memory:             ~0.46 GB   (same 4x reduction as step 2)
  Combined accuracy (constructed): ~86.9%     (1.1-point drop from baseline, versus 5.5 for uniform PTQ)

The memory win is identical to the uniform-PTQ pass — the same 4x reduction — but the accuracy cost shrinks from 5.5 points to roughly 1.1, because the one training run QAT required was spent on the 6-million-parameter component that actually needed it, not on the 450 million parameters of encoder weight that PTQ alone already handled well. This is a constructed illustration built to make the per-component reasoning legible; the specific percentages are not measurements from any named checkpoint, but the shape — a small, structurally sensitive component driving most of a uniform pass's damage — is the general pattern the source material's "risk to accuracy, mitigated by QAT" framing is describing at the component level rather than the whole-model level.

The worked example above generalizes into a small set of situations worth recognizing on sight, each with a default action and the reasoning behind it.

The failure modes below recur across both single-modality and multimodal quantization work, with the multimodal-specific ones concentrated in how the whole-model average can conceal a single component's damage.

Every row in that table traces back to the same root cause: treating a multimodal model's quantization as one whole-model decision instead of several separately-measurable ones.

Performance Optimization is Domain 5 of the NCA-GENM blueprint at 10% weight, and [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) names quantization as one of the domain's core techniques for getting more accuracy per unit of compute, memory, and energy out of a model you already have. The domain's foundational-level scope note asks you to recognize each technique, what it trades off, and when to reach for it — for quantization specifically, that means being able to state instantly that PTQ needs no retraining and QAT does, that QAT usually recovers more accuracy, and that neither one is a way to gain accuracy the full-precision model never had.

Questions in this family tend to arrive in a small number of recognizable shapes: a direct identification item asking what quantization primarily achieves, keyed to lower memory and latency with possible accuracy loss (the self-check material's own example, offering "higher model accuracy" as a tempting but wrong distractor); an item asking how QAT differs from PTQ, keyed to QAT simulating quantization during training so the model adapts and recovers accuracy; and a scenario item describing a multimodal model with one component performing worse after quantization, which tests whether you recognize that a uniform average can hide a component-level problem — the reasoning this lesson's sections 2 and 4 build directly. Distractors typically offer PTQ described as requiring retraining, QAT described as requiring no training data, or quantization described as a way to improve accuracy — each a real property of some other technique, misapplied here.

What the distractors typically look like

Expect an option that swaps PTQ and QAT's requirements — "PTQ requires labeled training data" or "QAT requires no gradient updates" — since the two methods' defining traits are exactly what a rushed reading confuses. Expect quantization offered as an accuracy-improving technique, trading on the general (true) fact that many optimization techniques do improve some metric, when quantization's metric is speed and size, never accuracy. And for multimodal-specific scenario items, expect an option that treats the model's overall average accuracy after quantization as the whole story, when the keyed reasoning requires noticing that the average conceals which specific component absorbed the damage.

Mixed-precision training, covered in M5-01, is a training-time technique: it runs most of a training step in FP16 while protecting three specific quantities at FP32, so the trained model reaches the same accuracy as if it had trained purely at FP32. Quantization is an inference-time technique applied to an already-trained model: it reduces the stored precision of weights and activations to shrink memory and speed inference, and it accepts some accuracy loss as the cost of doing so, mitigated but never eliminated by QAT. The two are easy to conflate because both involve running arithmetic at reduced bit width, but one preserves the accuracy a training run reaches and the other spends some of that accuracy to gain speed and size.

Nothing in the mechanism forces a single, whole-model decision, because a multimodal model is architecturally several separately-trained encoders joined by a fusion layer rather than one uniform network — this is inference from the general mechanism of quantization applied to multimodal architecture, not a claim traceable to a specific NVIDIA study-guide sentence about per-encoder policy. In practice, a vision encoder and a text encoder frequently tolerate PTQ well on their own, while a small fusion layer sitting at the narrowest point of the pipeline can be disproportionately sensitive to the same precision cut, making it the better candidate for QAT's extra cost. Measuring accuracy per component after a uniform PTQ pass, rather than trusting a single whole-model average, is the diagnostic step that reveals whether this split is worth making for a given model.

Because QAT recovers accuracy relative to what PTQ would have lost on the same model — it narrows the gap between the quantized model and its own full-precision original, it does not push the quantized model past that original's ceiling. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) states this as a standing exam trap: "quantization improves accuracy" is false; the honest framing is that quantization cuts memory and latency, accuracy can drop, and QAT is the mitigation for that drop, not a source of new accuracy the training run never produced.

Quantization also appears, under closely related names, on two other NVIDIA generative-AI tracks, and the differences are worth naming explicitly rather than leaving to chance. One sibling lesson, on the LLM-operations track, is framed around a deployment domain's largest single topic area and adds a third method, GPTQ, alongside PTQ and QAT — a one-shot, weight-only technique using approximate second-order information, aimed specifically at getting a many-billion-parameter language model down to three or four bits per weight in a few GPU hours. That lesson's decision unit is a single-tower language model, and its depth is calibration methodology: symmetric versus affine quantization, per-tensor versus per-channel versus per-group granularity, and the specific problem of systematic activation outliers in transformer hidden dimensions. A second sibling, on the associate-level LLM track, covers the same PTQ-versus-QAT identity at a similar foundational depth to this lesson, framed around a single decoder-only model's deployment and the discipline of re-running an eval set after any precision change.

Neither sibling has occasion to raise the question this lesson is built around, because neither sibling's model has more than one tower. M5-02's own objectives (5.1 through 5.4) sit inside a domain about optimizing a multimodal model you already have, and a multimodal model's defining structural fact — that it is genuinely several separately-trained encoders sharing one forward pass — is exactly what makes "PTQ or QAT, per component" a meaningful question rather than a single global setting. Where the LLM-serving lesson's depth is granularity and calibration methodology inside one network, this lesson's depth is deciding which network inside a multimodal system gets which method, and why the fusion layer joining two encoders is frequently the most sensitive point in the whole pipeline despite being architecturally the smallest. The vocabulary overlaps — PTQ, QAT, INT8, calibration — because it names the same underlying mechanism; the decision each lesson is teaching you to make does not.

Work through each item before checking the answer key. Every option is a real claim about some quantized model somewhere — the task is matching it to the described scenario, not spotting an obviously fabricated distractor.

1. A team quantizes a multimodal model to INT8 and reports it now scores higher on its held-out eval set than the FP32 original. What is the most likely explanation?
   • A. Quantization genuinely improved accuracy.
   • B. Something else changed between the two measurements — quantization does not raise a model's accuracy ceiling.
   • C. INT8 always outperforms FP32 for multimodal models.
   • D. The eval set was quantized along with the model.
2. Which method requires labeled training data and gradient updates?
   • A. PTQ.
   • B. QAT.
   • C. Both equally.
   • D. Neither — quantization never needs training data.
3. A vision encoder and a fusion layer are quantized to INT8 with the same PTQ pass. The fusion layer loses far more accuracy. What does this most directly suggest?
   • A. The fusion layer should be reverted to FP32 permanently, with no other option considered.
   • B. The fusion layer is a strong candidate for QAT while the vision encoder may not need it.
   • C. The vision encoder's parameter count is too large to quantize safely.
   • D. PTQ is broken and should not be used anywhere in the model.
4. What is the main practical cost of choosing QAT over PTQ for a component?
   • A. QAT requires more inference-time memory than PTQ.
   • B. QAT requires a training run — data, gradients, and GPU time — that PTQ does not.
   • C. QAT cannot be applied to multimodal models.
   • D. QAT always produces a larger model than PTQ.
5. Why might treating a multimodal model's post-quantization accuracy as a single whole-model average be misleading?
   • A. Averages are never mathematically valid for accuracy scores.
   • B. A small, sensitive component's damage can be hidden by a larger, more tolerant component's near-unchanged accuracy.
   • C. Whole-model averages always overstate accuracy loss.
   • D. Multimodal models cannot be evaluated with a single metric at all.
6. Which statement correctly separates quantization from mixed-precision training?
   • A. They are the same technique, applied at different times, with identical accuracy guarantees.
   • B. Mixed-precision training protects training-time accuracy; quantization accepts inference-time accuracy loss that QAT can only partially recover.
   • C. Quantization always happens before training; mixed-precision training always happens after.
   • D. Mixed-precision training and quantization both require a calibration dataset.

Answers

1. B. Quantization's ceiling is the full-precision model's own accuracy; a reported improvement points at a change in the comparison (different eval run, different data, measurement error), not a real quantization-driven gain — this is the exam's most direct "quantization improves accuracy" trap, restated as a scenario.
2. B. QAT is training: it needs labeled data and gradient updates. PTQ needs at most a small unlabeled calibration set.
3. B. This is the diagnostic pattern from section 4's worked example: a component that loses disproportionate accuracy under uniform PTQ is the candidate for QAT's extra cost, while a component that tolerates PTQ well can stay there.
4. B. QAT's cost is a genuine training run; PTQ's memory and inference-time behavior are not the axis the two methods differ on.
5. B. This is exactly the failure mode section 4's per-component measurement is designed to catch — the small fusion head's large drop was invisible in a single aggregate number until measured on its own.
6. B. Mixed-precision training (M5-01) is a training-time technique preserving accuracy; quantization is an inference-time technique that trades some accuracy away, with QAT narrowing but not eliminating the gap.

• Quantization shrinks memory and speeds inference; it never improves accuracy, and any option claiming otherwise is the exam's most direct trap.
• PTQ happens after training with no gradient updates; QAT happens during training and usually recovers more accuracy — that one distinction answers most exam questions on this topic.
• Mixed-precision training (M5-01) and quantization are easy to conflate but operate at different lifecycle stages with different accuracy guarantees; only quantization accepts a real accuracy cost.
• A multimodal model's separate encoders can tolerate the same precision cut very differently — the fusion layer is frequently the most sensitive component despite often being the smallest.
• Measuring accuracy per component, not just for the whole model, is what reveals whether a uniform PTQ pass is hiding damage in one specific part.
• Mixed PTQ-and-QAT decisions, applied per component, can recover most of QAT's accuracy benefit while spending the training-run cost only where it is actually needed.

This module now turns from precision to structure. Next: M5-03 covers neural network pruning — removing redundant weights, neurons, or filters from a trained network, and the structured-versus-unstructured distinction that determines whether the resulting sparsity is easy or hard for a GPU to actually accelerate.