Model-Type Compute Tradeoffs: Encoder vs. Decoder vs. Encoder-Decoder at Serving Time

A model's forward pass is the arithmetic it must perform to turn an input into an output. Infrastructure — batching, instance count, hardware partitioning — determines how efficiently many requests' worth of that arithmetic gets executed in parallel, but it does not change how many forward passes a single request requires, or how those passes depend on each other. That dependency structure is set by the model's architecture family, and it is the thing this lesson's three families differ on most sharply.

An encoder-only model reads its entire input and produces its entire output in one forward pass — there is no notion of the output depending on a previous piece of the same output, because the output is not generated piece by piece. A decoder-only model in generation mode produces its output autoregressively: token by token, where each new token's computation depends on every token generated before it, which means the model cannot know what its second output token will be until it has actually produced the first one. That single structural fact — one pass versus many sequentially dependent passes — is the whole source of this lesson's latency-profile differences, and no amount of clever batching or instance scaling changes it, because batching and instance scaling operate across requests, while this dependency operates within a single request's own generation.

Identity statement: an encoder-only model (BERT-family architectures) processes its entire input with bidirectional attention in a single forward pass and returns its entire output at once — there is no autoregressive, step-by-step generation involved. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) frames the encoder-only row of the domain's compute-tradeoff table around exactly this: "Bidirectional; single forward pass; lower latency," with typical uses in "embeddings, classification."

Because an encoder-only model's latency for one request is essentially the cost of one forward pass, batching multiple requests together (dynamic batching, per M8-01) is a highly effective lever here: combining, say, 32 independent classification requests into one batched forward pass costs roughly the same wall-clock time as processing one request alone, because the GPU parallelizes across the batch dimension efficiently for a single-pass computation. Doubling the batch size does not double the latency of that one batched pass — it makes much more efficient use of the same forward-pass cost, which is exactly why "bigger batches help" is a fair generalization for this family specifically. This is also why encoder-only models are the natural fit for embeddings and classification workloads: both tasks want one complete answer per input, with no notion of a partial answer that needs further steps to complete.

Identity statement: a decoder-only model (GPT-family architectures) generates output autoregressively, one token at a time, where producing token t requires the model to have already produced (and fed back in) tokens 1 through t-1 — this sequential dependency, not the batch size of concurrent requests, is what governs how long generating a response of a given length takes. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states this directly: decoder-only latency is "dominated by sequential decoding + KV cache growth," and the domain's own named trap states the consequence plainly: "autoregressive decoders can't be sped up simply by 'bigger batches' the way a single-pass encoder can — sequential decoding dominates their latency."

L1 — Intuition

Picture asking a decoder to generate a 200-token response. Batching that request together with 31 other, unrelated generation requests can make each of the 200 individual decoding steps more GPU-efficient — computing 32 requests' next-token predictions in one batched step is cheaper per request than computing them one request at a time. But batching does nothing to reduce the number of steps — the response still requires 200 sequential steps, one for each output token, and step 150 still cannot begin until step 149 has produced its token, no matter how many other requests are riding along in the same batch. Latency for that one response, end to end, is still bounded by 200 sequential steps' worth of time, batched or not.

L2 — Mechanism

This is the precise sense in which "bigger batches alone cannot fix" decoder latency the way they help an encoder: batching helps throughput — how many total responses across many concurrent requests the system can produce per unit time — but it does comparatively little for the latency of any single response, because that single response's own length in tokens sets a sequential-step floor no batching arrangement removes. An encoder-only request's latency and a batched-encoder request's latency are close to each other for a well-chosen batch size; a decoder-only request's latency for a long response is set mostly by its own output length, largely independent of how many other requests share its batch.

L3 — Where KV caching fits, cited rather than re-derived

The direct mitigation for the sequential-decoding cost itself — as opposed to a mitigation for cross-request throughput, which batching already handles — is KV caching, covered in full mechanism, scaling math, and worked arithmetic in M4-04. Rather than re-deriving that mechanism here, the fact worth carrying into this deployment-facing lesson is the connective one: KV caching does not eliminate the sequential dependency — token t still cannot be produced before token t-1 — but it eliminates the redundant recomputation that would otherwise make each sequential step even more expensive than it has to be, by storing each token's key and value tensors once rather than recomputing the entire prefix's attention inputs at every step. M4-04 frames this as memory traded for speed, and that framing holds unchanged here: KV caching is the reason decoder serving latency, while still scaling with output length, scales far better than it would without the cache. Nothing about batching or instance-group sizing substitutes for that mitigation, because both operate on a different axis of the problem — across requests, not within one request's sequential chain.

Identity statement: an encoder-decoder model (T5-family architectures) encodes its input once, in a single forward pass much like an encoder-only model, and then generates its output autoregressively, one token at a time, much like a decoder-only model — meaning its latency profile inherits the decoder's sequential-generation cost on top of a one-time encoding cost, rather than avoiding either. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) frames this directly: "encode once, then decode sequentially," with typical uses in "seq2seq (translation, summarization)."

The encoding step here behaves latency-wise like section 2's encoder-only case: one forward pass over the input, batch-friendly, largely insensitive to output length because it has not started producing output yet. Once encoding finishes, the decoding step behaves latency-wise like section 3's decoder-only case: sequential, one token at a time, resistant to being sped up by bigger batches, and mitigated by the same KV-caching mechanism M4-04 covers. For a short output (a short summary, a short translation), the one-time encoding cost can be a meaningful fraction of total latency; for a long output, the sequential decoding cost dominates total latency in essentially the same way it dominates a pure decoder-only model's latency, because the encoding step's one-time cost does not scale with output length while the decoding step's cost does.

The practical consequence worth holding onto is that an encoder-decoder model does not get to claim the encoder-only family's most attractive property (latency largely insensitive to output length) just because it has an encoder component — the decoder half of its architecture reintroduces exactly the sequential dependency section 3 describes, and a long-output seq2seq task (a lengthy document summarization, say, rather than a short headline) will show decoder-dominated latency behavior almost indistinguishable from a comparably-sized decoder-only model generating the same output length.

Take a constructed scenario in this domain's preferred style: a platform serves three models behind the same Dynamo-Triton deployment, all with dynamic batching enabled and reasonably sized instance groups. A stakeholder reports that one of the three models — a document-summarization model producing long, multi-paragraph summaries — has noticeably higher per-request latency than the other two, an embeddings model and a short-form classification model, even though all three receive comparable traffic volume and share the same infrastructure configuration.

Model A (embeddings, encoder-only): fast, largely batch-insensitive latency
Model B (classification, encoder-only): fast, largely batch-insensitive latency
Model C (summarization, encoder-decoder): noticeably higher per-request latency

Candidate explanation 1: "Model C's instance group is undersized relative
  to demand."
  -> Would predict QUEUEING and rising latency under load specifically,
     not a flat, consistently higher per-request latency regardless of
     load level.

Candidate explanation 2: "Model C's architecture requires sequential,
  autoregressive decoding to produce its long, multi-paragraph output,
  which neither batching nor instance-group sizing directly shortens."
  -> Fits the evidence: Model C's higher latency is a property of HOW
     LONG ITS OUTPUT IS and how that output is generated, not a property
     of infrastructure sizing shared identically across all three models.

The diagnostic habit worth keeping here mirrors the one M8-03 introduced for batching-versus-instance-count symptoms: before reaching for an infrastructure fix, check whether the symptom correlates with infrastructure load (queueing, GPU utilization) or with something intrinsic to the request itself, such as output length. A consistently higher latency that tracks output length rather than concurrent load is an architecture-family signature, not an infrastructure-sizing signature, and the correct response is to verify KV caching is actually enabled and correctly sized for Model C's decoding phase, per M4-04's treatment — not to add more instances or tune the batching window, both of which would leave the sequential decoding cost untouched.

A second constructed scenario isolates the batching claim precisely. A team runs two experiments on the same decoder-only model: Experiment 1 measures total throughput (responses completed per minute) at batch size 1 versus batch size 16 for many short, unrelated generation requests arriving concurrently. Experiment 2 measures the latency of one single 500-token generation request, run alone, at batch size 1 versus artificially forced into a batch of 16 (padded with 15 placeholder requests of the same length).

Experiment 1 (throughput, many concurrent short requests):
  batch size 1  -> low throughput, GPU underutilized between steps
  batch size 16 -> much higher throughput, GPU utilized far more efficiently
  CONCLUSION: batching materially helps here (a throughput measure, per M8-03)

Experiment 2 (latency, one 500-token request):
  batch size 1  -> latency roughly proportional to 500 sequential decode steps
  batch size 16 -> latency for THIS ONE REQUEST still roughly proportional
                   to 500 sequential decode steps -- the other 15 slots in
                   the batch do not shorten this request's own step count
  CONCLUSION: batching does little for this single request's own latency,
              exactly as the domain's named trap states

Both experiments are internally consistent with the same underlying fact: batching raises throughput by making each sequential step cheaper per request when many requests share it, but it does not reduce the number of sequential steps any single long-output request must still pass through. Treating Experiment 1's throughput win as proof that "batching fixes decoder latency" generally is exactly the overgeneralization this domain's named trap warns against.

Model Deployment is objectives 8.1 through 8.3, carrying 9% of the NCP-GENL blueprint, and objective 8.1 specifically asks the exam-taker to analyze tradeoffs across architecture families and optimize for memory and latency — which is precisely what this lesson has walked through for the three families the domain names. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) closes its own coverage of this topic with the exact trap this lesson has built around: "autoregressive decoders can't be sped up simply by 'bigger batches' the way a single-pass encoder can — sequential decoding dominates their latency."

How the question tends to be phrased

Expect a direct identification question ("why does decoder-only generation latency scale with output length?" with "it decodes autoregressively, one token at a time" as the keyed answer against distractors naming re-tokenization, bidirectional attention, or disabled KV caching — real-sounding but incorrect mechanisms). [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) contains this exact self-check item. Expect also a scenario question, in the style of section 6, describing a latency complaint on a specific model type and asking whether the fix is an infrastructure lever (batching, instances) from earlier in this module or an architecture-aware mitigation (KV caching) from Model Optimization.

What the distractors typically look like

The house style offers "increase the batch size" as a fix for a decoder's own single-request latency, when batching is a real, correct technique misapplied to the wrong axis of the problem (throughput, not per-response latency for a long, sequential generation). A second recurring distractor offers "add more instances" for the same symptom, which is equally a real, correct technique for a different problem (too little parallel capacity across requests) misapplied to a sequential, within-request dependency that more instances do not shorten.

An encoder-only model completes its entire computation in one forward pass, so batching many requests together lets the GPU compute all of them nearly as cheaply as computing one — batch size directly trades against per-request latency in a favorable way. A decoder-only model instead generates its response one token at a time, where token t cannot be computed until token t-1 has been produced; batching many concurrent requests together makes each of those sequential steps cheaper per request, raising throughput, but it does not reduce the number of sequential steps any single request's own response length requires. That sequential step count, not the batch size, is what sets a single response's latency floor.

KV caching removes the redundant recomputation that would otherwise make each sequential decoding step progressively more expensive, by storing each token's key and value tensors once and reusing them rather than recomputing the whole prefix's attention inputs at every step — the full mechanism and scaling math are covered in M4-04. It does not, and cannot, remove the sequential dependency itself: producing token t still requires token t-1 to already exist. What it buys a deployment is a serving latency that, while still scaling with output length, scales far more cheaply per token than an uncached decoder would, which is why every production decoder-serving stack treats KV caching as a baseline assumption rather than an optional extra.

• Encoder-only models complete in one forward pass, so latency is largely insensitive to output length and highly responsive to good batching.
• Decoder-only models generate sequentially, one token at a time, so latency scales with output length regardless of batch size or instance count — the module's closing named trap.
• Encoder-decoder models inherit the decoder's sequential-latency profile for long outputs, on top of a one-time, batch-friendly encoding cost.
• KV caching (M4-04) is the direct mitigation for sequential decoding's cost, not batching and not more instances — those two address cross-request throughput and parallel capacity, a different axis entirely.
• Diagnosing which layer a latency symptom belongs to — infrastructure (M8-01–M8-04) or architecture (this lesson) — is the practical skill objective 8.1 tests, and the fastest signal is whether the symptom tracks load or tracks output length.
• Model Deployment is 9% of the NCP-GENL blueprint, and this lesson closes the module's guiding question: once a model is trained and optimized, the serving technology chosen still has to be matched to what the model's own architecture actually requires.

This closes Module 8. Everything from M8-01's batching-mode choice through this lesson's architecture-level tradeoffs describes how a model, once trained and optimized, gets served correctly and efficiently. None of it yet addresses what happens after that model has been serving live traffic for months — whether its measured latency, throughput, and error rate stay within acceptable bounds, and whether the world the model was trained on has quietly shifted out from under it. That operational, post-launch question is the next module's subject.

Next: Module 9, Production Monitoring and Reliability, picks up where Model Deployment leaves off — moving from "is this model served correctly today" to "how do we know, and prove, that it is still working six months from now."