Streaming Attention and TensorRT Runtime Optimization

Identity statement: sliding-window (streaming) attention restricts the span of history a token's attention computation actually looks at — and therefore the span that ever needs to be cached — to a bounded window, rather than the full, ever-growing prefix a standard causal attention mechanism would otherwise attend to.

When it matters: whenever a scenario describes a very long sequence — a long-running conversation, a document far longer than the context the model was trained to attend over efficiently, or a streaming application that never really "ends" — and the stated constraint is memory that keeps growing without bound, rather than the per-step recomputation latency M4-04's KV cache already resolves.

M4-04 drew a precise line between two different costs in autoregressive decoding: the cost of recomputing keys and values for tokens already processed (which KV caching eliminates entirely, because keys and values never change once computed) and the cost of attending over however much history is cached (which KV caching does not touch — a model with a 10,000-token cache still runs attention over all 10,000 positions on every single decoding step, cache or no cache). As a conversation or a document grows longer, that second cost keeps growing too, and so does the memory the cache itself occupies, because more cached tokens means more stored keys and values, without any ceiling in a standard causal-attention setup. [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "Sliding-window / streaming attention. Bound the attention span to control memory for very long sequences." Streaming attention is the lever that puts an actual ceiling on that growth, by deciding in advance that a token only ever needs to attend to, and therefore only ever needs to keep cached, some fixed-size window of recent history rather than the complete, unbounded prefix.

Why this is a different problem than the one caching solved

It is worth being precise about why these are genuinely two separate levers rather than one idea described twice, because conflating them is the standing trap this pairing produces. KV caching's entire benefit is eliminating redundant work: recomputing a key or value that was already computed and has not changed is pure waste, and caching removes exactly that waste with no approximation and no accuracy cost, because the cached values are exactly what recomputation would have produced anyway. Streaming attention's benefit is different in kind: it does not eliminate redundant work, it eliminates some of the actual computation and storage that a full, unbounded-history model would otherwise need to do, by deliberately choosing not to attend to everything. That is not a free optimization the way caching is — bounding the attention window is a real, substantive change to what the model can see, and it can cost something a pure recomputation-elimination trick like caching never costs: the model may now be unable to refer back to something outside its window that would have mattered.

L1 — Intuition: a moving spotlight instead of a floodlit stage

Standard causal attention, once a KV cache is in place, is a floodlit stage: every token generated so far stays fully lit, fully attended-to, fully cached, no matter how long the performance runs. Sliding-window attention replaces the floodlight with a moving spotlight of fixed size — as new tokens enter the lit area at the front, the oldest tokens fall out of it at the back, and only what is currently inside the spotlight's fixed span is attended to or kept in the cache at all. The stage can be arbitrarily long; the lit portion of it, at any given moment, never grows past the spotlight's fixed width.

L2 — Mechanism: a fixed window, evicted history, and a bounded cache as the direct consequence

Mechanically, sliding-window attention constrains each token's attention computation to a fixed-size window of the most recent tokens — commonly some number of positions immediately preceding the current one — rather than the full causal prefix back to the sequence's start. Once a token falls outside that window relative to the current decoding position, it is no longer attended to, and because it is no longer attended to, its cached key and value tensors serve no further purpose and can be evicted from the cache. This is the direct mechanism by which streaming attention bounds memory: the KV cache's size, which M4-04 showed scales with sequence length among other factors, stops scaling with total sequence length once a fixed window is in force, because sequence length beyond the window no longer adds anything new to what is stored — old entries are evicted at roughly the same rate new ones are added, keeping the cache's size at a stable ceiling regardless of how long the overall sequence eventually runs.

This is a meaningfully different growth curve than the one M4-04's worked example computed. Without a bounded window, cache size is proportional to the full sequence length processed so far, growing without limit as generation continues. With a fixed window of size w, cache size is proportional to w alone, regardless of how many total tokens have been generated — a 100,000-token conversation and a 100-token conversation occupy the same cache footprint once the window has filled, because only the most recent w tokens are ever kept regardless of how many came before them.

L3 — The exam-relevant edge case: what a bounded window costs that an unbounded one does not

The tradeoff streaming attention accepts is real and worth stating precisely, because a scenario item can test whether you notice the cost alongside the benefit. A token generated far outside the current window can no longer be attended to at all — not approximately, not at reduced weight, but not at all, because it is neither cached nor considered. For a task where everything relevant to the current output sits within a bounded recent window — an ongoing dialogue focused on the last few exchanges, a streaming transcription task, a rolling summarization of recent events — this cost is negligible, because the information genuinely needed was never far outside the window to begin with. For a task that genuinely needs to refer back to something far earlier — a long document-analysis task where the answer depends on a detail stated at the very beginning of a very long document — a fixed sliding window can lose access to exactly the information the task needs, and no amount of caching efficiency compensates for information that was never kept.

Identity statement: TensorRT is NVIDIA's deployment-time inference compiler — it takes a trained model's computation graph and compiles it into an optimized, hardware-specific execution plan, applying kernel fusion, precision calibration, and auto-tuning for one target GPU.

When it matters: whenever a scenario describes preparing an already-trained model to run as fast as possible on a specific, known GPU target, as a build-time step that happens once before serving begins — as distinct from the runtime serving infrastructure that handles live requests once the optimized model is ready.

[GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "TensorRT. Compiles and fuses kernels, applies precision calibration and layer/tensor fusion, and auto-tunes kernels for a target GPU — the deployment optimizer." Unpack that into the three categories worth being able to name individually.

Kernel and layer fusion. A model graph, taken naively, executes as a sequence of separate operations — a matrix multiply, then a bias addition, then an activation function — each one reading its input from GPU memory and writing its output back before the next operation starts. Fusion merges adjacent operations like these into a single compiled kernel, so the intermediate result never has to round-trip through memory between steps. Because moving data through memory is frequently the actual bottleneck in modern GPU execution, not the arithmetic itself, eliminating that round-trip is often the single largest speedup fusion buys.

Precision calibration. TensorRT applies the same precision-reduction ideas M4-01 covers in depth for quantization broadly — compiling a model at FP16, INT8, or another reduced precision, using a calibration pass to derive appropriate scale factors where needed — as part of the compilation step itself, rather than as a separate standalone process. TensorRT is one of the concrete places where quantization's PTQ mechanics actually get executed against a real deployment target.

Kernel auto-tuning. For a given operation in the graph, there can be multiple different low-level implementations, each with different performance characteristics depending on the exact GPU architecture, tensor shapes, and available hardware resources. TensorRT benchmarks the candidate implementations against the actual target GPU during the build process and selects whichever one measures fastest for that specific hardware — which is exactly why a compiled TensorRT engine is tied to the GPU architecture it was tuned against, and why compiling for one target GPU does not guarantee anything about performance, or even correctness of the tuning choice, on a different one.

All three of these are decisions made once, at build time, before any inference request has been served — which is the structural reason TensorRT sits on the "optimizer" side of this domain's central distinction rather than the "server" side. It produces a compiled artifact; it does not itself accept live traffic, manage concurrent requests, or route work across multiple model instances.

This is the exam's most direct trap in this pairing, and it is worth stating in exactly the words the source material uses. [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "TensorRT optimizes; Triton serves. TensorRT is not an inference server." TensorRT's job ends the moment a compiled, optimized engine exists — it has fused kernels, calibrated precision, and tuned execution for a target GPU, and the artifact it produces is ready to run inference, but nothing about TensorRT itself accepts a network request, load-balances across multiple copies of a model, or manages a queue of concurrent users. Triton Inference Server, covered in full in Domain 8, is the serving layer: it is the piece of infrastructure that actually receives requests, routes them to a model (which may itself be a TensorRT-compiled engine, among other backend formats Triton supports), batches requests together, and returns results — the operational, always-running counterpart to TensorRT's one-time, build-step compilation.

The clean way to hold the two apart under time pressure: TensorRT answers "how fast can this specific model run on this specific GPU," resolved once at build time; Triton answers "how do I take live traffic and get it to a running model efficiently," resolved continuously at serving time. A scenario describing compiling, fusing kernels, or auto-tuning for a target GPU is describing TensorRT's job. A scenario describing accepting requests, batching them, or routing traffic across model instances is describing Triton's job — and a question that swaps one term for the other in an otherwise-correct description is testing exactly this line.

The row that resolves most scenario items fastest is "question it answers." A described memory problem that grows with total sequence length, unboundedly, over a very long conversation, points at streaming attention. A described problem about redundant computation at every decoding step points at KV caching. A described problem about compiling, tuning, or fusing kernels for a specific GPU target points at TensorRT. And a described problem about accepting live requests or serving multiple models concurrently points outside this domain entirely, to Triton in Domain 8.

Take the same illustrative 13-billion-parameter-class decoder M4-04 used — 40 layers, hidden size 5,120, 40 heads, BF16, roughly 0.8 MB of cache per token across the whole model — and compare unbounded caching against a sliding window at a genuinely long conversation length.

Per-token cache cost (from M4-04's derivation): ~0.8 MB per token, across all 40 layers

Unbounded cache at a 100,000-token conversation:
  100,000 x 0.8 MB = 80,000 MB ≈ 78 GB    for this one conversation alone

Sliding window, fixed at w = 4,096 tokens, at the same 100,000-token conversation:
  Cache never exceeds: 4,096 x 0.8 MB = 3,276.8 MB ≈ 3.2 GB
  regardless of whether the conversation is 5,000, 100,000, or 1,000,000 tokens long

Read the second block as the entire point of this lesson's first half: the unbounded cache's cost scales linearly with however long the conversation happens to run, with no ceiling in sight, while the sliding-window cache's cost is capped at roughly 3.2 GB permanently, the moment the window fills, no matter how much longer the conversation continues afterward. This is a constructed scenario built on M4-04's own model parameters and stated window size, illustrating the mechanism rather than measuring a specific deployed system, but the ratio it shows — 78 GB unbounded versus 3.2 GB bounded, a roughly 24x difference at this specific conversation length, growing without limit as the conversation gets even longer — is exactly why streaming attention is named as the mitigation for very long sequences specifically, rather than as a general-purpose replacement for ordinary KV caching at moderate context lengths where the unbounded cache was never going to be a problem in the first place.

Take a trained decoder-only model, already using KV caching and, optionally, a bounded sliding window, and walk it through a TensorRT build to see which properties change and which stay fixed.

Before compilation:
  Model graph: a sequence of separate matmul, bias-add, and activation
  operations, executed one at a time, each round-tripping through
  GPU memory between steps
  Precision: FP16, uncalibrated
  Target: unspecified — the graph runs correctly on any compatible GPU,
  at whatever speed that GPU's default execution happens to deliver

After a TensorRT build targeting one specific GPU architecture:
  Fusion: adjacent matmul + bias-add + activation operations merged
  into single compiled kernels — fewer memory round-trips per layer
  Precision: INT8, with scale factors derived from a calibration pass
  (the same PTQ mechanism M4-01 covers in full)
  Kernel selection: for each operation, the fastest of several candidate
  implementations, chosen by benchmarking against this specific GPU
  during the build
  Target: fixed to the exact GPU architecture the build ran against —
  the resulting engine is not guaranteed to run optimally, or at all,
  on a different architecture

Two things generalize past this specific illustration. First, everything that changed is a build-time property: nothing about serving, request handling, or concurrent traffic appears anywhere in this list, because none of that is TensorRT's job — it is entirely orthogonal to what the compiler does. Second, the precision change is not free: compiling to INT8 is the same kind of accuracy-affecting decision M4-01 insists must be measured on an evaluation set, not assumed, whether it happens through a standalone PTQ pipeline or as one step inside a TensorRT build. TensorRT bundling precision calibration into its compilation step does not exempt that step from the discipline objective 4.2 demands for every optimization in this domain: measure the tradeoff, do not assume it away because it happened to be convenient to apply during a build.

Model Optimization is Domain 4 of the NCP-GENL blueprint at 17% — the single largest domain — and [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) groups streaming attention and TensorRT together as two of the three named runtime optimizations, alongside KV caching, with an explicitly stated common trap distinguishing TensorRT's optimizing role from Triton's serving role. That "optimizes versus serves" line recurs across this cert's domains — it is the same distinction Domain 8's deployment-stack lessons build on — which makes getting it right here worth more than its single appearance in this module suggests.

Expect the question to arrive in these shapes:

• A direct role-identification item. "Which of the following compiles and fuses kernels for a target GPU but does not itself serve inference requests?" with TensorRT as the keyed answer against a distractor naming Triton or a generic "inference server."
• A memory-mechanism scenario. A description names a very long, growing conversation and a memory problem that keeps worsening despite KV caching already being in place — the keyed answer is sliding-window or streaming attention, distinguishing it from caching itself.
• A build-versus-runtime distinction item. A scenario names auto-tuning kernels for a specific GPU architecture, asking whether this happens once at build time or continuously at serving time — the keyed answer is build time, and the term is TensorRT, not Triton.
• A precision-discipline item. A scenario compiles a model to INT8 via TensorRT and asks what must happen before release — the keyed answer is re-running the evaluation set, exactly as any other precision-reducing step in this domain requires.

What the distractors typically look like

Expect TensorRT described as accepting requests, load-balancing, or managing a model repository — all Triton's job, offered as if they belonged to the optimizer instead. Expect a sliding window's fixed size described as having no accuracy cost, when information genuinely falls outside it and becomes inaccessible. Expect a TensorRT engine described as portable across GPU architectures without a rebuild, when kernel auto-tuning specifically ties the result to the hardware it was benchmarked against. And expect streaming attention offered as a substitute for KV caching rather than a complementary lever addressing a different axis of the same overall memory-and-latency picture.

No — they are complementary, not competing, and this exact relationship is the source of the standing trap this pairing produces. KV caching eliminates the redundant recomputation of history's keys and values at every decoding step, an exact optimization with no accuracy cost. Streaming attention bounds how much history is ever attended to or cached in the first place, a real architectural choice with a real, if often acceptable, cost to what the model can see. A production system with a very long-context requirement typically uses both together: KV caching so that whatever is inside the current window is never redundantly recomputed, and a sliding window so that the cache itself never grows past a fixed ceiling regardless of how long the overall interaction runs.

No. Kernel auto-tuning, one of TensorRT's three named optimization categories, works by benchmarking candidate kernel implementations against the actual target GPU during the build and selecting whichever measures fastest for that specific hardware. A compiled engine therefore encodes decisions that are valid for the architecture it was built against, and there is no guarantee those same decisions are optimal — or that the engine even loads and runs correctly at all — on a different GPU architecture. The operational consequence, though outside this module's own scope to fully develop, is that a fleet spanning multiple GPU architectures needs a separate TensorRT build per target rather than one universal compiled artifact.

Work through each item before checking the answer key. Every option describes something real about some deployed system — the task is matching the described mechanism to the correct term, not spotting an obviously fabricated distractor.

1. A team's KV cache is already in place, but memory keeps growing without limit as a conversation gets longer and longer. What addresses this specifically?
   • A. Increasing beam width.
   • B. Sliding-window (streaming) attention, bounding the span of history attended to and cached.
   • C. Switching from FP16 to FP32.
   • D. Adding more attention heads.
2. Which of the following is TensorRT's job?
   • A. Accepting live inference requests and routing them across model instances.
   • B. Compiling, fusing kernels, calibrating precision, and auto-tuning a model for a specific target GPU.
   • C. Hosting a model repository with versioning.
   • D. Managing a queue of concurrent users.
3. "TensorRT optimizes; Triton serves." What does this distinction mean?
   • A. TensorRT and Triton are two names for the same product.
   • B. TensorRT produces a compiled, hardware-specific artifact at build time; Triton accepts and routes live traffic at serving time.
   • C. Triton compiles kernels; TensorRT accepts requests.
   • D. TensorRT is a subset of Triton's features.
4. Why is a TensorRT-compiled engine generally not portable across different GPU architectures?
   • A. Fusion only works on one architecture.
   • B. Kernel auto-tuning benchmarks and selects implementations specific to the exact target GPU during the build.
   • C. Precision calibration is architecture-agnostic, so this is not actually true.
   • D. Engines are portable; only the model weights are architecture-specific.
5. What real cost does a fixed sliding-window attention span accept, that ordinary KV caching does not?
   • A. It increases recomputation of history's keys and values.
   • B. Information genuinely outside the current window becomes inaccessible to the model, not merely deprioritized.
   • C. It requires retraining the model from scratch.
   • D. It doubles the memory used per cached token.
6. A model is compiled with TensorRT at INT8 precision. What must happen before this build ships to production?
   • A. Nothing — precision calibration inside a compiler is exempt from accuracy checks.
   • B. The evaluation set must be re-run, because precision reduction is a measurable accuracy-affecting change.
   • C. The model must be retrained from scratch.
   • D. The KV cache must be disabled.
7. Which pair of levers are complementary rather than competing, because one eliminates redundant computation and the other bounds how much is ever attended to or cached?
   • A. TensorRT and Triton.
   • B. KV caching and sliding-window attention.
   • C. PTQ and QAT.
   • D. Distillation and pruning.
8. What does kernel and layer fusion change about a model's computed output?
   • A. It changes the model's mathematical function, producing different results.
   • B. It produces the same computed result faster, by removing intermediate memory round-trips between operations.
   • C. It reduces the model's parameter count.
   • D. It removes the need for a KV cache.

Answers

1. B. This is the exact mechanism section 1 and section 2 build: a bounded window is what caps cache growth as sequence length increases without limit, which caching alone never does.
2. B. This is TensorRT's identity statement from section 3 and the [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) citation: fusion, calibration, and auto-tuning for a target GPU, all at build time.
3. B. This is the standing distinction section 4 states directly from the source material — optimizer versus server, build time versus serving time.
4. B. Section 3's mechanism: auto-tuning selects the fastest of several candidate kernel implementations by benchmarking against the actual target GPU, tying the result to that specific hardware.
5. B. This is section 2's L3 discussion: a fixed window is a real tradeoff, not a free optimization, because history outside it is neither cached nor attended to at all.
6. B. This is the discipline section 7's worked example and objective 4.2 both require: any precision-reducing build, TensorRT-driven or not, must have its accuracy measured on an eval set before release.
7. B. Section 5's comparison table names this directly: caching eliminates redundant recomputation with no accuracy cost, while streaming attention bounds what is ever cached or attended to, at a real potential accuracy cost — complementary, not substitutable.
8. B. Fusion is an execution-mechanics change, not a model-behavior change — the computed result is identical, just produced with fewer memory round-trips, as section 3 and the common-mistakes table both state.

• Sliding-window attention bounds memory for very long sequences by fixing the span of history a token attends to and caches — a real tradeoff, not a free optimization, distinct from M4-04's KV caching, which eliminates redundant recomputation with no accuracy cost at all.
• TensorRT compiles, fuses kernels, and auto-tunes for a target GPU, applying M4-01's precision-reduction mechanics as one of its three named optimization categories.
• TensorRT optimizes; Triton serves. TensorRT is not an inference server — it produces a compiled artifact at build time, and Triton (Domain 8) is the always-running layer that accepts live traffic.
• A TensorRT engine is tied to the GPU architecture it was auto-tuned against and generally must be rebuilt per target architecture.
• Streaming attention and KV caching are complementary, not competing — one eliminates recomputation, the other bounds how much ever needs to be cached or attended to at all.
• Any precision-reducing TensorRT build is a quantization decision and inherits the same measured-accuracy-tradeoff discipline objective 4.2 demands everywhere else in this domain.
• Model Optimization is 17% of the NCP-GENL blueprint, the largest domain, and the "TensorRT optimizes, Triton serves" line is one of its most frequently tested single distinctions.

Streaming attention and TensorRT both operate on decoder-style, autoregressive generation — the setting where a growing cache and a per-token compile target both make sense as concepts. This module's final lesson turns to a different kind of foundation model entirely, one that reads a whole input at once rather than generating token by token, and closes the loop on every shrinking lever this module has covered by applying them to a concrete deployment scenario.

Next: M4-06 covers encoder foundation models and masked language modeling — how BERT-style pretraining differs from the decoder objective this module's runtime levers assumed, and how quantization, distillation, and pruning, all covered earlier in this module, apply to shrinking an encoder-based model for deployment.