Inference Batching: Static vs Dynamic vs Continuous (In-Flight) Batching

A forward pass through a transformer at decode time does two things: it moves bytes (weights, and the KV cache) from HBM into the compute units, and it performs arithmetic. For a single sequence, the arithmetic is a set of matrix-vector products — one token's hidden state against each weight matrix — offering roughly two floating-point operations per weight byte read. The hardware wants hundreds. So the step's duration is set almost entirely by how long it takes to stream the weights, and the tensor cores idle.

Now put B sequences into the same step. The weights are read exactly once. The matrix-vector products become matrix-matrix products of width B. Arithmetic rises by a factor of B; bytes read for weights do not rise at all. The step takes a little longer — you did more math and you read B KV caches instead of one — but nothing like B times longer. Total tokens produced per unit time rises steeply.

Two ways to say the same thing, both worth having:

• Cost framing. The fixed cost of a decoding step is the weight read. Batching divides that fixed cost across more useful work. It is the classic amortization argument.
• Utilization framing. Batching converts idle tensor cores into useful arithmetic. The GPU was going to sit waiting on HBM anyway; batching fills that wait with work.

The limits are equally important, because a claim of free throughput is always false somewhere:

1. KV cache memory. Each sequence in the batch carries its own cache, and cache size scales with batch × sequence length. 12-05 computed this: at 512 KB per token for a 32-layer, 4096-hidden model in BF16, thirty-two concurrent 2,000-token conversations need about 33.6 GB. Memory capacity, not compute, sets your maximum batch size in almost every real deployment.
2. Cache bandwidth. At large batch and long sequences, reading B caches every step becomes comparable to reading the weights. Once cache traffic dominates, adding sequences stops being nearly free.
3. Compute saturation. Eventually arithmetic intensity rises far enough that the tensor cores become the limit. At that point you have converted a bandwidth-bound workload into a compute-bound one, which is the ideal outcome and also the end of easy gains.
4. Latency. Every batching scheme except pure continuous batching makes some request wait for others. That waiting is the cost, and it lands in the tail — the subject of 12-10.

L1 — Intuition: the bus, the shuttle, and the escalator

Static batching is a bus with a fixed departure size. It leaves when exactly eight passengers are aboard, no sooner and no later. If only three people show up, everyone waits indefinitely; if twelve arrive, four wait for the next bus. Simple, predictable, and badly matched to bursty traffic.

Dynamic batching is a shuttle on a timer. It waits up to 10 milliseconds for passengers, then departs with whoever is aboard — one, four, or the maximum. It adapts to load: busy periods produce full batches, quiet periods produce small ones with a bounded wait. This is the right model for classical inference, where every request takes about the same time.

Continuous batching is an escalator. There is no departure at all. Riders step on whenever they arrive and step off whenever they reach their floor. The mechanism runs continuously and its occupancy changes step by step. This is the right model for generative inference, where one request may produce 12 tokens and another 1,200.

That last contrast is the entire reason continuous batching exists, and it is worth stating as its own paragraph.

The problem continuous batching solves. In a batch of eight sequences under dynamic batching, the batch is not released until every sequence has finished generating. LLM output lengths vary enormously — a yes/no answer and a long code explanation can arrive in the same batch. So seven sequences finish at token 40 and then occupy their slots, computing padding, for the 800 further steps the eighth sequence needs. GPU work is being spent on sequences that have already produced their final token, and those seven slots are unavailable to the requests queuing behind. Utilization collapses precisely in the workload where you most need it.

Continuous batching operates between decoding steps. After each step the scheduler checks which sequences hit an end-of-sequence token or their length limit, evicts them, admits waiting requests into the freed slots, and runs the next step on the new membership. The batch is a living set rather than a fixed cohort.

L2 — Mechanism: the three schemes precisely

Static batching. The batch size is fixed — compiled into an engine, set as a server configuration, or determined by the client sending an array of inputs. Every pass processes exactly that many items, padded if fewer are available. Its virtues are predictability and simplicity: you can size memory exactly, and a compiled engine can be optimized for one known shape. Its vices are that it cannot adapt to load and that a partly-filled batch wastes proportional capacity. Static batching remains sensible for offline/batch inference — scoring a million documents overnight, embedding a corpus — where you control all inputs and there is no user waiting.

Dynamic batching (server-side request batching). The server holds arriving requests in a queue with two knobs:

The scheduler dispatches when either the preferred batch size is reached or the window expires. This is exactly the mechanism Triton Inference Server implements and is best known for, and the course index names dynamic batching as Triton's signature capability alongside multi-framework backends, concurrent model execution, model versioning, and ensembles. Dynamic batching is a general-purpose feature: it works for image classifiers, embedding models, rerankers, and any request whose service time is roughly uniform.

Its weakness for generative LLMs is structural rather than a configuration mistake. Dynamic batching batches requests, and an LLM request occupies its slot for an unpredictable number of decoding steps. The batch composition is fixed at dispatch, so head-of-line blocking within the batch is unavoidable.

Continuous / in-flight batching (iteration-level scheduling). The scheduler runs per iteration — per decoding step — rather than per request. Each step it decides which sequences participate. The consequences:

• A finished sequence's slot is freed immediately, not at the end of the batch.
• A newly arrived request can be admitted between any two steps, so queueing delay is on the order of one decoding step rather than one full generation.
• Prefill and decode work must be interleaved, since new arrivals need a prefill pass while incumbents need decode steps. Serving stacks handle this differently — some run a prefill iteration whenever new work arrives (which briefly slows every incumbent's token stream), some chunk long prefills into pieces to bound that interference, and some separate prefill and decode onto different workers entirely. This is where implementations genuinely differ, and it is the origin of the "my token stream stutters when new users arrive" complaint.
• Memory management becomes the hard part. Membership changes every step, so cache allocation must be able to grow and release in small units. Contiguous per-request pre-allocation defeats the purpose, which is exactly why PagedAttention (12-07) and continuous batching are usually deployed together — the paged block allocator is what makes step-by-step admission memory-feasible.

The naming is a small trap worth defusing. In-flight batching is NVIDIA's term for this in TensorRT-LLM; continuous batching is the widely used generic term; iteration-level scheduling is the descriptive term from the research literature. They refer to the same idea. The course index lists in-flight/continuous batching among the LLM-specific features TensorRT-LLM adds on top of TensorRT, next to KV cache management, paged attention, and speculative decoding — so the association runs both ways and either name can appear in a question.

L3 — Scheduling policies, prefill interference, and where batching interacts with everything else

Once you accept iteration-level scheduling, several policy questions appear that a batch-per-request scheme never had to answer.

Admission control. How many sequences may be resident? Bounded by cache memory, so the scheduler must predict cache demand. A request declaring a 4,000-token maximum output may reserve capacity it never uses, so schedulers either reserve pessimistically (safe, wasteful) or admit optimistically and preempt when memory runs short. Preemption means either swapping a sequence's cache out to host memory and back, or discarding it and recomputing the prefill later. Both are real strategies with different cost profiles: swapping pays bandwidth, recomputation pays compute.

Fairness and priority. With slots freeing continuously, the scheduler chooses who gets them. First-come-first-served is the default and is fair but lets a long-running request hold a slot indefinitely. Priority classes let interactive traffic preempt bulk traffic. This is where multi-tenant serving policy lives.

Prefill/decode interference. A prefill pass over a 4,000-token prompt is a large compute burst. Slot it into a stream of decode steps and every incumbent sequence's inter-token latency spikes for that iteration. Chunked prefill — splitting the prompt into pieces processed across several iterations — bounds the spike at the cost of slightly longer TTFT for the arriving request. There is no free configuration here; you are choosing whose latency to protect.

Batching's effect on the latency distribution. This is the point the module brief flags and it deserves emphasis. Batching improves throughput and improves cost per token. Its effect on latency is mixed and asymmetric:

The tail moves more than the middle because queueing effects are multiplicative near saturation. A configuration that looks excellent on average throughput can be the one your users complain about, which is why 12-10 insists on percentiles rather than means and on reporting TTFT separately from inter-token latency.

Speculative decoding deserves a mention as an adjacent throughput technique that is often confused with batching. A small draft model proposes several tokens ahead; the large model verifies them in a single pass; accepted tokens are kept and rejected ones discarded. It attacks the same bandwidth bottleneck from the other side — instead of amortizing the weight read across many requests, it amortizes it across many candidate tokens of one request. That makes it a latency optimization for single requests, where batching is a throughput optimization across requests. It is another feature the course index attributes to TensorRT-LLM rather than TensorRT.

Where batching does not help. If you are serving one request at a time with no concurrency — a single-user desktop application, a nightly job with one document — batching has nothing to amortize across and the answer is quantization or a smaller model. And batching cannot fix a workload whose latency floor is already below the per-step time; you cannot batch your way to a faster first token.

The one-line discriminators, which is what exam questions actually turn on:

• Static: batch size is fixed in advance.
• Dynamic: the server forms batches from a queue with a time window. Batches requests. Triton's hallmark.
• Continuous / in-flight: the scheduler revises batch membership every token step. Batches iterations. TensorRT-LLM's and vLLM's hallmark.

And the distinction most likely to be tested as a pair: dynamic batching is not sufficient for LLMs, because it batches requests of unpredictable duration and therefore keeps finished sequences resident until the slowest one completes. Continuous batching removes exactly that inefficiency.

A constructed scenario with every figure derived from the stated assumptions. Assume:

• One decoding step for a batch of any size up to 32 takes 20 ms (a simplification that holds reasonably while the workload stays bandwidth-bound).
• Prefill for a 200-token prompt takes 60 ms and is not batched with decode in the dynamic case.
• Eight requests arrive simultaneously. Their output lengths are: 20, 25, 30, 40, 50, 60, 400, 600 tokens. Total output = 1,225 tokens.

Scheme A — one request at a time (no batching).

per request time  = 60 ms prefill + (output_tokens × 20 ms)
total decode steps = 20+25+30+40+50+60+400+600 = 1,225 steps
total time         = 8 × 60 ms + 1,225 × 20 ms = 480 + 24,500 = 24,980 ms ≈ 25.0 s
throughput         = 1,225 tokens / 25.0 s ≈ 49 tokens/s
last request TTFT  = it waited behind 7 full requests ≈ 24.4 s

Scheme B — dynamic batching, batch of 8, formed once.

prefill (batched)  = 60 ms  (all eight prompts prefilled together)
decode steps       = max(output lengths) = 600 steps   ← the batch runs until the slowest finishes
total time         = 60 + 600 × 20 ms = 12,060 ms ≈ 12.1 s
throughput         = 1,225 tokens / 12.1 s ≈ 101 tokens/s

Twice the throughput of no batching. But now count the waste:

slot-steps available = 8 slots × 600 steps      = 4,800 slot-steps
slot-steps used      = 1,225 (actual tokens)
utilization          = 1,225 / 4,800            = 25.5%
wasted slot-steps    = 3,575                    = 74.5%

Three-quarters of the GPU's batched capacity was spent on sequences that had already finished. Six of the eight requests completed within 60 steps and then occupied their slots, computing padding, for up to 540 further steps. That number — 74.5% waste — is the argument for continuous batching in a single figure.

Scheme C — continuous batching, up to 8 concurrent, plus 8 more requests queued behind. Extend the scenario: a second identical group of eight arrives immediately after the first. Under continuous batching, as each short sequence finishes its slot is refilled from the queue.

Steps 1–20:    8 resident. At step 20, request 1 (20 tokens) finishes → slot freed
Steps 21–25:   request 9 admitted (prefill + decode begins), request 2 finishes at 25
Steps 26–30:   request 10 admitted, request 3 finishes at 30
… and so on: every completion is immediately backfilled

The total decode work for sixteen requests is 2 × 1,225 = 2,450 token-steps. With 8 slots kept genuinely busy, the floor is:

minimum steps   = 2,450 / 8 slots ≈ 307 steps
minimum time    ≈ 307 × 20 ms ≈ 6.1 s  (plus prefill interleaving overhead)
throughput      ≈ 2,450 / 6.1 s ≈ 400 tokens/s

Against Scheme B's approach — where sixteen requests in two sequential batches of eight would take about 2 × 12.1 = 24.2 s for 2,450 tokens, or roughly 101 tokens/s — continuous batching is doing roughly four times the throughput on the same hardware. The gain is not from faster arithmetic; it is entirely from not spending slot-steps on finished sequences.

The caveat that keeps this honest. The 6.1 s figure ignores prefill interleaving: each of the eight newly admitted requests needs a prefill pass, and those passes displace decode steps and lengthen the incumbents' inter-token latency momentarily. Real throughput lands below the floor. The shape of the result — a large multiple, driven by output-length variance — is the robust conclusion, and it is why every modern LLM serving stack implements this. Note also the mechanism's sensitivity: if all eight requests had produced exactly 200 tokens each, dynamic batching's waste would be near zero and continuous batching would gain almost nothing. Continuous batching's advantage is proportional to the variance in output length, which in real chat and RAG traffic is very large.

The memory check that must accompany this. Eight resident sequences at, say, 2,000 tokens of context each, at 512 KB per token (from 12-05), is 8 × 2,000 × 512 KB ≈ 8.4 GB of KV cache. Sixteen resident would be 16.8 GB. Batch size is a memory decision before it is a throughput decision, and a scheduler that admits without checking cache capacity produces out-of-memory failures instead of throughput.

The general tuning rule: the queue window and the maximum concurrency are the two dials that trade latency for throughput, and neither has a universally correct value. Set them from your latency budget, then measure percentiles under realistic load.

Batching is claimed by objectives 4.1 (assist in deployment and evaluation of model scalability, performance, and reliability) and 4.4 (identify the system, hardware, or software components required to meet user needs). It also carries two product-identity facts that the course index flags as high-frequency exam material:

• Dynamic batching is the capability most associated with Triton Inference Server, alongside multi-framework backends, concurrent model execution, model repository and versioning, and ensembles. NIM-versus-Triton is a top reported confusable and dynamic batching sits on the Triton side of it — 12-13 handles the full comparison.
• In-flight (continuous) batching is one of the LLM-specific features that distinguishes TensorRT-LLM from TensorRT, together with KV cache management, paged attention, and speculative decoding. TensorRT-versus-TensorRT-LLM is the other top reported confusable and 12-08 resolves it.

So batching is worth learning twice: once as a mechanism, and once as a set of product associations. Where two answer options are technically defensible, the course index notes the NVIDIA-stack answer tends to be keyed, which makes "Triton's dynamic batching" and "TensorRT-LLM's in-flight batching" high-value phrases.

Question phrasings:

• "Which Triton Inference Server feature combines individual inference requests into a batch on the server?" — dynamic batching.
• "What is the primary benefit of batching during inference?" — higher throughput and GPU utilization by amortizing weight reads across requests.
• "Why is standard request-level batching inefficient for LLM text generation?" — output lengths vary, so finished sequences hold their slots until the longest sequence in the batch completes.
• "Which batching approach allows new requests to join a running batch?" — continuous / in-flight batching.
• "In-flight batching is a feature of which product?" — TensorRT-LLM.
• "What is the main trade-off when increasing the dynamic batching queue delay?" — better throughput at the cost of added latency, especially in the tail.
• "Which resource limits the maximum batch size for LLM serving?" — GPU memory, because each sequence carries its own KV cache.

Distractor families:

Dynamic batching operates at request granularity on the server: arriving requests are held in a queue, and the scheduler dispatches a batch when either a preferred batch size is reached or a configured maximum queue delay expires. Batch membership is then fixed for the whole batch's execution. Continuous batching — also called in-flight batching or iteration-level scheduling — operates at token granularity: after every decoding step the scheduler removes sequences that have finished, admits waiting requests into the freed slots, and runs the next step on the revised membership. The practical difference is decisive for LLMs, because output lengths vary wildly. Under dynamic batching a batch of eight where six requests finish in 40 tokens and one runs to 600 spends most of its slot-steps computing padding for already-finished sequences; under continuous batching those slots are refilled immediately. In the constructed example above, that difference was 25.5% versus near-full slot utilization. Dynamic batching is the feature Triton Inference Server is known for; in-flight batching is one of the LLM-specific features TensorRT-LLM adds over TensorRT.

Because LLM decoding is memory-bandwidth-bound, so the dominant cost of a decoding step is streaming the model's weights out of HBM — a cost that is paid once regardless of how many sequences participate in that step. With one sequence, the step performs matrix-vector products offering roughly two arithmetic operations per weight byte read, leaving the tensor cores idle while they wait on memory. With B sequences the same weight read supports B times the arithmetic as matrix-matrix products, so total tokens produced per unit time rises steeply while step duration rises only modestly. Batching therefore converts idle memory-wait time into useful computation. The gains continue until one of three things happens: KV-cache memory runs out (usually first), reading B caches per step becomes comparable to reading the weights, or arithmetic intensity rises far enough that compute becomes the genuine bottleneck.

Yes, and the honest framing is that batching trades latency for throughput. Under dynamic batching a request waits up to the configured queue window before its batch is dispatched, and then shares each forward pass with other sequences, so both time-to-first-token and per-token time rise. Under continuous batching the queueing component is much smaller — roughly one decoding step to be admitted — but resident sequences still share every step, so inter-token latency rises with occupancy, and interleaved prefill passes for newly admitted requests cause visible jitter in the token stream. The crucial detail is that the effect is not uniform: the p95 and p99 latencies rise more than the median, because queueing effects compound near saturation. Batching hides exactly the tail users complain about, which is why average latency is the wrong metric for tuning a batching configuration and percentiles are the right one.

GPU memory, almost always — specifically the memory needed for KV caches, since every sequence in the batch carries its own and cache size scales with batch_size × sequence_length. Using the arithmetic from 12-05, a 32-layer model with hidden size 4096 in BF16 costs about 512 KB of cache per token, so eight concurrent 2,000-token conversations need roughly 8.4 GB and thirty-two need about 33.6 GB — on top of the model's weights. Compute capacity is rarely the first limit for decode, because decode is bandwidth-bound and has arithmetic to spare. The secondary limits are cache read bandwidth, which grows with batch size and eventually rivals the weight read, and the latency budget, which caps how much sharing your users will tolerate. The practical consequence is that maximum batch size and maximum context length must be chosen together, because they multiply into the same memory term.

Yes. In-flight batching is NVIDIA's terminology for the technique in TensorRT-LLM, continuous batching is the common generic name popularized alongside vLLM, and iteration-level scheduling is the descriptive term used in the research literature. All three describe a scheduler that revises batch membership between decoding steps so completed sequences release their slots immediately and queued requests are admitted without waiting for the current batch to drain. Either name can appear in an exam question, and the association with the product matters: if a question asks which feature TensorRT-LLM adds beyond TensorRT, in-flight batching belongs on that list along with KV cache management, paged attention, and speculative decoding. If a question asks which Triton feature groups requests server-side, the answer is dynamic batching, which is a different mechanism at a different granularity.

They are complementary and are usually deployed together, because continuous batching creates the memory-management problem that PagedAttention solves. When batch membership changes every step, cache memory must be allocated and released in small increments at unpredictable times — a request admitted at step 40 needs cache immediately, and one that finishes at step 41 should return its memory instantly. A conventional allocator that reserves one contiguous block per request, sized to that request's maximum possible output length, wastes most of the reservation and cannot support fine-grained churn. PagedAttention stores the KV cache in fixed-size blocks that need not be contiguous, with a per-sequence block table mapping logical positions to physical blocks — exactly the arrangement an operating system uses for virtual memory. Blocks are allocated as a sequence grows and freed the moment it finishes, so high occupancy becomes achievable rather than theoretical. 12-07 develops the mechanism in full, including how block sharing enables copy-on-write reuse for shared prefixes and parallel samples.

• Batching works because decode is memory-bandwidth-bound: the weight read is paid once per step regardless of how many sequences share it.
• Static = fixed in advance. Dynamic = server queues requests within a time window. Continuous/in-flight = membership revised every token step.
• Dynamic batching is Triton Inference Server's hallmark. In-flight batching is one of the LLM-specific features TensorRT-LLM adds over TensorRT.
• Dynamic batching fails for LLMs because output lengths vary: finished sequences hold their slots until the longest one completes. The constructed example wasted 74.5% of slot-steps.
• Continuous batching's advantage is proportional to output-length variance, which is large in real chat and RAG traffic and zero in a badly designed benchmark.
• Maximum batch size is set by KV-cache memory, not compute. Batch size and context length multiply into the same term and must be chosen together.
• Batching raises throughput and lowers cost per token; it raises latency, and it raises p95/p99 more than the median.
• The two dials that trade latency for throughput are the queue window (dynamic) and maximum concurrency (continuous).
• Prefill passes for newly admitted requests interfere with incumbents' token streams; chunked prefill bounds the damage.
• Continuous batching and PagedAttention are complementary — variable membership demands block-based cache allocation.

Next: 12-07 takes up that last point in full. Storing a KV cache as one contiguous block per request, pre-sized to the longest output the request might produce, wastes most of the memory it reserves — and the fix is a 1960s operating-systems idea, virtual memory with paging, applied to attention.