The KV Cache Explained: Why LLM Generation Is Memory-Bound

Self-attention computes three projections of each token's hidden state: a query Q, a key K, and a value V. 04-01 derives the mechanism; the property that matters here is a structural asymmetry in how those three are used during generation.

When a decoder-only model generates token at position t, it needs:

• One query: the query vector of the new token at position t.
• All keys and values: K and V for every position from 1 to t, because attention scores the new query against every previous key and then takes a weighted sum of every previous value.

Now the crucial observation. Because of the causal mask, the key and value at position 5 depend only on the tokens up to position 5. They do not change when token 6 arrives, or token 600. They are computed once and are correct forever. The query, by contrast, is only ever needed for the current token and is discarded immediately.

So K and V are cacheable and Q is not. That is the whole idea, and it is why the structure is called a KV cache rather than a QKV cache.

What sits in memory is, per request, per layer, per attention head: a tensor of cached keys of shape [sequence_length, head_dim] and a matching tensor of cached values. Multiply by the head count, by the layer count, by the batch size, and you have the total. Two entries — one for K, one for V — which is why the constant 2 appears in every cache-size formula you will ever write.

Three consequences follow immediately, and they organize the rest of the lesson:

1. The cache is per-request state. Two users' caches are unrelated and cannot be shared (except for a shared prefix, discussed below). A stateless service that scales by adding replicas suddenly has stateful memory in it.
2. The cache grows with every token generated. A request that has produced 50 tokens holds a smaller cache than the same request at 2,000 tokens. Memory demand during a request is not constant.
3. The cache is read in full on every decoding step. This is the fact that makes generation memory-bandwidth-bound.

L1 — Intuition: the difference between re-reading the book and keeping a bookmark

Without a cache, generating a 1,000-token response means: run the model over 1 token, then over 2 tokens, then over 3, all the way to 1,000. The total work is proportional to 1 + 2 + 3 + … + 1000, which is about 500,000 token-forward-passes to produce 1,000 tokens. You have re-read the entire story from page one, five hundred times.

With a cache, generating that same response means: run the model over 1 token, 1,000 times. That is 1,000 token-forward-passes. Same output, 500× less work in this example — and the saving grows with length, because the wasted work in the uncached version grows quadratically while the cached version stays linear.

That is the trade in one line: the KV cache converts redundant computation into memory occupancy. You stop paying in FLOPs and start paying in gigabytes.

L2 — Mechanism: prefill, decode, and the exact memory formula

A single LLM request has two phases with fundamentally different behaviour. Recognizing that they are different is a genuine dividing line between people who understand LLM serving and people who do not.

Phase 1: prefill (also called the prompt phase or context phase). The whole input prompt is processed in one forward pass. Every token's K and V are computed and written into the cache, and the model produces the first output token. Because all prompt tokens are processed simultaneously, the GPU has large matrices to multiply and the arithmetic-to-memory-traffic ratio is high. Prefill is compute-bound. Its duration scales with prompt length and it is what determines time to first token (TTFT).

Phase 2: decode (also called the generation phase). Tokens are produced one at a time. Each step processes exactly one token: a tiny amount of arithmetic against the model's entire weight set plus the entire cache. The ratio of arithmetic to bytes moved is terrible. Decode is memory-bandwidth-bound. Its per-step duration is roughly constant and determines inter-token latency, which 12-10 develops in full.

The memory formula. This is the arithmetic you must be able to reproduce:

kv_cache_bytes = 2 × num_layers × num_kv_heads × head_dim
                   × sequence_length × batch_size × bytes_per_element

Term by term:

• 2 — one tensor for keys, one for values.
• num_layers — every layer keeps its own cache. A 32-layer model caches 32 times.
• num_kv_heads × head_dim — the size of one token's K vector in one layer. In standard multi-head attention num_kv_heads equals the query head count, so num_kv_heads × head_dim = hidden_size. With grouped-query or multi-query attention it is deliberately smaller, which is the point of those designs.
• sequence_length — prompt tokens plus generated tokens so far. This is what grows during the request.
• batch_size — number of concurrent requests sharing the GPU. Each carries its own cache.
• bytes_per_element — 2 for FP16/BF16, 1 if the cache is quantized to 8 bits (12-02).

A convenient simplification when heads are standard: 2 × num_layers × hidden_size × bytes gives you the per-token cache cost, and everything else is multiplication.

Grouped-query attention (GQA) and multi-query attention (MQA) deserve a named mention because they exist precisely to attack this formula. In standard multi-head attention every query head has its own K and V heads. MQA shares a single K/V head across all query heads. GQA is the middle ground: query heads are divided into groups and each group shares one K/V head. Since num_kv_heads sits directly in the formula, reducing it from, say, 32 to 8 cuts cache size by 4× with a modest quality cost — which is why GQA became near-universal in models designed for serving. If an exam question asks how to reduce KV-cache memory architecturally, GQA/MQA is the architectural answer, alongside cache quantization and shorter contexts as the operational ones.

L3 — Why bandwidth, not FLOPs, is the decode bottleneck

Here is the reasoning that makes the "memory-bound" claim precise rather than hand-wavy.

To generate one token in the decode phase, the GPU must:

1. Read every weight matrix in the model out of HBM. There is no way around it — every layer participates.
2. Read the entire KV cache for the batch out of HBM.
3. Perform matrix-vector products: a single token's hidden state against each weight matrix.

Step 3 is the tell. A matrix-vector product does about 2 × M × N floating-point operations while reading M × N weight values. That is roughly two arithmetic operations per weight element — an arithmetic intensity of about 2. Modern GPUs have arithmetic-to-bandwidth ratios in the hundreds: their tensor cores can perform hundreds of operations in the time it takes to fetch one byte from HBM. When your workload offers 2 operations per element and the hardware wants hundreds, the tensor cores sit idle waiting on memory. The step's duration is bytes_to_read / memory_bandwidth, and the arithmetic is essentially free.

Compare prefill. Processing 500 prompt tokens at once turns those matrix-vector products into matrix-matrix products: the same weights are read once and used against 500 token vectors. Arithmetic intensity jumps by roughly the number of tokens in the pass, the tensor cores become the limit, and the phase is compute-bound.

This single asymmetry explains an enormous amount:

• Why batching works so well for decode. Adding a second concurrent request to the batch reads the same weights and does twice the arithmetic. The weight-reading cost is amortized across requests, so throughput rises nearly linearly while per-token latency barely moves — up to the point where the cache exhausts memory or the arithmetic finally becomes the limit. This is the entire economic argument of 12-06.
• Why weight quantization speeds up decoding even without integer math. Halving bytes per weight halves step 1's traffic, and step 1 is the bottleneck. 12-01 states this; here is the mechanism.
• Why decode throughput is capped by the cache, not by FLOPs. You cannot batch beyond the memory available for concurrent caches. Memory capacity sets your maximum batch size, and maximum batch size sets your throughput ceiling.
• Why the KV cache is read in full every step and therefore adds bandwidth cost of its own. A large batch with long sequences can make cache traffic rival weight traffic, at which point growing the batch stops helping.
• Why long contexts are expensive twice over. Long prompts cost compute in prefill, and long sequences cost both memory capacity and bandwidth throughout decode.

Prefix caching / prompt caching is the one form of cache sharing that works. If many requests begin with the same long system prompt or the same retrieved document, the K and V for that shared prefix are identical across those requests and can be computed once and reused. This is a real and substantial optimization for RAG and chat systems with heavy shared preambles, and it is a direct consequence of keys and values depending only on preceding tokens. It does not extend to the divergent parts of requests, which remain strictly per-request.

What the cache does not change. Attention itself still scores the new query against all cached keys, so attention's cost per decoding step grows with sequence length even though the model's weight-reading cost does not. The cache eliminates the recomputation of keys and values; it does not eliminate attention's dependence on history. The quadratic cost 04-01 describes is about processing a full sequence; the cache converts generation to linear-per-token but each of those steps still touches the whole history.

And the two comparisons that get confused most often:

A constructed scenario, every number derived from the stated assumptions. Model: 32 layers, hidden size 4096, 32 attention heads, head_dim = 4096 / 32 = 128, standard multi-head attention (so num_kv_heads = 32), cache in BF16 at 2 bytes per element, roughly 7 × 10⁹ parameters.

Step 1 — cost of one token's cache, across the whole model.

per layer, per token: 2 (K and V) × 32 heads × 128 head_dim × 2 bytes
                    = 2 × 4096 × 2 = 16,384 bytes = 16 KB
across 32 layers:     16,384 × 32 = 524,288 bytes = 512 KB per token

Half a megabyte per token. That is the number worth carrying in your head for a model of this shape. Note the shortcut: 2 × num_layers × hidden_size × bytes = 2 × 32 × 4096 × 2 = 524,288.

Step 2 — one request with a 2,000-token context.

2,000 tokens × 524,288 bytes = 1.049e9 bytes ≈ 1.05 GB

One conversation, one gigabyte. Against 14 GB of BF16 weights this is already 7.5% of the model's own footprint — for a single user.

Step 3 — thirty-two concurrent requests at 2,000 tokens.

32 × 1.05 GB ≈ 33.6 GB of KV cache

The cache now exceeds the weights by 2.4×. On a device with 24 GB total this configuration is impossible; on one with 80 GB it consumes most of what remains after the 14 GB of weights. This is the arithmetic that makes concurrency a memory-planning problem rather than a threading problem.

Step 4 — the capacity question, answered properly. Suppose 40 GB of usable device memory.

weights (BF16)                      14.0 GB
runtime, workspace, fragmentation    2.0 GB   (assumed)
                                    -------
available for KV cache              24.0 GB

max concurrent requests at 2,000 tokens = 24.0 / 1.05 ≈ 22 requests
max concurrent requests at 8,000 tokens = 24.0 / 4.19 ≈  5 requests
max concurrent requests at   500 tokens = 24.0 / 0.26 ≈ 91 requests

Read that last block carefully, because it is the most commercially important consequence in the module: quadrupling the context length quarters your concurrency. Not "slows it down" — quarters it. A product decision to raise the context limit from 2,000 to 8,000 tokens is a decision to serve roughly a quarter as many simultaneous users per GPU, and therefore to multiply your cost per request by about four. 12-09 turns that into currency.

Step 5 — the levers, quantified against this baseline.

Two readings. First, attacking the cache multiplies concurrency, while attacking the weights adds a fixed amount of headroom — because the cache term scales with concurrency and the weight term does not. Second, GQA is an architectural property of the model you chose, which means model selection is a serving-capacity decision, not only a quality decision. That is a genuinely non-obvious insight and it belongs in the model-selection conversation that 03-03 and 11-09 frame.

Step 6 — where the memory actually goes when it goes wrong. The formula above assumes the cache is packed perfectly. Naive implementations pre-allocate a contiguous block per request sized to the maximum possible sequence length, because the cache must be contiguous and you do not know in advance how many tokens will be generated. A request that could produce 4,000 tokens but stops at 200 has reserved 20× the memory it used. Aggregate that across a batch and most of your cache memory is reserved and empty. That is internal fragmentation, and eliminating it is exactly what 12-07 is about.

The last row is the operational habit to leave with: the per-token cache cost is computable from a model's config before you deploy anything. Layers, hidden size, KV head count, and dtype are all published. Anyone who has done this arithmetic can predict a serving configuration's capacity on paper, and anyone who has not will discover it through out-of-memory errors in production.

The KV cache is claimed by objectives 4.1 (deployment and evaluation of model scalability, performance, and reliability) and 4.4 (identifying the system, hardware, and software components required to meet user needs), and it is a named feature in the material the study guide points candidates at. Specifically, the course index records that TensorRT-LLM is distinguished from plain TensorRT by the LLM-specific features it adds — KV cache, paged attention, in-flight/continuous batching, speculative decoding — and that TensorRT vs TensorRT-LLM is one of the most-reported confusables on the exam. You cannot answer that question reliably without knowing what a KV cache is. This lesson therefore does double duty: a serving mechanism and half of a product-identity question.

The exam also asks the memory-and-capacity form of the question, since capacity planning falls squarely under 4.4. Expect the reasoning, not the algebra: you are more likely to be asked what the cache scales with than to be asked to compute bytes.

Question phrasings:

• "What is the purpose of the KV cache in autoregressive generation?" — to avoid recomputing keys and values for previous tokens on every decoding step.
• "KV cache memory grows with which of the following?" — batch size and sequence length. (Not parameter count; that is the weights.)
• "Why is LLM token generation described as memory-bandwidth-bound?" — each step reads all weights and the whole cache while performing very little arithmetic per byte.
• "Which phase of inference is compute-bound?" — prefill, because the whole prompt is processed in one pass.
• "A team must serve more concurrent users on the same GPU. Which change most directly increases concurrency?" — reduce per-request KV-cache size (shorter context, cache quantization, a GQA model).
• "Which of these is an LLM-specific feature of TensorRT-LLM rather than TensorRT?" — KV cache management, paged attention, in-flight batching, speculative decoding.
• "Why does reducing weight precision improve decode latency?" — fewer bytes to stream from HBM in a bandwidth-bound phase.

Distractor families:

A KV cache is the GPU-resident store of the key and value tensors a transformer has already computed for the tokens it has already processed in a given request. Its purpose is to make autoregressive generation efficient: without it, producing each new token would require recomputing keys and values for the entire preceding sequence, making a 1,000-token response cost on the order of 500,000 token-forward-passes instead of 1,000. Keys and values are cacheable because the causal mask means position n's K and V depend only on tokens up to n and therefore never change as generation continues; queries are not cached because a query is used once, for the token being generated, and then discarded. The cache is per-request, it grows by one entry per layer per token as generation proceeds, and it is read in full on every decoding step.

Because the decode phase performs a tiny amount of arithmetic per byte of memory it must read. To generate one token the GPU streams every weight matrix in the model out of HBM, plus the entire KV cache for the batch, and then performs matrix-vector products — roughly two floating-point operations per weight element. Modern GPUs can perform hundreds of operations in the time it takes to fetch a byte from HBM, so at an arithmetic intensity of about 2 the tensor cores are idle and the step's duration is set by bytes_read / memory_bandwidth. The prefill phase is the opposite: processing an entire prompt in one pass turns those matrix-vector products into matrix-matrix products, reusing each weight across every prompt token, which pushes arithmetic intensity high enough that compute becomes the limit. Prefill is compute-bound; decode is memory-bandwidth-bound, and nearly every LLM serving optimization is an attempt to move fewer bytes during decode or to amortize those bytes across more requests.

Use 2 × num_layers × num_kv_heads × head_dim × sequence_length × batch_size × bytes_per_element. The leading 2 is keys plus values. When attention is standard multi-head, num_kv_heads × head_dim equals the hidden size, so the formula simplifies to 2 × num_layers × hidden_size × bytes_per_element for the cost of a single token across the whole model, and everything else is multiplication by sequence length and batch size. For a 32-layer model with hidden size 4096 in BF16 that is 2 × 32 × 4096 × 2 = 524,288 bytes, or 512 KB per token — so a 2,000-token conversation holds about 1.05 GB and thirty-two such conversations hold about 33.6 GB. Every input to that formula is published in a model's configuration file, which means serving capacity is predictable on paper before any hardware is provisioned. Grouped-query and multi-query attention change num_kv_heads directly and are therefore the largest single architectural lever on the result.

Dramatically faster in compute, at a real cost in memory. Without a cache, generating token t requires a forward pass over all t tokens, so the total cost of an n-token response is proportional to n²/2. With a cache, each step processes exactly one token and the total cost is proportional to n. For a 1,000-token response that is a roughly 500× reduction in redundant work in the constructed example above, and the advantage widens as responses get longer. What you pay is GPU memory that scales linearly with both sequence length and batch size, plus the bandwidth cost of reading that cache on every step. The cache is not optional in any practical serving system — the question is never whether to use it but how to manage its memory, which is why paging, quantization, and eviction policies exist.

Prefill processes the entire input prompt in a single forward pass, populating the KV cache for every prompt token and emitting the first output token; decode then produces subsequent tokens one at a time, reading the cache and appending one entry per layer per step. The operational differences are large enough that they must be measured separately. Prefill is compute-bound, scales with prompt length, and determines time to first token. Decode is memory-bandwidth-bound, has roughly constant per-step cost, and determines inter-token latency and therefore the perceived streaming speed. Optimizations diverge accordingly: prefill benefits from arithmetic throughput and from prefix caching that lets a shared prompt be prefilled once, while decode benefits from fewer bytes moved — quantized weights, a smaller cache — and from larger batches that amortize weight reads. Reporting a single "latency" number that averages the two phases hides exactly the behaviour users notice, which is the argument 12-10 makes.

Only the part of it that corresponds to an identical prefix. Because a key or value at position n is a function of the tokens up to n, two requests that begin with the exact same tokens — the same system prompt, the same retrieved document, the same few-shot examples — produce identical K and V for that shared span, so it can be computed once and reused. That is prefix caching (also called prompt caching), and it is genuinely valuable in RAG and chat systems where a long preamble is common to most traffic: it removes redundant prefill work and its share of cache memory. The moment the token sequences diverge, everything after the divergence point is request-specific and cannot be shared, because each subsequent key depends on the divergent tokens. So the accurate statement is: the KV cache is per-request state with a shareable common prefix, not shared state with per-request exceptions.

• The KV cache stores keys and values for tokens already processed so each decoding step runs a forward pass over one token instead of the whole sequence. Queries are never cached.
• It trades compute for memory: generation goes from quadratic to linear in work, and memory grows linearly with sequence length × batch size.
• Per-token cache cost is 2 × num_layers × hidden_size × bytes under standard attention — about 512 KB per token for a 32-layer, 4096-hidden model in BF16.
• The cache is per-request state. Only an identical prefix can be shared, via prefix caching.
• Prefill is compute-bound; decode is memory-bandwidth-bound. They need separate measurement and separate optimizations.
• Decode is bandwidth-bound because it does about two operations per weight byte read while the hardware wants hundreds. The tensor cores wait on HBM.
• That is why batching raises throughput almost for free, and why quantizing weights speeds up decoding.
• Concurrency is inversely proportional to context length. Quadrupling the context limit quarters the users per GPU — and roughly quadruples cost per request.
• The largest architectural lever on cache size is GQA/MQA, which makes KV-head count a model-selection criterion.
• Every input to the cache formula is in the model's config file, so capacity is predictable on paper before deployment.

Next: 12-06 cashes in the central insight of this lesson. If decode is bandwidth-bound and weight reads are amortized across whatever requests share a pass, then how you group requests into batches is the highest-leverage throughput decision in the whole stack — and static batching, dynamic batching, and continuous (in-flight) batching are three very different answers with three very different latency profiles.