The context window: what it is and how to budget it

A context window is a limit on the length of the token sequence a model can attend over at once, expressed in tokens. Not characters, not words, not "pages" — tokens, as produced by the model's tokenizer from 02-02. If a model is documented with a 128,000-token context window, that is the total, and the total is shared.

Four properties define it, and every one of them is a place systems break:

It is a total, not a per-component allowance. There is no separate budget for the prompt and the response. In a decoder-only model the prompt and the generated output occupy the same sequence — 04-04 showed generated tokens being appended to the input — so a 4,000-token prompt in an 8,000-token window leaves at most 4,000 tokens of room for the answer, and usually fewer once overheads are counted.

It is measured in the model's own tokens. From 02-03: tokens are not words. The same document is a different number of tokens for different tokenizers, and code, non-English text, unusual names, and long numbers all tokenize less efficiently than plain English prose. You cannot budget in characters and expect it to hold.

It is hard. There is no "mostly fits." A model with a maximum sequence length either has a representation for position n and the memory to process it, or it does not.

It is a property of a specific model version, not of a model family or a vendor. Context-window figures change between releases, differ across variants of the same family, and are sometimes different for input and for what a given API will let you generate. Every number you carry in your head about a specific model is a number that will expire. Read it from the current documentation for the exact version you are calling, and quote it that way.

The definition worth reproducing under time pressure: the context window is the maximum total number of tokens — prompt plus completion — that a model can process in one pass.

L1 — Why a limit exists at all

04-01 and 04-02 already supplied the answer, and the reason it matters here is that the causes are separate and get confused constantly. There are three independent reasons a model has a maximum usable length.

Representational. With learned absolute positional embeddings, the position table has a last row. Position 2,049 in a model built with 2,048 rows has no vector to look up. This is categorical: not slow, not degraded — undefined. A computed scheme such as sinusoidal, or a relative/rotary scheme, does not have this particular wall.

Distributional. Whatever the scheme, positions far beyond the lengths that training exercised are undertrained. A sinusoidal encoding is defined at position 50,000; a model that never saw a 50,000-token sequence has not learned to use position 50,000 well. Defined is not the same as competent, and this is the single most under-appreciated fact about long-context models.

Resource. From 04-01, the attention score matrix is n × n, and during training it must be materialised for the backward pass. At inference, per-token keys and values accumulate in the KV cache. Even a model that can represent a long sequence may not fit one in memory alongside a useful batch size, or may not process one inside a latency budget.

Vendors extend context windows by working on all three at once — rescaling or changing the position scheme, continuing training on longer sequences, and adopting memory-efficient attention implementations. Which is why the honest framing of a context-window increase is: the ceiling moved; the quality at the new ceiling is an empirical question you must test on your own data.

L1 — What counts against the window

Everything in the sequence. Enumerated, because teams routinely forget half of it:

The last two rows are where surprises come from. A team budgets prompt components carefully, forgets to reserve room for the answer, and gets truncated output that looks like a model quality problem.

L2 — The budgeting formula

Budget in one direction only: reserve the completion first, then allocate what remains.

context_window                                  (from the model's docs, for this version)
  − reserved_completion    (max_tokens you will allow)
  − system_prompt
  − chat_template_overhead
  − tool_definitions
  − safety_margin          (tokenization is not exactly predictable)
  ─────────────────────────────────────────────
  = available for history + retrieved context + the question

Then split what is left between history and retrieval according to which matters more for the task, and enforce the split in code. "Enforce in code" is the part that distinguishes a system that degrades predictably from one that fails mysteriously: something must decide what to drop, and if you do not decide, the API or the framework will decide for you, silently, and usually by cutting whatever came last.

Two rules make this reliable.

Reserve the completion before anything else. If you need 800 tokens of answer, 800 tokens are gone before you consider retrieval. Systems that allocate retrieval first and hope the answer fits produce truncation under exactly the conditions where the answer matters most — complex questions that pulled in a lot of context.

Carry a safety margin. You cannot always predict the token count of text you have not tokenized yet, template overheads vary by library version, and a retrieved chunk may tokenize worse than its character count suggests. A margin of a few percent turns a class of hard failures into a class of slightly-less-context.

L2 — What happens when you exceed it: five distinct behaviours

This is where the practical damage is, and the behaviours are genuinely different from one another:

The fourth and fifth rows are the reason this lesson is the deepest in the module. A loud error costs you an incident and a fix. A silent drop costs you trust in the model, weeks of chasing a phantom quality regression, and — if the dropped item was a safety instruction or an access-control constraint — a genuine incident. Instrument this. Log the token count of every component of every request. That single piece of telemetry converts the entire failure class from mysterious to obvious, and it belongs in the monitoring discipline 12-14 sets out.

L3 — In the window is not the same as used well

A token inside the window is available to attention. It is not attended to. Those are different claims and the gap between them is measurable.

From 04-01, attention distributes a limited amount of weight across all positions. As the sequence grows, the competition grows. Published work on long-context behaviour has repeatedly found that models use information at the beginning and end of a long input more reliably than information in the middle — the finding usually called lost in the middle. The practical upshot is that a 100,000-token window does not mean 100,000 tokens of equally usable context, and that placing critical material at the extremes of a long prompt is a real technique rather than folklore. 07-08 owns this and turns it into chunk-ordering policy.

This produces the most important distinction in long-context engineering, and it is worth naming as a pair of numbers:

• The supported length is what the documentation says the model accepts.
• The reliable length is how much context your system can put in front of the model and still get correct answers, measured on your own data.

They are not the same number, the second is smaller, and only you can measure it — with the evaluation harness from 01-08, scaled up in 09-01. A team that treats the supported length as the reliable length ships a long-context feature that works in the demo and degrades in production.

L3 — Why a bigger window is not free

Three costs, each independent:

Money. Priced-per-token APIs charge for input tokens. Doubling the context you send doubles that component of the bill on every request, forever, whether or not the extra context helped. 12-09 does this arithmetic properly.

Latency. Prefill from 04-04 processes the whole prompt, so time to first token grows with prompt length. And attention's contribution to prefill grows with the square of length, per 04-01. A long prompt is felt by the user as a slow start.

Memory and concurrency. Every token in the sequence contributes keys and values to the KV cache, for every layer, for every concurrent request. Long contexts therefore reduce how many requests you can serve simultaneously on fixed hardware — the throughput cost is real even when the latency cost is tolerable. 12-05 and 12-06 are the lessons where this becomes capacity planning.

Put together: the context window is the most expensive dial in an LLM system, and "just use a bigger window" is the most expensive answer to any question in this module. The cheaper answers are almost always retrieval that returns less and better, history that is summarised rather than accumulated, and prompts that are shorter because they are better written.

Two of these confusions are high-consequence enough that the course flags them forward to the lessons that resolve them. Both are here.

Context window vs chunk size

This is the confusion that produces broken RAG pipelines, and it is worth being precise: chunk size is a granularity decision about how you split documents for retrieval; the context window is a ceiling on how much you can send. They interact, they constrain each other, and they are not the same variable.

The mistake is to reason "my window is large, so I will use large chunks." Chunk size should be set by what makes a retrievable, semantically coherent unit — a decision about embeddings and precision, from 03-02 — not by what the window can hold. A large window buys you more chunks, not bigger ones. 06-02 owns chunking strategy and settles this pair properly: chunk size is granularity, the context window is a ceiling.

Context window vs the KV cache

The second flagged pair, and the confusion here is about what kind of thing each one is.

The relationship is worth stating carefully because it is the useful part: the context window tells you what a single request may contain; the KV cache tells you how much memory that request occupies while it is being served. A model with a huge advertised window may be practically limited to shorter requests on your hardware, because serving several concurrent long sequences exhausts the cache memory before it exhausts the window. That is why the advertised number and the deployable number differ, and it is 12-05's subject.

Context window vs the other length limits people conflate

The embedding row is a real production bug, not a hypothetical: a pipeline that chunks to fit the generator's window and then feeds those chunks to an embedding model with a much shorter maximum silently truncates every chunk at embedding time, so the second half of every chunk is never represented in the index. Retrieval quality collapses for reasons that are invisible in the retrieval code. 03-03 warns about it; this is where the arithmetic makes it obvious.

A support assistant answering questions over a product manual. All figures below are constructed for illustration — a stated model window and hand-chosen component sizes, not measurements of any deployed system. The point is the procedure, and the procedure is what transfers.

Given: a model documented at an 8,000-token context window for this version. Requirement: answers up to roughly 600 words, grounded in retrieved manual sections, with a few turns of conversation memory.

Step 1 — reserve the completion. 600 words of English prose is roughly 800 tokens at the rule of thumb from 02-03 — and a rule of thumb is all it is; verify with the actual tokenizer. Round up for safety.

reserved completion (max_tokens)                =    900

Step 2 — subtract the fixed overheads. Measured once with the tokenizer, not estimated:

system prompt (role, tone, refusal policy)      =    220
chat template and role markers, ~6 messages     =     60
tool definitions (one search tool)              =    130
safety margin (~3% of the window)               =    240
                                                  ─────
fixed overheads                                 =    650

Step 3 — compute what is available.

8,000  −  900 (completion)  −  650 (overheads)  =  6,450 tokens available

Step 4 — allocate the available pool. Two claimants: conversation history and retrieved context, plus the question itself.

user's current question                          =    60
conversation history budget                      = 1,500
retrieved context budget                         = 4,890
                                                  ─────
                                                   6,450

Step 5 — turn the retrieval budget into a retrieval policy. With 4,890 tokens for context and chunks of roughly 400 tokens each:

4,890 / 400  ≈  12 chunks

So k = 12, with a hard truncation rule if the retrieved chunks come back larger than expected. Note what just happened: the retrieval parameter k was derived from the budget, not chosen because 12 felt right. That is the whole discipline.

Step 6 — decide the eviction policy before you need it. When the conversation grows past its 1,500-token allowance, something must give. State the rule explicitly:

1. Never drop the system prompt.
2. Never drop the current question.
3. Summarise the oldest turns into a running summary (see 12-11);
   if summarisation is unavailable, drop the oldest turns first.
4. If retrieval overshoots, drop the lowest-ranked chunks — which is
   why rank order must be preserved through assembly (07-08).
5. Log every drop, with counts, per request.

Rule 5 is the one teams omit and the one that pays. Without it, you find out about eviction from a user.

Step 7 — now change the model and re-derive. Suppose you move to a version documented at 32,000 tokens. The naive move is to raise k from 12 to 12 × 4 = 48 chunks. Consider what that actually buys and costs:

• Cost: roughly four times the input tokens per request, on every request, forever.
• Latency: a longer prefill and therefore a slower first token.
• Quality: 48 chunks means chunks ranked 13 through 48 — by construction, the ones retrieval judged least relevant. You have added mostly noise, and the lost-in-the-middle effect means the genuinely relevant early chunks now compete with far more material.

The better use of a larger window, in order of usual value: raise k modestly and spend the rest on reranking so the chunks you do send are better (07-07), on a longer completion budget if answers were being truncated, and on more conversation history if the assistant was forgetting. A bigger window is a budget increase, not a licence to stop budgeting.

Step 8 — sanity-check the whole thing against the embedder. If the chunks are 400 tokens and the embedding model's maximum sequence length is 512, you are fine. If the embedding model's maximum is 256, every chunk has been silently halved at index time and the retrieval quality problem you are about to debug has nothing to do with the generator at all.

The RAG budget above is the static case. The harder case is a conversation, because one component grows without bound while every other one stays fixed.

The setup. Same 8,000-token model. System prompt 220 tokens. No retrieval, to isolate the effect. Each exchange averages 150 tokens of user message and 350 tokens of assistant reply, so roughly 500 tokens per turn including template overhead. Completion reserved at 900. All figures constructed.

Trace the accumulation.

turn   history tokens   + system + completion reserve   total committed
  1            500          220           900               1,620
  4          2,000          220           900               3,120
  8          4,000          220           900               5,120
 12          6,000          220           900               7,120
 14          7,000          220           900               8,120   ← over

At turn 14 the request no longer fits, and it fails for a reason that has nothing to do with turn 14's content. The user experiences it as "the assistant broke," or worse, as "the assistant suddenly forgot the rules" — which is exactly what happens if the framework's eviction policy trims from the left and takes the system prompt with it.

Three things this trace makes visible.

First, failure is a function of turn count, not of any single message. No individual request looked large, which is why long-conversation failures are hard to reproduce from a single log line and easy to predict from arithmetic. No exotic telemetry is needed to catch it either: a running total logged per request would have shown the slope from turn 4.

Second, the completion reserve is being paid on every turn while the history grows underneath it. Reserving 900 tokens is right, and it means the effective history ceiling is lower than the window suggests.

Third, the fix is a policy, and the policy choice has real consequences. Which brings us to the decision table.

The general decision rule: fix the growing component first, and buy a bigger window last. Summarising history, trimming a bloated system prompt, and reranking so you can retrieve fewer chunks are all cheaper than paying for four times the input tokens on every request for the life of the product.

And one honest caveat about summarisation: it is itself a generation step, so it costs a call, adds latency, and can drop or distort a detail that later turns out to matter. It converts a hard failure into a soft, silent one. That is usually the right trade, and it is a trade — say so in your design documents rather than presenting summarisation as free.

The context window is examinable on two separate blueprint fronts, which is why it earns the most depth in this module.

Under Core ML and AI, objective 1.3 requires you to build LLM use cases such as RAG, chatbots, and summarisers — and every one of those three is a context-budgeting problem before it is anything else. Objective 1.9 on prompt engineering depends on it too, since a prompt technique that does not fit is not a technique. The blueprint's own must-know content for tokenization names "token counting and context-window budgeting" explicitly, which puts the arithmetic itself in scope rather than just the concept.

Under Software Development, objective 4.4 requires identifying the system data, hardware, or software components required to meet user needs. Context length drives memory, latency, and cost, so a candidate who cannot reason about it cannot size a deployment. The blueprint's chatbot content names "context-window budgeting" and "truncation vs summarization memory" directly — the exact material in §5 above.

Tokenization is in the highest-frequency reported topic tier, and context-window budgeting is the practical half of tokenization. The exam is pitched at general level: expect to be asked what counts against the window, what happens when you exceed it, and which lever to pull — not to compute a byte-exact cache size.

Question phrasings to expect:

• "What is a context window?" → the maximum total number of tokens, prompt plus completion, a model can process in one pass.
• "Which of the following counts against the context window?" → all of them: system prompt, few-shot examples, history, retrieved context, tool definitions, the question, and the generated output.
• "A model has a 4,096-token window and the prompt is 3,900 tokens. What is the likely outcome of requesting a 500-token answer?" → the completion is truncated or the request fails; prompt and completion share one budget.
• "What happens when input exceeds the context window?" → truncation or an error, depending on the implementation — not graceful degradation.
• "A multi-turn chatbot degrades after many turns. What is the most likely cause and the standard remedies?" → history has consumed the window; truncate or summarise it.
• "What is the difference between chunk size and the context window?" → chunk size is a retrieval-granularity choice you set at ingestion; the window is a model limit on total tokens.
• "Why does doubling the context length increase cost more than double?" → attention's quadratic term, plus per-token KV cache growth.
• "A team upgrades to a model with a much larger window and quality does not improve. Why?" → more context is not more useful context; low-ranked chunks add noise, and long-context attention is uneven.
• "Which parameter reserves room for the model's answer?" → max tokens, which caps generation within the shared window.

Distractor families, and why each is wrong:

Every token in the sequence, and the list is longer than most teams' mental model. In rough order of how often each is forgotten:

1. The generated completion. It shares the sequence with the prompt. Reserve it first.
2. Tool and function definitions. JSON schemas are verbose in tokens, and they are sent on every request whether or not a tool is called.
3. Chat template overhead. Role markers and special tokens per message are small individually and accumulate across a long conversation.
4. The full conversation history, including the assistant's own previous replies — which are often longer than the user's messages.
5. Tool call results, which are unbounded unless you truncate them yourself. A search API returning a full page of results can consume more window than everything else combined.
6. Retrieved context, including any metadata, headers, or source labels you prepend to each chunk. The labels are not free.
7. Few-shot examples, which are usually the largest fixed block in a well-engineered prompt.
8. The system prompt, present on every request forever — which is why trimming it is unusually high-leverage.
9. Formatting. Markdown, indentation, and pretty-printed JSON cost tokens. Compact serialisation is a legitimate optimisation when budgets are tight.
10. The user's question, which is usually the smallest item and the one crowded out by everything above.

The operational discipline that follows: count each component with the model's own tokenizer and log the counts. Not estimates from character counts, not a rule of thumb — the tokenizer, at request time, logged. 02-03 gave you the counting; this is the habit it was for. The instrumentation is cheap and it converts an entire class of silent failures into a dashboard.

One of five things, and which one depends entirely on the layer that notices first. §2 tabulated them; here is what each means for you.

A hard error is the outcome you should design for. The request is rejected with a length error, you catch it, you trim or summarise and retry. It is loud, it is attributable, and it is the only behaviour that guarantees you find out.

Truncation from the left drops the oldest tokens. In a chat request that means the system prompt goes first — the tone rules, the refusal policy, the "never reveal internal pricing" instruction. The request then succeeds, and returns an answer produced by a model that was never told the rules. If any of your safety or access constraints live in the system prompt, this is a security-relevant failure mode, and it looks exactly like a normal response.

Truncation from the right drops the newest tokens, which can include the user's actual question. The model answers something adjacent to what was asked and sounds fine doing it.

Mid-generation stopping happens when prompt plus completion crosses the limit during decode. Output ends abruptly. Teams reliably misdiagnose this as a model quality issue, because it presents as incoherent or incomplete reasoning rather than as an error.

Framework-managed eviction is the most dangerous: an orchestration library trims something to make the request fit, does not surface what it removed, and returns a confident answer built on an incomplete prompt. You cannot debug what you were not told about.

Three defences, in priority order:

• Count before you send. Tokenize every component, sum, compare against the budget, and act. This turns every one of the failure modes above into a decision you made.
• Make eviction explicit and ordered. Write the priority list — system prompt, current question, top-ranked chunks, recent history, older history — and enforce it in your own code rather than inheriting a library's default.
• Log the drops. Count of tokens dropped, and from which component, per request. This is a monitoring requirement, not a nice-to-have, and it belongs alongside the drift and quality telemetry in 12-14.

No, and the reasoning is worth having ready because the question comes up in architecture reviews as often as on exams.

Cost. Input tokens are billed. Sending an entire 200,000-token corpus on every request costs orders of magnitude more than sending twelve well-chosen chunks, on every request, forever. Retrieval is a cost-reduction technology as much as a quality one. 12-09.

Latency. Prefill processes the whole prompt, so time to first token scales with prompt length, and attention's contribution scales worse than linearly. A short prompt is a fast first token, and users feel first tokens.

Precision. More context is not more signal. Padding the prompt with material ranked as less relevant adds distractors, and the lost-in-the-middle effect means the good material now competes with more noise. 07-03 and 07-08 both come back to this.

Freshness and scope. A corpus that changes does not fit in a prompt you wrote yesterday, and most real corpora are far larger than any window. Retrieval is how you address a body of knowledge you cannot enumerate.

Access control. Retrieval is where per-user permission filtering happens. Stuffing everything into the context means every user sees everything — a hard blocker in most enterprises, and the reason 07-05 exists.

Provenance. Retrieval knows which document an answer came from and can cite it. Context-stuffing loses the mapping between claim and source, which is exactly the property that makes grounded systems auditable. 07-11.

The correct nuance, and the one that separates a good answer from a rote one: a larger window genuinely does change RAG design. It relaxes the pressure toward tiny chunks, makes reranking-then-sending-a-few-more viable, allows more conversation history alongside retrieval, and makes whole-document processing feasible for genuinely single-document tasks. It changes the parameters of retrieval; it does not remove the reasons for it. And there are cases where RAG is the wrong tool for reasons unrelated to window size — no corpus to retrieve from, a need to change style rather than facts, a hard latency floor — which 07-12 handles.

Four steps, in this order, and the order is the content.

Step 1 — read the limit for your exact model version. Not the family, not last quarter's documentation, not a remembered number. Context windows change between releases, and some APIs document input and output limits separately. Record it in configuration, not in a comment.

Step 2 — reserve the completion first. Decide the longest answer you will allow, set max_tokens to it, and subtract it before allocating anything else. This one habit prevents the most common production truncation.

Step 3 — measure the fixed overheads once, with the real tokenizer. System prompt, chat template, tool definitions. These are constants per deployment; measure them, do not estimate them, and re-measure when you change the prompt or add a tool. Then subtract a safety margin, because tokenization of unseen text is not perfectly predictable.

Step 4 — allocate the remainder with an explicit priority order, and enforce it.

priority 1  system prompt          — never evicted
priority 2  the user's question    — never evicted
priority 3  top-ranked retrieved chunks, in rank order
priority 4  recent conversation turns
priority 5  older turns            — summarised, then dropped

Then instrument all of it: per-component token counts, total, and any drops, logged per request. If you do only one thing from this lesson, do that — it is the difference between a system whose limits you know and a system whose limits you discover from users.

Two closing habits worth the discipline. Budget in tokens from the model's own tokenizer, never in characters or words, because 02-03's lesson is that the conversion rate is not a constant. And measure your reliable length rather than trusting the supported one: run your evaluation set at increasing context lengths and find where accuracy starts to fall. That number — not the documented maximum — is your real budget, and it is the sort of thing only your own harness from 01-08 can tell you.

• Context window — the maximum total number of tokens, prompt plus completion, a model can process in one forward pass.
• Maximum sequence length — the same limit stated as a model-architecture property; also the term used for embedding models, where the number is usually much smaller.
• Supported length vs reliable length — what the documentation permits versus how much context your system can use and still be correct, measured on your own data.
• Context budget — an explicit allocation of the window across system prompt, examples, history, retrieval, question, and reserved completion.
• Reserved completion — the portion of the window set aside for output, enforced by max_tokens.
• Safety margin — headroom held back because token counts of unseen text are not perfectly predictable.
• Truncation (left / right) — dropping the oldest or newest tokens to make a request fit; the mechanism behind vanishing system prompts and vanishing questions.
• Eviction policy — the explicit, ordered rule for what gets dropped when the budget is exceeded.
• Context stuffing — sending large amounts of unretrieved material in the hope the model finds what matters; the alternative to retrieval, and usually the expensive one.
• Lost in the middle — the observed tendency for models to use material at the start and end of a long context more reliably than material in the middle.
• Representational, distributional, and resource limits — the three independent reasons a maximum length exists: no position vector, no training at that length, or no memory and time budget.

1. The context window is a total, in tokens, shared by everything — system prompt, examples, history, tool definitions, tool results, retrieved context, the question, and the generated answer.
2. Reserve the completion first. Then subtract fixed overheads and a safety margin. Only then allocate history and retrieval. Doing it in the other order is the most common cause of production truncation.
3. Exceeding it is a cliff, not a slope. Hard error, left truncation, right truncation, mid-generation stop, or silent framework eviction — and the silent one is the dangerous one.
4. Instrumentation is the single highest-leverage habit here. Per-component token counts and drop logs turn an invisible failure class into a dashboard.
5. Chunk size is granularity; the context window is a ceiling. A larger window buys more chunks, not bigger ones. 06-02.
6. The context window is a token limit; the KV cache is a memory allocation. The window bounds one request; the cache bounds how many requests you can serve at once. 12-05.
7. Being inside the window is not being used. Supported length exceeds reliable length, and only your own evaluation tells you by how much.
8. A bigger window is the most expensive answer available — money on every request, slower first tokens, and less concurrency. Trim the prompt, summarise the history, and rerank the retrieval first.
9. A larger window changes RAG's parameters and not its reasons. Cost, latency, precision, freshness, access control, and provenance all survive any window size.
10. Every context-window number is version-sensitive. Read it from current documentation for the exact model version and put it in configuration, not in your memory.
11. Nothing persists between requests. The window is not memory; conversational continuity is something your application constructs by resending context every time.

You can now say exactly how many tokens you have to work with, what competes for them, and what happens when you run out. That makes the next question the interesting one: given a finite and now explicitly-budgeted number of tokens, what is the highest-value thing to spend them on? Instructions, or examples? Two examples, or ten? Does showing the model what a good answer looks like beat describing it — and what is it actually doing when a few demonstrations in the prompt change its behaviour without changing a single weight?

That is in-context learning, and it is the first thing in this course you will spend your newly-budgeted context on deliberately.

Next: 05-01 covers zero-shot versus few-shot prompting and in-context learning — what a demonstration buys you, how many you need, and why it works at all when nothing about the model has changed.