How to count tokens: why tokens are not words

Token counting is the measurement of how many vocabulary entries a given string encodes to under a given tokenizer. It is a property of a (text, tokenizer, tokenizer version) triple, not of the text alone. The same paragraph sent to two different models will have two different token counts, and the same paragraph sent to the same model family across a major version change can have a third.

The operation itself is trivial: encode the string, take the length of the ID list. What makes token counting a 55-minute topic is not the operation but the four downstream budgets it feeds, the systematic ways the estimates fail, and the discipline of counting the right string — which is almost never the string a user typed.

Three framing facts to hold before any arithmetic:

Fact one: the count applies to the fully assembled request, not the user's text. A production request carries a system prompt, chat role and turn markers, few-shot examples, retrieved context, tool or function schemas, and often a JSON output template. Then the model's own output consumes tokens on top. Counting only the user message can understate a real request by an order of magnitude.

Fact two: input and output are counted separately and usually priced differently. Prefill (processing the input) and decode (generating the output) are different computational regimes — 12-05 and 12-10 develop why — and providers typically bill them at different rates. A cost model that adds them into one number is structurally wrong even if the token count is right.

Fact three: the ratio is content-dependent, and its worst cases are exactly the content engineers most often forget. Prose in English is the cheap case. Code, UUIDs, hashes, long numbers, minified JSON, and non-Latin scripts are all systematically more expensive per unit of meaning, for reasons 02-02 explained: they find fewer long vocabulary matches and fall back to shorter pieces.

L1 — The intuition: weigh it, don't estimate it

If you needed to know the weight of a parcel to buy postage, you would put it on a scale. You would not multiply its dimensions by a density you read in a blog post. Token counting is the same: there is a scale, it is free, it runs in milliseconds, and it is the tokenizer that ships with the model.

Estimates have exactly one legitimate use: sizing a decision before you have the text — capacity planning, a budget forecast, an architecture sketch. The moment you have real text, weigh it. And when you do estimate, carry the error bar out loud, because a "3× cheaper" architectural conclusion drawn from a ratio that is itself 2× uncertain is not a conclusion.

L2 — The mechanism: four counting procedures, in increasing fidelity

Procedure 1 — character-based estimate. Divide the character count by a chars-per-token ratio. For ordinary English prose a commonly quoted rule of thumb is about 4 characters per token, which is consistent with the constructed segmentations in 02-02 where common words came out at 4–5 characters per token. Treat this as a heuristic, not a measurement: it is not a figure sourced from official exam material, it varies by tokenizer, and it degrades badly on non-prose. Use it only for order-of-magnitude sizing.

Procedure 2 — word-based estimate. Multiply the word count by a tokens-per-word ratio. For English prose the same rule of thumb implies roughly 1.2–1.5 tokens per word — a five-letter average word plus a space is about six characters, which at four characters per token is about 1.5 tokens. Slightly more intuitive than characters, and slightly less stable, because average word length varies more across registers than character counts do.

Procedure 3 — corpus-calibrated ratio. Take a representative sample of your own content, tokenize it with the deployed tokenizer, and compute your own chars-per-token ratio. Now you have an estimate with a known error bar on the content you actually handle. This is the professional version of Procedures 1 and 2 and it takes about ten minutes. It is also the deliverable this course's week one asks for: a token-count audit of your own corpus.

Procedure 4 — exact count of the assembled request. Build the full request exactly as it will be sent — system prompt, chat template, few-shot examples, retrieved chunks, tool schemas, everything — and tokenize that. This is the only number you should ever put in a contract, a capacity plan, or a context-budget check at runtime. For a hosted API, use the provider's own tokenizer library; a different tokenizer will give you a count that disagrees with your bill, and the disagreement will not be small for code or multilingual text.

L3 — Why the ratio degrades, mechanically

The chars-per-token ratio measures how often the tokenizer finds long matches. Anything that reduces long-match frequency pushes the ratio toward 1.0, the character-level floor. Five mechanisms account for nearly all of it:

Rare surface forms. A word absent from the vocabulary decomposes into fragments (02-02). Proper nouns, product names, technical jargon, misspellings and inflected forms of rare stems all pay this.

Digits. Tokenizers handle numbers inconsistently — some reserve entries for one-, two- and three-digit groups, some split every digit individually. A 16-digit account number can be anywhere from a handful of tokens to sixteen. This is also, incidentally, a mechanistic contributor to LLMs' unreliability at arithmetic: the model does not necessarily see 1000000 as one object.

Punctuation and symbol density. Code, markup and minified JSON are dense in braces, quotes, colons, operators and indentation. Each of these tends to be its own short token, and there are few long matches to be had. Structured data is systematically token-expensive relative to the information it carries — which is why "just send the whole JSON" is a costly habit.

Whitespace structure. Indentation, tabs and repeated newlines consume tokens. Some tokenizers reserve entries for common indentation runs, which helps; some do not, in which case a deeply nested file pays per space.

Non-Latin scripts and UTF-8 expansion. Under a byte-level tokenizer the base alphabet is bytes, and a non-ASCII character occupies two to four UTF-8 bytes. If the vocabulary contains few multi-byte sequences for that script, a single character can cost more than one token. Combined with a vocabulary fitted mostly on English, the result is that the same meaning in some languages costs materially more tokens than in English. The exact factor depends entirely on the tokenizer and the language — do not quote a number you have not measured on your own corpus.

The unifying statement: the chars-per-token ratio is a compression ratio, and it degrades on exactly the content that is least like the tokenizer's training corpus.

Five units get conflated. This table is the exam-critical asset on this page.

Two of these conflations do real damage.

Context window vs chunk size. These are different kinds of quantity. The context window is a hard limit imposed by the model. Chunk size is a granularity decision you make: smaller chunks give sharper retrieval and more of them fit, larger chunks preserve more surrounding context per hit. Someone who thinks "my context window is 128k so my chunks can be 128k" has confused a ceiling with a design parameter and will build a retriever that returns one enormous mostly-irrelevant blob per query. 06-02 resolves this pairing properly.

Input tokens vs total tokens. The context window counts input plus generated output. If you fill the window with input, you have left the model no room to answer — and depending on the serving stack you will either get a truncated answer or an error. Always reserve output space explicitly. That reservation is a line item in the budget worked below.

Here is the full arithmetic for a support-assistant request. The token counts for individual strings below are constructed illustrations derived from the estimation ratios stated in §2, not measurements from a specific tokenizer — the point is the structure of the budget and the arithmetic, both of which are exact. In production you would replace every estimate with an exact count from your deployed tokenizer.

The request has six components.

Step 1 — total the input.

system prompt        400
chat template         30
few-shot examples    600
retrieved context  3,000
user question         30
                  ------
input subtotal     4,060 tokens

Step 2 — add the output reservation.

input subtotal     4,060
output reserve       500
                  ------
window occupancy   4,560 tokens

Step 3 — check the fit. Against a hypothetical 8,192-token context window, 4,560 leaves 3,632 tokens of headroom — about 44%. That is comfortable. Against a hypothetical 4,096-token window it does not fit at all, and the failure would land on the retrieved context, because that is the only component large enough to matter.

Step 4 — find the dominant term. Retrieved context is 3,000 of 4,060 input tokens: 74% of the input. This is the single most useful output of a token budget, and it is almost always the retrieval that dominates. It tells you where optimisation effort belongs. Halving the system prompt saves 200 tokens; dropping from six retrieved chunks to four saves 1,000. 07-07's reranking exists precisely so you can retrieve fewer chunks without losing recall.

Step 5 — cost the request. Prices change constantly and vary by model, so use placeholder rates and keep the arithmetic symbolic. Assume P_in per million input tokens and P_out per million output tokens.

input cost  = 4,060 / 1,000,000 × P_in  = 0.00406 × P_in
output cost =   500 / 1,000,000 × P_out = 0.00050 × P_out

With placeholder rates of P_in = 1.00 and P_out = 3.00 per million:

input cost  = 0.00406 × 1.00 = $0.00406
output cost = 0.00050 × 3.00 = $0.00150
                              ---------
per request                    $0.00556

Step 6 — scale it. At 50,000 requests per day:

50,000 × $0.00556          = $278.00 per day
$278.00 × 30               = $8,340 per month

Step 7 — read the sensitivity. Now redo step 4's optimisation with the monthly figure attached. Cutting retrieved chunks from six to four removes 1,000 input tokens per request:

saving = 1,000 / 1,000,000 × $1.00 × 50,000 × 30 = $1,500 per month

That is an 18% reduction in total spend from one retrieval-parameter change. Meanwhile trimming the system prompt by half saves 200 tokens per request, or $300 per month — real, but a fifth as much. This is what token counting buys you: it ranks your optimisations instead of leaving you to guess. 12-09 takes cost modelling further, but the mechanism is exactly this.

Step 8 — sanity-check the estimate itself. Every token figure above came from the 4-chars-per-token heuristic. If the true ratio for your content is 3.2 (plausible for text with code snippets and identifiers in it), every input figure rises by 25% and the monthly bill goes to roughly $10,000. That is the error bar, and it is why Procedure 3 in §2 — calibrating the ratio on your own corpus — is not optional for a number anyone will hold you to.

Token counts drive GPU memory as directly as they drive cost, and the arithmetic is worth doing once because it explains most of what 12-05 and 12-10 later say about serving.

During generation, a transformer caches the key and value vectors for every token it has already processed, so that producing token n+1 does not require recomputing tokens 1 through n. The cache size is exact given the model configuration:

KV bytes = 2 × layers × sequence_length × kv_heads × head_dim × bytes_per_value

The leading 2 is for keys and values. Take a hypothetical configuration — 32 layers, 8 key/value heads, head dimension 128, FP16 at 2 bytes per value — and the 4,560-token occupancy computed above:

2 × 32 × 4,560 × 8 × 128 × 2
= 2 × 32 = 64
64 × 4,560 = 291,840
291,840 × 8 = 2,334,720
2,334,720 × 128 = 298,844,160
298,844,160 × 2 = 597,688,320 bytes
≈ 570 MiB per concurrent request

Now the point. Serve 64 concurrent requests at that sequence length:

570 MiB × 64 ≈ 36,480 MiB ≈ 35.6 GiB of KV cache alone

Thirty-six gigabytes, before a single model weight is loaded. And it is linear in sequence length: the same 64 concurrent requests at 9,120 tokens each need about 71 GiB. Which means the retrieval decision in step 7 above — six chunks or four — is not only a cost decision, it is a concurrency decision. Dropping 1,000 tokens per request cuts KV cache by roughly 22% at fixed concurrency, or lets you serve roughly 28% more concurrent requests on the same GPU.

This is the deepest reason token counting matters and the reason the blueprint calls this lesson "the one that makes cost, context, chunking and GPU memory computable rather than gestural." One number — the token count of your assembled request — appears in the cost model, the context check, the chunk-size decision and the memory model. It is the same number every time.

Decision table — which counting method for which purpose:

And when not to count: if the content is short, English, prose, and the decision is reversible — a one-off exploratory prompt, say — the heuristic is fine and counting is ceremony. Reserve rigour for numbers that feed budgets, limits or capacity.

Token counting sits at the intersection of several NCA-GENL objectives rather than inside one. Objective 1.6 covers familiarity with Python natural-language package capabilities, which includes tokenizers. Objective 4.4 — identifying the system data, hardware, or software components required to meet user needs — is where context-window fit and GPU-memory sizing live, because you cannot specify hardware for an LLM service without a token budget. Objectives 2.1 and 2.3 on extracting insight from and conducting analysis over large datasets are where corpus auditing belongs. And objective 1.5 on machine-learning fundamentals underpins the whole thing.

Published candidate reports place tokenization in the highest-frequency tier of exam topics, and they specifically flag that questions are general-level rather than deep-technical: the reported winning posture is knowing at a high level what each thing is and when to use it, with detailed arithmetic described as overkill. That is field calibration, not official documentation — so treat it as guidance on where to spend study time, not as a promise about any individual question.

The practical reading for this lesson: you will almost certainly not be asked to compute a KV-cache size on the exam. The arithmetic above exists because it is the fastest route to genuinely understanding why the count matters, and because the associate role in the blueprint's own job-role frame includes assessing and resolving performance issues. What you will be asked is the conceptual layer:

• "Tokens and words have what relationship?" — None fixed. The ratio depends on the text and the tokenizer.
• "What is the most accurate way to determine how many tokens a prompt will consume?" — Encode it with the model's own tokenizer. Not a word count, not a character heuristic, not a different model's tokenizer.
• "A context window of N tokens accommodates what?" — Input plus generated output together, including system prompt and template overhead.
• "Which text will consume the most tokens for a given number of characters?" — Minified JSON, code, long digit strings, or non-Latin script, over English prose.
• "Why does the same request cost more when the user writes in a language other than English?" — The vocabulary was fitted mostly on English, so other scripts find fewer long matches and fall back to shorter pieces.
• "A team's cost forecast was 40% under actual. What is the most likely cause?" — They counted the user message only, omitting system prompt, few-shot examples, retrieved context and output tokens.

Distractor families:

For ordinary English prose, roughly 700 to 800 words — that follows from the 1.2–1.5 tokens-per-word rule of thumb in §2, and it is a heuristic rather than a sourced figure. Invert it and the more useful direction is: a 1,000-word document is around 1,250 to 1,500 tokens.

But treat the number as a bracket, not a value, and know when the bracket breaks. A 1,000-word document that is mostly prose lands near the low end. The same word count made of code, product SKUs, chemical names or file paths can easily land at double that, because each of those finds few long vocabulary matches. And a 1,000-"word" count in a language written without spaces between words is not a meaningful measurement at all, since the word boundaries were imposed by whatever tokenizer produced the word count.

The professional habit: quote the bracket, name the content class it applies to, and replace it with a measurement the moment real text exists.

Because the tokenizer's vocabulary was fitted on a corpus, and if that corpus is predominantly English, the vocabulary's entries are predominantly English words and English fragments (02-02). Text in another language finds fewer long matches, so segmentation falls back to shorter pieces — sometimes to individual characters, and under a byte-level tokenizer sometimes to individual UTF-8 bytes, of which a non-ASCII character has two to four.

Three consequences follow, and they are all real rather than theoretical:

1. Higher cost for the same meaning, since billing is per token.
2. Less usable context, since the same context window holds less content — a document that fits in English may not fit in translation.
3. More KV-cache memory per request, and therefore lower achievable concurrency on the same GPU, by the arithmetic in §5.

The mitigation is a larger, deliberately multilingual vocabulary, which is the direction tokenizers have moved across model generations. Do not quote a specific multiplier for a specific language: it depends entirely on the tokenizer and is version-sensitive. Measure it on your own corpus, which takes minutes, and then you have a defensible number.

They count the same against the context window — the window is a single budget covering input plus generated output — but they are usually priced differently, and output is typically the more expensive side.

The reason is computational rather than commercial. Processing the input is prefill: the whole prompt goes through the model in parallel, which uses the GPU efficiently. Generating the output is decode: one token at a time, each step reading the entire model's weights and the whole KV cache to produce a single token. Decode is memory-bandwidth-bound and far less efficient per token, which 12-05 and 12-10 develop properly. Providers pass that asymmetry through in their pricing.

The practical rule: model input and output as separate line items with separate rates, as §4 did. A workload that produces long answers from short prompts has a completely different cost profile from a RAG workload that feeds enormous context and asks for two sentences, even at identical total token counts.

Yes. Spaces, tabs and newlines are ordinary vocabulary entries and are billed like any other token. In many byte-level tokenizers a leading space is bundled into the following word's token, so ordinary prose spacing is nearly free — but that bundling is exactly why you must tokenize an assembled string rather than tokenizing fragments and summing (see the mistakes table).

Where whitespace stops being nearly free is structured text. Deeply indented code, pretty-printed JSON, and markdown tables all carry substantial whitespace that may or may not compress into reserved indentation tokens depending on the tokenizer. This is a genuine, measurable optimisation in RAG pipelines: normalising away redundant whitespace and collapsing pretty-printed JSON before embedding or before inserting into context can meaningfully reduce token counts, with no information loss. Measure before and after rather than assuming the size of the win.

For a first-pass sizing exercise, yes — with the error bar stated out loud. For anything a budget, a limit or a capacity plan depends on, no.

The heuristic fails in a specific and dangerous pattern: it fails most on the content that is most expensive. Prose, where the ratio is well-behaved, is also the cheap case. Code, identifiers, dense JSON and non-Latin scripts break the ratio in the direction of more tokens, and those are precisely the workloads where the bill is large. So the heuristic's errors are correlated with the stakes, which is the worst possible property for an estimate to have.

Use it as Procedure 1 in §2 and upgrade to Procedure 3 or 4 as soon as you have text. A corpus-calibrated ratio costs ten minutes and gives you a number that survives being challenged.

Four steps, in this order — reversing them is how teams end up shipping truncation bugs:

1. Look up the window for the exact model and version you will call. Context limits are configuration values that change between model releases, so read them from the model card or configuration rather than from memory.
2. Subtract your output reservation — the max_tokens you intend to allow. What remains is your input budget, not the window.
3. Count the full fixed overhead exactly: system prompt, chat template markers, few-shot examples, tool or function schemas, output-format instructions. This is a constant per deployment and worth computing once and caching.
4. Compare the variable content against what is left. If it does not fit, you have four levers, in rough order of preference: retrieve fewer chunks (07-07's reranking makes this cheap), chunk smaller (06-02), summarise or compress the context, or fall back to a larger-window model.

The check must run at request time, not at design time, because the variable content is variable. And it must run on the assembled string — the fixed overhead plus the actual retrieved chunks — because that is the only string whose count is the truth.

1. The only exact token count comes from the model's own tokenizer applied to the fully assembled request. Word counts and character heuristics are estimates, and their errors are largest on the most expensive content.
2. Count the whole request, not the user's message. System prompt, chat template, few-shot examples, tool schemas and retrieved context are all tokens. Omitting them is the single most common cause of a cost forecast landing far under actual.
3. The context window covers input plus output. Reserve output space explicitly and check the fit at request time against the assembled string.
4. One number feeds four budgets. Cost per request, context fit, chunk size and KV-cache memory are all denominated in the same token count — which is why getting it wrong is wrong four times over, silently and in the same direction.
5. Retrieved context usually dominates the input. In the worked budget it was 74% of input tokens, which means retrieval parameters, not prompt wordsmithing, are where optimisation pays. Token counting is what ranks those options instead of leaving you to guess.
6. KV cache is linear in sequence length, so token count is a concurrency decision as well as a cost decision. Fewer tokens per request means more concurrent requests on the same GPU.
7. Ratios degrade predictably: rare surface forms, digits, punctuation density, whitespace structure and non-Latin scripts all push chars-per-token toward the character-level floor. Keep separate calibrated ratios per content class.
8. Input and output are priced differently because prefill and decode are different computational regimes. Never collapse them into one number.
9. Treat the tokenizer as a versioned artefact. Pin it with the model and re-run your token audit after any upgrade, because a count is not a property of text alone.

Everything in this lesson took the tokenizer as a given: you encode with it, you count what comes out. But which tokenizer a model has is not arbitrary, and it is not derivable from first principles either — it is a fact about the model family that you either know or you do not. Three algorithms account for essentially all of the mainstream ones, they produce visibly different splits of the same word, and their marker conventions differ in ways that show up the first time you print a token list.

Next: 02-04 drills the algorithm identities cold, because the course blueprint flags this as a memorization item rather than something you can reason your way to: GPT uses BPE, BERT uses WordPiece, T5 uses SentencePiece. It explains what each algorithm's training procedure actually optimises, how to recognise each one's output on sight from its continuation markers, and why SentencePiece's refusal to assume whitespace word boundaries is what makes it the right choice for languages that do not use spaces.