Tokens, vocabulary, and subword tokenization explained

A token is the atomic unit a tokenizer emits and a model consumes: one entry from a finite, frozen vocabulary, identified by an integer ID. In a modern LLM's vocabulary you will find, side by side, all of these kinds of entry:

• Whole common words, usually with a leading space attached: the, and, model, because
• Word fragments: ization, ing, pre, un, ness, tokeni
• Single characters: q, z, ?, %, a newline
• Raw byte values, in byte-level tokenizers, so that arbitrary binary or unusual Unicode is always representable
• Special tokens that mark structure rather than content: [CLS], [SEP], [PAD], <|endoftext|>, chat role and turn markers

The critical property is that the vocabulary is fixed and finite. It is decided once, when the tokenizer is trained on a corpus, and then frozen for the life of the model, because the model's embedding matrix has exactly one row per entry. Fifty thousand entries means fifty thousand rows. Adding a word later would mean adding an untrained row, and the model would have no idea what to do with it.

The second critical property is that a token is not a linguistic unit. It is a statistical unit. Nothing guarantees that a token boundary falls where a morpheme boundary falls. A subword tokenizer might split tokenization as token + ization, which happens to be morphologically sensible, or as tok + en + iz + ation, which is not, or as Token + izers in the constructed example from 02-01. The pieces are whatever the training procedure found statistically efficient. Expecting them to be morphemes is a category error, and the exam occasionally probes it.

L1 — The intuition: spell rare words out of common pieces

You want a vocabulary small enough that the embedding matrix is affordable, and complete enough that nothing is ever unrepresentable. Those goals conflict, unless you notice one thing: language is Zipfian. A few hundred words account for most of running text; a long tail of rare words accounts for the rest, and that tail is effectively infinite because people invent names, misspell things and write code.

So spend the vocabulary where it pays. Give the frequent words their own entries — one token each, maximally efficient. For everything in the tail, don't try to store it; store the pieces it is made of. antidisestablishmentarianism does not need an entry; it can be spelled from anti, dis, establish, ment, arian, ism. Sagemaker does not need an entry; Sage + maker will do. A misspelling like tokeniztion does not need an entry either — it gets a slightly clumsier split and the model copes.

Common text gets short sequences. Rare text gets longer sequences. Nothing ever fails. That is the whole idea.

L2 — The mechanism: vocabulary training, then greedy segmentation

Subword tokenization has two phases that operate at completely different times, and keeping them apart resolves most confusion about it.

Phase 1 — vocabulary training (offline, once). You take a large text corpus, start from a base alphabet of characters or bytes, and iteratively grow the vocabulary until it reaches a target size. The specific growth rule is what distinguishes the algorithms in 02-04: byte-pair encoding repeatedly merges the most frequent adjacent pair; WordPiece merges the pair that most improves the training corpus's likelihood; the unigram approach starts from a large candidate set and prunes. What they share is the output: an ordered merge table or a scored vocabulary, plus the final list of entries.

This phase involves statistics over a corpus, so people call the vocabulary "learned." It is, in a loose sense — but not by gradient descent, and not with any parameters. It is a fitted frequency structure, and once fitted it never changes.

Phase 2 — segmentation (online, every request). Given a string, the tokenizer applies its stored table to produce the split. This is deterministic table lookup and string matching: fast, gradient-free, and identical every time for the same input and the same tokenizer version. There is no model, no probability sampling, no learning.

The practical consequence is that the tokenizer is data, not code you can retrain casually. Change the vocabulary and every embedding row's meaning changes. That is the mechanism behind the tokenizer/model mismatch failure named in 02-01, and behind the re-embedding migration problem in 12-12.

L3 — The vocabulary-size trade-off, stated precisely

The one genuinely quantitative decision in tokenizer design is vocabulary size V. It trades three things against each other.

Sequence length. A larger vocabulary contains more whole words and longer fragments, so the same text encodes to fewer tokens. Fewer tokens is unambiguously good: it means more text fits the context window, fewer tokens billed per request, less KV-cache memory, and — because attention cost grows with the square of sequence length per 04-01 — disproportionately less compute.

Embedding and output-layer parameters. The embedding matrix is V × d, and in most architectures the output projection over the vocabulary is another V × d (sometimes tied to the embedding, sometimes separate). So parameter count in these two layers is linear in V. With hidden size d = 4,096:

V =  32,000 → 32,000 × 4,096 = 131,072,000 params ≈ 131 M per matrix
V =  50,000 → 50,000 × 4,096 = 204,800,000 params ≈ 205 M per matrix
V = 128,000 → 128,000 × 4,096 = 524,288,000 params ≈ 524 M per matrix
V = 256,000 → 256,000 × 4,096 = 1,048,576,000 params ≈ 1.05 B per matrix

At V = 256,000 with an untied output layer, the two vocabulary-facing matrices together are over two billion parameters — potentially a large share of a mid-size model's budget spent on lookup tables rather than on the layers that do the reasoning. This arithmetic is exact given d and V; the specific V and d values a released model uses are configuration choices that vary by model and version.

Per-token training signal. Every vocabulary row must be trained by seeing its token in context often enough. Split a fixed corpus across a larger vocabulary and the rare rows are each seen fewer times, so they are worse trained. Very large vocabularies therefore accumulate a tail of poorly-estimated embeddings — entries that exist but whose vectors are close to noise. These are the "glitch tokens" occasionally reported for production models: entries that survived vocabulary training but were starved of training signal in the language model itself.

The optimum is therefore an interior one, and it depends on the language mix. A vocabulary tuned on English gives English users short sequences and multilingual users long ones, which is exactly the fairness-and-cost issue 02-03 quantifies. Larger multilingual vocabularies are the standard mitigation, and the direction of travel across model generations has been toward larger vocabularies as models grew large enough that a few hundred million extra embedding parameters stopped being decisive. Treat specific numbers as version-sensitive: read them off the tokenizer configuration of the exact checkpoint you deploy.

This is the comparison that justifies the entire module. Learn it as a table.

Read the two failure modes as a pair, because they fail in opposite directions:

Word-level fails on coverage. No finite word list covers a living language. Every proper noun, product name, hashtag, misspelling, chemical name and variable name you have never seen becomes <UNK>. And <UNK> is catastrophic in a way that is easy to underrate: it does not merely lose the word, it maps every unknown word to the same vector. Zolgensma, Xiaohongshu and qwertyuiop become indistinguishable. A model cannot reason about, copy, or generate a word it receives as <UNK> — and it cannot generate one at all, because there is no vocabulary entry to emit. This is why word-level tokenization is not merely inefficient for generative models but disqualifying.

Character-level fails on length. Coverage is perfect and the vocabulary is tiny. But English prose runs at roughly four to five characters per subword token, so a character-level model needs four to five times the sequence positions for the same content. Since self-attention cost scales with the square of the sequence length (04-01), a 4× longer sequence implies roughly a 16× attention cost, plus 4× the KV-cache memory and 4× the positions consumed inside a fixed context window. It also forces the model to spend early layers rediscovering that t,h,e is a word — work the tokenizer could have done for free.

Subword tokenization takes coverage from the character-level approach and most of the length efficiency from the word-level approach. That is the trade, and it is why the field settled.

Here is the mechanism made concrete. This is a constructed illustration with a hand-built vocabulary; it demonstrates how greedy longest-match segmentation behaves, and is not a claim about any released tokenizer's output.

Suppose a tiny vocabulary containing, among other entries:

whole words:      the, token, ize, run, ing, ed, s, pre, un, able
fragments:        ##iz, ##ation, ##er, ##ly, ##ness
characters:       a b c d e f g h i j k l m n o p q r s t u v w x y z

Now segment four inputs, always taking the longest match available at the current position.

Input 1: token. Longest match at position 0 is the whole entry token. Done in 1 token.

Input 2: tokenize. Longest match at 0 is token. Remaining is ize, which is in the vocabulary. 2 tokens: token + ize. Note the model can see that this shares a piece with input 1 — the relationship between token and tokenize is partly visible in the representation, which is a real benefit over word-level encoding where they would be unrelated IDs.

Input 3: tokenization. Longest match at 0 is token. Then iz — available as the continuation fragment ##iz. Then ation — available as ##ation. 3 tokens: token + ##iz + ##ation. Notice the split is not the morphologically clean token + ization; it is whatever the vocabulary permits. Statistical unit, not linguistic unit.

Input 4: qzzq. No multi-character entry matches. Fall through to single characters: q + z + z + q. 4 tokens, no failure, no <UNK>. This is the guarantee character-level fallback buys you.

Now count the cost across all four inputs:

token          →  5 chars →  1 token  →  5.0 chars/token
tokenize       →  8 chars →  2 tokens →  4.0 chars/token
tokenization   → 12 chars →  3 tokens →  4.0 chars/token
qzzq           →  4 chars →  4 tokens →  1.0 chars/token

Totals:          29 chars → 10 tokens →  2.9 chars/token

Two things fall out of that arithmetic and both matter for 02-03. First, the chars-per-token ratio is a property of the text, not a constant — it ranged from 5.0 to 1.0 across four short strings. Second, unusual text is systematically more expensive, because it falls back to shorter pieces. Random identifiers, base64 blobs, dense JSON, minified code and non-Latin scripts all sit at the expensive end.

Compare the three strategies on a single input to see the failure modes side by side. Take the invented product name Nemotronix, which no tokenizer trained before its invention could contain.

Word-level vocabulary (say 200,000 English words). Nemotronix is not present. Output: <UNK> — 1 token, all content destroyed. The model receives the same input it would receive for Xylophonium or asdfgh. It cannot answer a question about the name, cannot copy it into its output, and cannot generate it, because there is no ID to emit. If your prompt was "summarise this document about Nemotronix," the model is now summarising a document about a hole.

Character-level vocabulary (~100 entries). Output: N e m o t r o n i x — 10 tokens, all content preserved. The model can copy it exactly, because it can emit each character. But it burned 10 positions on one word, and it must learn from scratch that this run of characters is a single referent.

Subword vocabulary (~50,000 entries). Plausible output: Nem + otron + ix — 3 tokens, all content preserved. Positions used are close to the word-level cost, content survival is as good as character-level, and the pieces are meaningful enough that the model may generalise from otron appearing in other technology names.

Decision table — when each still makes sense:

Tokenization sits under NCA-GENL objective 1.6 (familiarity with the capabilities of Python natural language packages) and objective 1.10 (using Python packages to implement traditional ML analyses), and it is load-bearing for the Software Development objective 4.4 on identifying the components required to meet a user need. Published candidate reports place it in the highest-frequency tier of exam topics — that is field calibration rather than official documentation, so use it to allocate study time rather than as a guarantee, but it agrees with the structural argument: token counting alone has 61 transitive dependents in this course's concept graph, meaning almost everything about cost, context, retrieval and serving memory is downstream of it.

The same field reports say questions are general-level: know what each thing is and when to use it, rather than being able to derive it. For this lesson that means you must be able to state, cold:

• what a token is (one entry in a fixed vocabulary — word, fragment, character or byte)
• why subword tokenization exists (it eliminates out-of-vocabulary failure without the sequence-length blowup of character-level encoding)
• the direction of the vocabulary-size trade-off (bigger vocabulary → shorter sequences but more embedding parameters and less training signal per row)
• that tokens are not words, and that the ratio varies with the text

Question phrasings that recur:

• "What is the primary advantage of subword tokenization over word-level tokenization?" — Handling out-of-vocabulary words without an <UNK> collapse. Beware the plausible-but-secondary "it reduces sequence length," which is the advantage over character-level.
• "Increasing a tokenizer's vocabulary size has which effect?" — Shorter token sequences and a larger embedding matrix. Both halves must be in the answer.
• "A model encounters a proper noun absent from its training data. What happens under subword tokenization?" — It is split into known subword units and processed normally.
• "Which of these is a token?" — All of "a whole word," "part of a word," "a punctuation mark," "a byte" can be tokens; the trap is an option asserting tokens are always words.
• "Why is <UNK> a problem for a generative model specifically?" — Because it is not invertible: the model has no vocabulary entry with which to produce the original word.

Distractor families:

No. Tokens are vocabulary entries, and a vocabulary entry may be shorter than a word, exactly a word, or — with a leading space attached — a word plus its preceding whitespace. Common short English words are usually one token each, which is why the two look interchangeable in casual examples. Rare words, long words, proper nouns, code identifiers, numbers with many digits, and text in non-Latin scripts routinely take several tokens each.

The practical framing: tokens are the unit the model, the context limit and the invoice all use; words are the unit humans use. Any time a number matters — a cost estimate, a chunk size, a context budget — convert to tokens with the actual tokenizer. 02-03 is entirely about doing that correctly.

Under subword tokenization, nothing bad. The word is spelled out from the pieces the vocabulary does contain, falling all the way back to single characters or bytes if necessary. The content survives; the only cost is extra tokens.

Under word-level tokenization, the word is replaced by a single <UNK> token, and this is genuinely destructive: every distinct unknown word collapses to the same representation, and the model has no way to reproduce the original. For an encoder doing classification you might tolerate it. For a generative model it is disqualifying, which is the historical reason the field moved to subwords.

Byte-level tokenizers make the guarantee absolute. Because their base alphabet is the 256 byte values, any byte sequence whatsoever — including malformed Unicode and binary data — is representable. There is no such thing as out-of-vocabulary input.

Because it costs three things and buys one. It buys shorter sequences, which is real and valuable: fewer tokens billed, more content per context window, and less than linear savings in attention compute. Against that:

1. Embedding parameters grow linearly in V. With hidden size 4,096, every extra 10,000 vocabulary entries adds about 41 million parameters to the embedding matrix, and roughly the same again to an untied output projection.
2. The output softmax gets wider. Computing a distribution over the vocabulary at every generation step scales with V, so a very large vocabulary makes the final layer a real cost.
3. Training signal per row thins out. A fixed corpus split across more entries means each rare entry appears fewer times, so its embedding is estimated from less evidence. Very large vocabularies accumulate a tail of poorly-trained rows.

The right V is an interior optimum that depends on corpus size and language mix, and it moves as models grow. Do not memorise a number; memorise the direction of each effect, which is what an exam question at this level asks for.

Not without consequences. The vocabulary and the embedding matrix are locked together by construction: entry i of the vocabulary corresponds to row i of the matrix. Libraries do let you append tokens, and the matrix is resized to match — but the new rows start untrained, so the model has no learned representation for them. That is model surgery, requiring at minimum some fine-tuning to make the new rows useful.

Replacing the vocabulary wholesale is worse: every existing row's meaning is invalidated, so the model must be retrained. This is the same lock-in that appears in retrieval, where changing the embedding model means re-embedding the entire corpus (12-12). Treat tokenizer choice as a decision with the same permanence as architecture choice.

No, and this is one of the more consequential practical facts in the module. A vocabulary fitted mostly on English spends its entries on English words and English fragments. Text in another language finds fewer long matches and falls back to shorter pieces, so the same meaning costs more tokens.

The effect compounds for scripts far from the training distribution. Languages written without spaces between words cannot benefit from whitespace pre-tokenization. Non-Latin scripts encode to multiple UTF-8 bytes per character, so a byte-level tokenizer may spend several tokens on a single character.

The consequences are concrete rather than abstract: for the same content, a non-English user gets less usable context, pays more per request, and consumes more KV-cache memory. Larger multilingual vocabularies are the standard mitigation. The precise ratios depend on the tokenizer and the language, so measure them for your own corpus rather than quoting a figure — 02-03 gives the procedure.

1. A token is one entry in a fixed, frozen vocabulary — a word, a fragment, a character, or a byte. Not a linguistic unit; a statistical one.
2. Subword tokenization exists to eliminate out-of-vocabulary failure without paying character-level sequence lengths. That is its primary purpose, and the most commonly mis-stated fact about it.
3. Word-level fails on coverage; character-level fails on length. <UNK> destroys content irrecoverably and cannot be generated; character sequences run four to five times longer for English prose, and attention cost grows with the square of length.
4. Vocabulary size is a three-way trade-off: larger V gives shorter sequences but linearly more embedding parameters, a wider output softmax, and thinner training signal per row. Know the direction of every arm.
5. Splits are not morphemes. Expect statistically efficient pieces, not linguistically clean ones, and do not try to "fix" a tokenizer that produces an ugly split.
6. The vocabulary and the embedding matrix are locked together. Adding entries leaves untrained rows; replacing the vocabulary invalidates the model. Tokenizer choice has architecture-level permanence.
7. Tokenization efficiency is not uniform across languages or content types. English prose is cheap; non-Latin scripts, code, identifiers and dense JSON are expensive. Measure your own corpus.

You now know what a token is and why the vocabulary is built from fragments. What you cannot yet do is put a number on it — and almost every practical decision in an LLM project turns on a number. How much of a 100-page PDF fits in one request? What does a million user turns cost per month? How large should a RAG chunk be so that ten retrieved chunks plus the system prompt plus the answer still fit? How much GPU memory does the KV cache need at your target concurrency?

Next: 02-03 is the deepest lesson in this module, and it turns all four of those questions into arithmetic you can actually perform. It establishes the working estimation ratios and their error bars, shows why the ratio differs for code and for non-English text, walks through a full cost and context budget end to end, and draws the line between a defensible estimate and a measurement. After it, "that probably fits" stops being an answer you accept from yourself.