Why text must be converted to numbers before an LLM can read it

Text-to-numbers conversion is the pipeline that turns a human-readable string into the numeric tensor a model's first layer expects. In a modern LLM it is exactly three steps: normalise and split the text into tokens, map each token to its integer ID via the tokenizer's vocabulary, and map each ID to a learned embedding vector via the embedding matrix. The output is a tensor whose shape you already know from 01-03 — batch × sequence × hidden size. Everything downstream, from the first attention head to the final softmax over the vocabulary, operates on that tensor.

The reason this conversion is unavoidable is structural, not historical. A neural network layer is a function of the form "multiply the input by a weight matrix, add a bias, apply a nonlinearity." Matrix multiplication is defined over numeric fields. There is no matrix multiplication over the string "tokenization". There is no gradient of a loss with respect to a character. Gradient descent, as 01-06 describes it, requires that the quantity you differentiate with respect to be a continuous number you can nudge. Characters are discrete symbols with no meaningful notion of "slightly more q." So the model's parameters cannot live in symbol space; the symbols have to be moved into number space first.

Three properties of that move matter for everything else in this module:

• It is lossy in one direction and exact in the other. Given IDs you can usually reconstruct the original string exactly, because the tokenizer stores the surface form of every vocabulary entry. But information about how you would have preferred it to be split is gone, and any normalisation applied before the split (lowercasing, accent stripping, whitespace collapsing) is genuinely destroyed.
• It has a fixed alphabet. The vocabulary is frozen when the tokenizer is trained, typically alongside the model. You cannot add a word to it after the fact without invalidating the embedding matrix, which has exactly one row per vocabulary entry.
• It is the unit of every cost you will ever pay. Context limits, API pricing, GPU memory for the KV cache, and chunk sizes for retrieval are all denominated in tokens, not words or characters. That is the subject of 02-03, and it is why this module sits so early in the course.

L1 — The intuition: a dictionary and a filing cabinet

Imagine a dictionary with a fixed number of numbered entries — say fifty thousand. Every entry holds one small piece of text: a whole common word like the, a word fragment like ization, a single character like q, or a raw byte. To encode a sentence, you walk along it, greedily matching the longest dictionary entry you can at each position, and write down the entry numbers. That list of numbers is your encoded text.

Now imagine a filing cabinet with the same fifty thousand drawers, where each drawer contains a list of, say, 4,096 numbers. To turn your entry numbers into something the model can compute with, you open the drawer matching each number and pull out its list. The dictionary gives you addresses; the cabinet gives you content. The cabinet's contents were learned during training; the dictionary's contents were fixed before training began.

That is the whole mechanism. The dictionary is the tokenizer vocabulary. The filing cabinet is the embedding matrix. The drawer number is the token ID.

L2 — The mechanism: normalisation, segmentation, lookup, embedding

The real pipeline has four stages, and each one is a place where behaviour can surprise you.

Normalisation. Before splitting, the tokenizer may apply Unicode normalisation (so that a precomposed é and an e followed by a combining accent become the same sequence), and depending on the tokenizer it may lowercase, strip accents, or collapse runs of whitespace. This is the only genuinely destructive step. A tokenizer that lowercases cannot distinguish Apple from apple, and no downstream layer can recover the difference. Modern LLM tokenizers do very little normalisation precisely because case and punctuation carry meaning the model should be allowed to use.

Pre-tokenization. Many tokenizers first split on a coarse boundary — usually whitespace and punctuation — so that the subword algorithm never merges across a word boundary. This is a design choice with consequences: a whitespace pre-tokenizer implicitly assumes the language uses spaces between words, which is why 02-04's treatment of SentencePiece matters for Chinese, Japanese, and Thai.

Segmentation and vocabulary lookup. The subword algorithm cuts each pre-token into vocabulary entries and emits their IDs. This is the step that makes 02-02 and 02-04 necessary: which pieces you get depends entirely on which algorithm trained the vocabulary. The output is a flat list of integers, plus — for encoder models — a set of special tokens marking the sequence boundaries.

Embedding lookup. The list of IDs indexes the embedding matrix. If the vocabulary has V entries and the model's hidden size is d, the embedding matrix has shape V × d. Looking up a sequence of L tokens produces an L × d matrix. Batch B such sequences and you have the B × L × d tensor from 01-03. This lookup is mathematically identical to multiplying a one-hot row vector by the embedding matrix, which is a useful thing to know because it explains why the operation is differentiable: gradients flow back into the rows that were selected, and those rows are trained like any other parameters.

L3 — Why the ID cannot be the number the model computes with

A natural first idea is to skip embeddings: assign each word an integer and feed the integers straight in. This fails for a reason worth being able to state cleanly, because it is the conceptual core of the whole module.

Integers carry an ordering and a metric. If cat is 41 and dog is 42 and helicopter is 43, then a model consuming those integers directly will compute that cat is closer to dog than to helicopter — which happens to be true — but also that dog is closer to helicopter than cat is, and that dog is roughly the average of cat and helicopter. Those are arbitrary artefacts of alphabetical or frequency ordering. The numbers impose a geometry the language does not have. This is the same defect as encoding a categorical feature as an ordinal one, which 01-02-era classical ML has known about for decades.

The fix is to make each symbol's numeric representation learned and multidimensional. A one-hot vector — a vector of length V that is 1 in position i and 0 everywhere else — carries no false ordering, because every pair of distinct one-hot vectors is exactly the same distance apart. But one-hot vectors are enormous and carry no similarity information either. The embedding matrix is the compromise: it maps each ID to a dense vector of a few thousand dimensions whose values are trained, so tokens that behave similarly in context end up with similar vectors. That is the subject of Module 3, and 03-01 picks it up directly.

So the ID is an address, chosen for bookkeeping convenience, and the embedding is the content, chosen by gradient descent. The exam likes this distinction because a well-written distractor will offer "the token ID encodes the token's meaning" as an option, and it is confidently wrong.

Four terms get used interchangeably in casual writing and are distinct on the exam. This table is the highest-value thing on this page.

Two of these collisions cause real damage. The first is "encoding" meaning both UTF-8 and token IDs — an engineer debugging a mojibake problem and an engineer debugging a token-count problem are using the same word for different layers. The second is treating embedding as part of tokenization, which leads people to believe that changing the tokenizer is a cosmetic change. It is not: change the vocabulary and every row of the embedding matrix becomes meaningless, which is exactly why 12-12's re-embedding problem exists.

Take the string "Tokenizers are not magic." and walk it through the pipeline. The specific split below is a constructed illustration chosen to show plausible subword behaviour — it is not a claim about any particular released tokenizer's exact output, because that varies by tokenizer and by version. What is exact is the arithmetic.

Step 1 — the raw string. 25 characters including the two spaces, the final period, and no trailing newline. Count them: Tokenizers (10) + (1) + are (3) + (1) + not (3) + (1) + magic (5) + . (1) = 25.

Step 2 — normalisation. Suppose the tokenizer applies Unicode NFC and nothing else. Nothing changes; the string was already normalised and we are not lowercasing.

Step 3 — pre-tokenization on whitespace, keeping the space attached to the following word. This is the convention most byte-level BPE tokenizers use, and it matters: the leading space becomes part of the token, so are and are are different vocabulary entries.

["Tokenizers", " are", " not", " magic", "."]

Step 4 — subword segmentation. are, not, magic and . are common enough to be single vocabulary entries. Tokenizers is rarer and gets split. A plausible split into common English fragments:

["Token", "izers", " are", " not", " magic", "."]

That is 6 tokens for 4 words and 25 characters. Already the headline fact of 02-03 is visible: tokens outnumber words here, 6 to 4.

Step 5 — vocabulary lookup. Each token becomes its ID. Using invented IDs to make the shape concrete:

["Token", "izers", " are", " not", " magic", "."]
   ↓        ↓        ↓       ↓        ↓        ↓
[ 30712,   6924,    553,    407,     11204,    13 ]

Note what these numbers do not mean. 13 for . is not smaller than 30712 for Token in any sense that matters. If this tokenizer assigned IDs partly by merge order during training, then low IDs correlate loosely with "learned early / very common," but no model layer reads that correlation. The IDs are addresses.

Step 6 — special tokens, if the model wants them. An encoder model in the BERT family wraps the sequence: [CLS] at the front, [SEP] at the end. That adds 2 tokens, taking us to 8. A decoder-only model in the GPT family typically adds none for a plain completion, though chat-formatted requests add role and turn markers that do cost tokens. This is why the same text can have two different token counts depending on which model you send it to — and why "how many tokens is this?" is only answerable once you name the tokenizer.

Step 7 — embedding lookup. With hidden size d = 4,096, each of the 6 IDs becomes a 4,096-dimensional float vector. The result is a 6 × 4,096 matrix. Batch four such prompts, pad them all to length 6, and you have a 4 × 6 × 4,096 tensor. At FP16 — 2 bytes per value — that activation tensor occupies:

4 × 6 × 4096 × 2 bytes = 196,608 bytes ≈ 0.19 MB

Small. But notice the shape of the dependency: the memory scales linearly in sequence length at this layer, and 04-01 will show it scaling quadratically inside attention. That is the entire reason token counting is worth 55 minutes in 02-03.

Step 8 — decoding back. The model's output is a distribution over the vocabulary; sampling from it gives an ID; the tokenizer's decoder concatenates surface forms to rebuild a string. Because the leading spaces were baked into tokens, concatenation alone reproduces the spacing. This round trip is why 01-01's next-token prediction is literally next-token prediction, not next-word prediction.

Not every text-to-numbers problem needs a transformer's embedding matrix. Choosing the wrong representation is a real failure mode in the associate role, and the exam asks about it in scenario form.

And the inverse — when not to convert text yourself: if you are calling a hosted model API, the provider tokenizes server-side, and re-implementing the split locally with a different tokenizer will give you a token count that disagrees with your bill. Use the provider's own tokenizer library, or accept that your estimate is an estimate.

The NCA-GENL blueprint places tokenization under Core Machine Learning and AI Knowledge, objective 1.6 — familiarity with the capabilities of Python natural language packages — and it reappears through the Software Development objective 4.4 on identifying the data and system components required to meet user needs, and Data Analysis objectives 2.1 and 2.3 on extracting insight from and conducting analysis over large datasets. Published candidate reports place tokenization in the highest-frequency tier of exam topics, alongside transformer architecture, prompt engineering, NVIDIA NIM, and text-generation parameters. That calibration comes from field reports rather than official documentation, so treat it as a study-time allocation signal rather than a guarantee — but it points in the same direction as the concept graph, where token counting alone has 61 transitive dependents in this course's dependency map.

The same field reports converge on a second finding that shapes how this is tested: questions sit at general level, not deep-technical level. Candidates report that detailed attention arithmetic and GPU spec sheets were overkill and did not appear. The winning posture is knowing at a high level what each thing is and when to use it. For this lesson, that means you should be able to state in one sentence why text must become numbers, and to place tokenization and embedding on the right side of a boundary — not to reimplement a tokenizer.

Question phrasings to expect at this depth:

• "Why must textual input be converted to a numerical representation before being processed by a neural network?" — The answer is about matrix operations and differentiability, not about efficiency or compression.
• "Which component maps a token to a dense vector representation?" — the embedding layer, not the tokenizer.
• "A team assigns each word in its vocabulary a sequential integer and feeds those integers directly to a feed-forward network. What is the primary problem?" — The integers impose an artificial ordinal relationship among unordered categories.
• "What is the output of a tokenizer's encode step?" — A sequence of integer token IDs, not vectors and not words.

Distractor families that recur:

Note the last one carefully, because it is the most sophisticated trap in this family. Byte-level and character-level models exist and are legitimate. They do not escape text-to-numbers conversion; they just choose a tiny vocabulary — 256 byte values, or a few thousand characters — and pay for it in sequence length.

No, and this is the single most important distinction in the lesson. A token ID is a single integer whose only job is to name a row. An embedding is a vector of hundreds or thousands of floating-point values that occupies that row and was shaped by training. The ID contains no information about meaning; the embedding contains almost nothing but information about meaning-in-context.

A useful test: if you shuffled every token ID in a vocabulary — reassigning the from 262 to 44,109 and so on — and permuted the rows of the embedding matrix by exactly the same shuffle, the model's behaviour would be bit-for-bit unchanged. That is what it means for the IDs to be arbitrary. Now try the same shuffle on the embedding rows without permuting the IDs, and the model produces garbage. The information lives in the matrix.

Only when a letter happens to be its own token. This is why LLMs are famously unreliable at character-level tasks like counting the letters in a word, reversing a string, or spotting an acrostic. The model receives strawberry as perhaps two or three subword chunks, not as ten letters, so questions about its letters ask the model to introspect about information it was never given in that form.

It is worth being precise about the claim, though: models are not incapable of character-level reasoning, because their training data contains plenty of text spelling words out, and the embeddings of subword pieces do carry orthographic information. They are simply not reliably good at it, and the mechanism explains why. If your application depends on character-level manipulation, do it in code outside the model.

Because every operation a neural network is made of — matrix multiplication, addition, and elementwise nonlinearities — is defined on numbers, and every parameter update requires a derivative with respect to a continuous quantity. Strings support neither. There is no partial derivative of cross-entropy loss with respect to the letter k.

You could imagine a model that manipulates symbols directly with discrete rules. That model exists, historically: it is the rule-based and statistical NLP tradition that gave us stemmers, stop-word lists and n-gram counts, which 02-05 and 02-06 cover because the exam still tests them. Those systems work on symbols. They also cannot be trained end-to-end by gradient descent, which is precisely the capability that made deep learning displace them.

UTF-8 is a character encoding: a rule for representing every Unicode code point as a sequence of one to four bytes. It sits upstream of tokenization and is not part of the model at all. By the time the tokenizer runs, the text is already a sequence of bytes or code points; the tokenizer's job is to group them into vocabulary units.

The two layers touch in one important place. Byte-level tokenizers — the "byte-level BPE" family — define their base alphabet as the 256 possible byte values rather than as characters. This has a valuable property: no input can ever be out-of-vocabulary, because any byte sequence whatsoever decomposes into bytes the tokenizer knows. The cost is that a single non-ASCII character, which occupies multiple UTF-8 bytes, may consume several tokens. That is one of the concrete reasons non-English text often costs more tokens per word than English, a point 02-03 quantifies with worked arithmetic.

On every request, for the input. Tokenizing a prompt is cheap — it is a string-matching pass over a trie or a merge table, measured in microseconds to milliseconds — so nothing about it constrains serving throughput. The embedding lookup is likewise a gather operation, trivially cheap compared with the attention and feed-forward layers behind it.

What is not cheap and does get cached is everything computed from those embeddings. That is the KV cache in 12-05, which stores per-token intermediate state so that generating token 500 does not require recomputing the first 499 from scratch. The distinction is worth holding: the conversion is cheap, the consequences of its output length are expensive.

1. A neural network computes only over numeric tensors, and its training requires derivatives with respect to continuous quantities. Text must therefore become numbers before any layer can touch it. This is a structural requirement, not an optimisation.
2. The conversion is two stages, not one. Tokenization produces integer IDs by deterministic lookup against a fixed vocabulary; embedding produces dense vectors by indexing a learned parameter matrix. Keep the boundary sharp.
3. A token ID is an address, not a magnitude. Shuffling IDs and embedding rows together changes nothing. Feeding IDs to a model as numeric features imposes a false ordering — the ordinal-ID fallacy, and a favourite distractor.
4. The vocabulary is frozen and matched to one model. The embedding matrix has exactly one row per vocabulary entry, so a tokenizer and its model are a matched pair. Mismatching them fails silently, which is worse than failing loudly.
5. UTF-8 is upstream and unrelated to token counts — except in byte-level tokenizers, where the byte alphabet guarantees nothing is ever out-of-vocabulary at the price of multiple tokens per non-ASCII character.
6. Tokens are the unit of every budget. Context limits, price per million tokens, KV-cache memory and RAG chunk sizes are all denominated in tokens. That is why this module precedes almost everything else in the course.
7. For the exam, hold the identity statement and the when-to-use rule. Field reports say questions are general-level: know what each component is and when it applies, and do not spend study time on tokenizer internals beyond the algorithm identities drilled in 02-04.

This lesson established that text becomes integers and why it must. It deliberately left the central question open: what exactly is a token, and who decides? A vocabulary of fifty thousand entries cannot hold every word of every language, yet a model must never choke on an unseen word — so the pieces cannot be whole words, and they cannot be single characters either without making sequences ruinously long.

Next: 02-02 resolves that tension. It defines the token as the unit of the vocabulary, explains why subword tokenization is the settled answer to the out-of-vocabulary problem, and works through the vocabulary-size trade-off — larger vocabularies mean shorter sequences but a bigger embedding matrix and rarer, worse-trained rows. That is the groundwork for 02-03's token arithmetic and 02-04's algorithm identities, and it is where the phrase "tokens are not words" stops being a slogan and becomes something you can compute.