Subword Tokenization: BPE vs. WordPiece Merge Rules Compared

Identity statement: a subword tokenizer's merge rule is the specific criterion its training procedure uses, at each iteration, to choose which pair of adjacent symbols in the current vocabulary gets combined into one new, larger symbol.

Subword vocabulary training starts from a base alphabet — individual characters or bytes — and grows outward by repeatedly combining pairs of existing symbols into new, longer symbols, stopping once the vocabulary reaches a target size. That much is shared by every subword algorithm, and it is the reason [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states plainly that both BPE and WordPiece are subword methods, neither purely character-level nor purely word-level: at any point mid-training, the vocabulary contains a mix of single characters, short fragments, and — as training proceeds — increasingly long whole-word entries, and the final tokenizer will fall back all the way to individual characters for any input rare enough that nothing longer matches.

What is not shared, and what this lesson is entirely about, is which pair gets merged next at each of those iterations. That single choice — the merge rule — is the one part of the training procedure that actually distinguishes one subword algorithm from another, because everything else (start from characters, grow by merging, stop at a target size) is common scaffolding around it.

L1 — Intuition

BPE's merge rule is the simplest one to state and the easiest one to compute by hand: at every iteration, count how many times each adjacent pair of symbols occurs across the entire training corpus, and merge whichever pair occurred the most.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states this directly: BPE "iteratively merges the most frequent adjacent symbol pair into a new subword." Concretely, training starts with every word in the corpus represented as a sequence of individual characters (often with a boundary marker so word starts and ends stay distinguishable). The algorithm then scans the entire corpus, tallies every adjacent symbol-pair occurrence, identifies the single most frequent pair, and merges every occurrence of that pair into one new symbol. That new symbol is added to the vocabulary, the corpus representation is updated to reflect the merge everywhere it occurred, and the whole counting-and-merging step repeats from scratch on the updated representation. Each iteration produces exactly one new vocabulary entry, so the number of merge iterations run is also the number of vocabulary entries added beyond the base alphabet.

The computation at each step is a pure counting exercise — no probability model, no likelihood function, nothing beyond tallying occurrences and picking the maximum. That simplicity is precisely why BPE is the algorithm most commonly used to teach subword tokenization's mechanics: the entire training procedure reduces to "count pairs, merge the winner, repeat," with no additional machinery to explain.

L3 — The exam-relevant edge case: frequency shifts between iterations

The detail that separates a shallow grasp of BPE from a correct one is that pair frequencies are recomputed fresh after every merge, not calculated once up front and then applied in a fixed order. Merging one pair can change which pairs are now adjacent to each other, and it removes the merged symbols' individual counts from future tallies in favor of the new combined symbol's count — so the second-most-frequent pair at iteration 1 is not necessarily the pair merged at iteration 2. A learner who assumes BPE just merges the top-N most frequent pairs from a single initial count, computed once, will get merge order wrong on any corpus complex enough for the ranking to shift between rounds — and the worked example in section 5 walks through exactly this shift happening.

L1 — Intuition

WordPiece's merge rule asks a different question than BPE's. Instead of "which pair occurs most often," it asks "which pair, if merged, most improves how well the resulting vocabulary explains the training corpus" — a criterion grounded in a likelihood computation rather than a raw count.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states WordPiece's rule as merging "the pair that most increases training-corpus likelihood." The underlying idea is that the vocabulary at any point in training defines a way of scoring how probable the training corpus is, given that vocabulary's set of tokens and their observed frequencies as a unigram-style model over the corpus. For each candidate pair of adjacent symbols, WordPiece's training procedure evaluates how much the corpus's overall likelihood score would increase if that specific pair were merged into a new token — and it selects the pair with the largest likelihood gain, not the pair with the largest raw occurrence count.

Those two criteria are not the same computation, even though they often correlate on real corpora, which is exactly why the exam's standing distractor is tempting rather than obviously wrong. A pair can occur extremely often in raw counts while contributing comparatively little to the corpus's likelihood score under the vocabulary's current model, if each of its two component symbols already independently explains the data about as well on its own. Conversely, a somewhat less frequent pair can produce a larger likelihood gain if merging it captures a genuinely more informative unit — one whose two halves, taken separately, poorly predicted the contexts they actually appear in, so combining them into one token meaningfully sharpens the model's fit. WordPiece is therefore not "BPE with a different counting method bolted on"; it is optimizing a materially different quantity at every step, one that accounts for how well the resulting vocabulary models the corpus rather than how often the input pair happened to co-occur.

L3 — The exam-relevant edge case: why the two criteria often agree anyway, and when they do not

The reason this distinction is genuinely testable, rather than a technicality nobody would ever ask about, is that BPE's frequency criterion and WordPiece's likelihood criterion frequently select very similar-looking merges on ordinary text, which is exactly what makes the two algorithms easy to conflate in practice. Both tend to merge common short fragments and frequent whole words early, and both tend to leave rare, unusual character sequences fragmented late into training — so a tokenizer trained by either rule, run on the same English-language corpus, will often look broadly similar at a glance. The place the two criteria diverge more visibly is on pairs whose components are each already well-represented individually: raw frequency can still favor merging such a pair simply because it co-occurs often, while a likelihood-based criterion is more selective about paying a new vocabulary entry's "cost" (one more token consuming vocabulary budget) unless the merge earns a real gain in how well the corpus is explained. Knowing that the two criteria are computationally distinct — not merely two names for the same idea — is what the exam is actually testing when it pairs BPE and WordPiece together, because a description that swaps "most frequent" for "most increases likelihood" between the two algorithms is describing a real algorithm's rule, attached to the wrong algorithm.

Both algorithms start from individual characters and both stop well short of encoding every possible word as its own single entry, which is exactly the property [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) calls out directly: "Both are subword methods — neither is purely character-level nor word-level." This matters as more than a definitional footnote, because a description that calls BPE "character-based" or WordPiece "a word-level tokenizer" is not a minor imprecision — it misstates the entire reason either algorithm exists. A purely character-level scheme never needs merges at all, since single characters already cover every possible input; a purely word-level scheme has no fallback once a word is missing from its fixed list. BPE and WordPiece each occupy the middle ground specifically by merging characters into progressively longer fragments up to, but not exceeding, a target vocabulary size — common enough words end up as single tokens, and rare or novel words decompose into the smaller fragments the training process already learned, with single characters as the final fallback for anything the vocabulary has genuinely never seen.

This is a constructed illustration using an invented four-word toy corpus, small enough to compute by hand; it demonstrates the mechanics of frequency-shift between iterations and is not a claim about any released tokenizer's actual training run.

Take the toy corpus low low low lower lowest newest (six word occurrences total), and represent every word as a sequence of characters plus an end-of-word marker _:

Starting representation (character sequences, with counts):
l o w _        (3 occurrences: "low")
l o w e r _    (1 occurrence: "lower")
l o w e s t _  (1 occurrence: "lowest")
n e w e s t _  (1 occurrence: "newest")

Round 1 — count every adjacent pair across all six word occurrences:

l-o:  5   (appears in low, low, low, lower, lowest -> 5 occurrences)
o-w:  5   (same five words)
w-_:  3   (only "low" ends immediately after w)
w-e:  2   (lower, lowest)
e-r:  1   (lower)
r-_:  1   (lower)
e-s:  2   (lowest, newest)
s-t:  2   (lowest, newest)
t-_:  2   (lowest, newest)
n-e:  1   (newest)

l-o and o-w are tied at 5, the highest count. Take l-o as the merge (a real implementation breaks ties by a fixed rule; either choice is defensible for this illustration). New vocabulary entry: lo.

Round 2 — recompute pair counts on the updated representation:

lo w _         (3 occurrences)
lo w e r _     (1 occurrence)
lo w e s t _   (1 occurrence)
n e w e s t _  (1 occurrence, unchanged — no "lo" here)

lo-w:  5   (now the top pair, since l-o and o-w both collapsed into this)
w-_:   3
w-e:   2
e-s:   2
s-t:   2
t-_:   2
...

lo-w is now the clear single top pair at count 5 — note that this pair did not exist as a countable pair in round 1's tally at all; it only became countable once round 1's merge created the lo symbol. Merge lo-w into low.

Round 3 — recompute again:

low _          (3 occurrences)
low e r _      (1 occurrence)
low e s t _    (1 occurrence)
n e w e s t _  (1 occurrence, still unchanged)

low-_:  3   (only "low" alone ends right after "low")
low-e:  2   (lower, lowest)
e-s:    2
s-t:    2
t-_:    2
e-r:    1
r-_:    1
n-e:    1

low-_ and low-e are the leading candidates now, at 3 and 2 respectively — low-_ wins this round, adding a token that represents "the whole word low, standing alone." Notice what happened over three rounds: l-o was the winner once, and by round 2 that exact pair no longer existed anywhere in the tally, replaced by an entirely new pair (lo-w) that round 1's counts never mentioned. A learner who tried to precompute a fixed ranking of pairs from the original round-1 counts and simply "take the top three" would get round 2 and round 3's actual merges completely wrong, because those merges depend on symbols that did not exist until the previous round created them.

Consider a second constructed scenario — hypothetical numbers chosen to make the divergence point concrete, not measured from a real tokenizer training run. Suppose a corpus contains two candidate adjacent pairs at the same point in training:

Pair A ("th"):  raw frequency = 8,000 occurrences
  — but "t" alone and "h" alone are each already common, well-modeled symbols
  individually; merging them captures a real but modest likelihood gain,
  because the vocabulary could already predict "t" and "h" reasonably well
  as separate symbols in most contexts.

Pair B ("qu"):  raw frequency = 1,200 occurrences
  — but "q" is almost always followed by "u" in this corpus's language, and
  neither symbol alone predicts its context nearly as well as the combined
  unit does; merging them captures a comparatively larger likelihood gain
  per occurrence, because the two symbols were poorly explaining their
  contexts separately.

Under BPE's frequency-only rule, Pair A wins outright — 8,000 beats 1,200, and raw count is the entire criterion, so the comparison never reaches the question of how well either symbol was already being modeled. Under WordPiece's likelihood-based rule, the outcome is not automatically the same: if Pair B's per-occurrence likelihood gain is large enough to outweigh its lower raw frequency in the aggregate improvement WordPiece's criterion computes, WordPiece can select Pair B at this step even though it is the less frequent pair by a wide margin — precisely because WordPiece is not scoring frequency at all, it is scoring how much the merge improves the vocabulary's fit to the corpus. This is a constructed, illustrative divergence, not a claim about which specific pairs a real English-language BPE or WordPiece tokenizer would select — but it is exactly the shape of disagreement that makes the two algorithms genuinely different procedures rather than two names for one idea.

Two axes drive nearly every distractor built against this table. The first is the criterion itself — frequency versus likelihood — and a description that attributes "most frequent" to WordPiece or "most increases likelihood" to BPE has simply swapped the two algorithms' defining property. The second is the character-or-word-level mislabeling from section 4, which applies identically to both algorithms and is testable independently of which specific merge rule a question is asking about.

⭐ THE EARNED INSIGHT: > BPE and WordPiece are not "the same idea with two different names" — they optimize two genuinely different quantities at every training iteration, frequency versus likelihood-improvement, and the fact that both criteria often select similar-looking merges on ordinary text is exactly what makes the swap tempting rather than obviously wrong. The two rules will occasionally diverge — a frequent pair whose components are already well-modeled loses to a rarer pair whose merge earns a real likelihood gain — and knowing that divergence is possible, not just knowing which model family uses which name, is what separates recognizing the rule from merely memorizing a lookup table.

A subword tokenizer's vocabulary is fit to whatever corpus trained it, and a corpus dominated by general text will build a vocabulary that fragments unusual domain terms — a medical or legal term that never appears often enough in a general corpus gets split into several smaller, individually meaningless pieces, the same fallback behavior any rare word gets. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names the fix directly: a custom or domain-specific tokenizer, trained on a corpus that actually contains the domain's jargon at meaningful frequency, preserves that vocabulary as coherent tokens instead of shattering it into fragments a general tokenizer would produce.

The trap named explicitly in the source material is treating "remove the jargon to standardize the text" as the correct response to this problem — it is not. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states that removing jargon is "usually wrong" for a domain-specific model, because the jargon is frequently the exact content a specialized model exists to understand and generate correctly; stripping it out to make the text look more like general prose optimizes for the wrong objective. The correct move, when a domain's vocabulary matters, is training or extending a tokenizer so that jargon survives as intact, meaningful tokens — not scrubbing the jargon out of the corpus so a general-purpose tokenizer no longer has anything unusual to fragment.

Multilingual corpora raise a related but distinct point: language-aware tokenization — training on a corpus that genuinely represents each language's own vocabulary and structure, rather than one dominated by a single language — preserves the nuances of every represented language rather than treating one language's patterns as the default and every other language as an exception the vocabulary happens to handle poorly.

It is worth being precise about what the merge-rule choice actually controls, because it is easy to overstate. The merge rule determines which specific vocabulary entries end up in the tokenizer and in what order they were added during training — it does not determine the segmentation algorithm's behavior at inference time in a way that fundamentally changes what kind of tokenizer the result is. Once training is finished and the vocabulary is frozen, both a BPE-trained and a WordPiece-trained tokenizer segment new input by matching against their respective learned vocabularies — greedy longest-match segmentation is a common inference-time strategy for either family, applying the already-fixed vocabulary to new text with no further counting or scoring involved. The interesting difference between the two algorithms lives entirely in the training phase, in which merges got selected and therefore which specific fragments exist in the final vocabulary; by the time a model is actually tokenizing user input in production, the difference has already been baked into the vocabulary itself; the segmentation mechanics applied at that point look broadly similar regardless of which merge rule built the vocabulary being applied.

This distinction matters for a specific class of exam item: a question describing tokenizer behavior at inference time — how an already-trained tokenizer splits a novel input — is not really testing the BPE-versus-WordPiece distinction at all, even if it names one of the two algorithms, because the segmentation mechanics at that stage are not where the two algorithms differ. The BPE-versus-WordPiece distinction is squarely a training-time question: which pair got merged, and why, at some specific point while the vocabulary was being built.

Tokenization sits inside objective 3.3 of the Data Preparation domain, and the source material's own scope note is explicit that professional-level questions here expect reasoning about tradeoffs and knowing which tokenizer serves which purpose, not merely recognition-depth recall. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names the BPE-versus-WordPiece merge-rule swap as a common exam trap directly, alongside the character-level/word-level mislabeling and the jargon-removal misconception from section 8.

Expect the material in a small number of recurring shapes: a direct identification item asking which merge rule belongs to which named algorithm, with the swap itself as the primary distractor; a scenario describing a specific merge outcome (a frequent pair not being selected, or a less-frequent pair being selected instead) and asking which algorithm's behavior that outcome is consistent with; a mislabeling item asking whether BPE or WordPiece is "character-level" or "word-level" — the correct answer rejects the premise for either algorithm; and a domain-corpus scenario asking what to do about jargon in a specialized dataset, where the keyed answer is a custom or domain-aware tokenizer, and "strip the jargon out" is the named wrong answer.

What the distractors typically look like

The standing traps in this material's style: attributing BPE's frequency rule to WordPiece or vice versa; describing either algorithm as character-level (ignoring that both build multi-character subword units) or as word-level (ignoring that both fall back to characters for unseen input); offering jargon removal as a solution to a domain-vocabulary fragmentation problem, when a custom tokenizer is the material's own named fix; and presenting BPE and WordPiece as functionally identical procedures that merely happen to carry different names, which is the softest and most tempting version of the swap because the two algorithms genuinely do produce similar-looking vocabularies on ordinary text even though their underlying criteria differ.

Because on ordinary text, a pair's raw frequency and its likelihood-improvement contribution are correlated more often than not — a pair that occurs very often in the corpus is also frequently a pair whose merge meaningfully improves the vocabulary's fit to that corpus, simply because common patterns tend to be genuinely useful units to represent as single tokens. That correlation is exactly why the two algorithms look interchangeable at a glance and why the swap between them is a tempting distractor rather than an obviously wrong one — the algorithms diverge specifically in the less common case where a frequent pair's components are already individually well-modeled (so the likelihood gain from merging them is modest despite the high frequency) or where a less-frequent pair's components are poorly modeled separately (so the likelihood gain from merging them is large despite the lower frequency).

Because "subword" describes the unit the vocabulary is built from, not a ceiling on how long any single token is allowed to be. Both algorithms grow their vocabulary by merging progressively longer fragments, and a sufficiently common whole word will, after enough merge iterations, end up represented as one single vocabulary entry — the same outcome a word-level tokenizer would produce for that specific word. The difference from true word-level tokenization shows up only on words the training corpus did not see often enough to earn their own entry: a subword tokenizer decomposes that rare word into smaller pieces it does have entries for, while a strictly word-level tokenizer has no fallback and maps the entire word to a single out-of-vocabulary symbol instead.

• BPE merges by raw frequency; WordPiece merges by likelihood improvement. These are two genuinely different optimization criteria, not two names for the same rule, and swapping them is this domain's standing distractor.
• Both are subword methods. Neither BPE nor WordPiece is character-level or word-level — both start from characters and grow toward whole words, falling back to characters only for genuinely unseen input.
• BPE's merge order is not fixed from an initial count — pair frequencies are recomputed after every merge, and a merge can create an entirely new countable pair that did not exist in the previous round's tally.
• WordPiece's likelihood criterion can diverge from raw frequency — a less-frequent pair can win if its likelihood-improvement contribution is larger, which is exactly why the two algorithms are not interchangeable despite often looking similar on ordinary text.
• Removing domain jargon is the wrong fix for fragmentation. A custom or domain-aware tokenizer, trained on a corpus that represents the jargon at meaningful frequency, is the correct response.
• GPT family associates with BPE; BERT associates with WordPiece — a useful memorization anchor, but the merge-rule distinction itself is what the exam actually tests underneath that anchor.

Knowing which merge rule built a tokenizer's vocabulary is half of this domain's permanent, undoable decision. The other half is a question about size rather than mechanism: once a vocabulary exists, how large should it actually be, and what does that size decision cost elsewhere in the model.

Next: M3-04 picks up exactly there — the vocabulary-size tradeoff between shorter sequences and a larger embedding table, and why that same tradeoff is the reason perplexity scores stop being comparable the moment two models use different tokenizers.