Stemming vs lemmatization, and stop-word removal explained

Three distinct operations, each reducing text to a more canonical form so that superficially different strings are treated as the same thing.

Stemming strips suffixes (and occasionally prefixes) according to a fixed set of rules, aiming to reduce inflected forms to a common stem. It does not consult a dictionary and does not know what a word is. It is fast, deterministic, language-specific in its rules, and it frequently produces strings that are not valid words. Classic implementations are the Porter stemmer, the more aggressive Snowball (Porter2) family, and the Lancaster stemmer.

Lemmatization maps a word to its lemma — the canonical dictionary headword — using a lexical resource and, critically, knowledge of the word's part of speech. The output is always a real word. Because it is dictionary-backed it handles irregular forms that no suffix rule could reach: went → go, mice → mouse, better → good (as an adjective). It is slower than stemming and needs more machinery: a lexicon and usually a part-of-speech tagger.

Stop-word removal deletes high-frequency function words — the, a, is, of, and — on the theory that they carry little topical information and mostly add noise and index size. The stop-word list is a choice, not a fact, and choosing it badly destroys meaning.

The unifying purpose is vocabulary collapse for a bag-of-words world. If your representation is a count of terms (02-06), then run, runs, running and ran occupying four separate dimensions is a problem: it fragments the evidence for a single concept across four sparse features. Collapsing them to one term concentrates the signal. That was the governing constraint of classical NLP, and it is why these tools were standard.

A transformer does not have that constraint. It sees subword pieces in context (02-02) and learns for itself that running and ran relate — from data, not from a rule table. Which is why the honest framing of this whole lesson is: these are the right tools for sparse lexical representations and usually the wrong tools ahead of a language model.

L1 — The intuition: scissors versus a dictionary

Stemming is a pair of scissors with a printed instruction sheet. The sheet says things like "if the word ends in sses, replace with ss," "if it ends in ies and is long enough, replace with i," "if it ends in ing and what remains contains a vowel, drop the ing." You apply the rules in order and keep whatever is left. The scissors have never heard of English. They cannot tell you whether the result is a word, and they do not care.

Lemmatization is a dictionary with a grammar consultant attached. You hand it a word and — importantly — tell it what part of speech the word is playing in this sentence. It looks the form up, follows the entry back to the headword, and hands you that. The result is always a real word because it came out of the dictionary. It costs more: you need the dictionary, and you need the consultant to tell you whether saw is a noun (→ saw) or a past-tense verb (→ see).

That last example is the whole reason lemmatization needs part of speech, and it is worth carrying: saw lemmatizes to saw as a noun and to see as a verb. No amount of suffix-rule cleverness resolves that; it requires knowing the word's grammatical role.

L2 — The mechanism

Stemming, mechanically. The Porter stemmer is the canonical example and it works in ordered phases. Roughly: a first phase handles plurals and past participles (sses→ss, ies→i, and conditional removal of ed and ing), later phases handle derivational suffixes (ational→ate, izer→ize, fulness→ful), and a final phase trims residual es and doubled consonants. Each rule is guarded by a condition on the stem's structure — typically a measure of how many vowel-consonant sequences remain — so that rules do not fire on words too short to survive them.

Two properties follow directly from this design, and both are testable:

• Over-stemming (false positives): two words with different meanings collapse to the same stem. universe, university and universal can all reduce to univers. The index now cannot distinguish them.
• Under-stemming (false negatives): two forms of the same word fail to collapse. Irregular verbs are the standard case — no suffix rule turns went into go, or mice into mouse.

The Lancaster stemmer is more aggressive: shorter stems, more collapse, more over-stemming. Snowball sits between Porter and Lancaster and has variants for many languages. The aggressiveness dial is the practical choice, and it trades recall against precision in a search index.

Lemmatization, mechanically. Three steps:

1. Part-of-speech tagging. Determine each token's grammatical role from its sentential context. This is itself a learned model in modern toolkits.
2. Lexicon lookup. Consult a lexical resource — WordNet is the canonical English one — for the base form of that surface form under that part of speech. WordNet organises words into synsets and links inflected forms to headwords, which is exactly the structure a lemmatizer needs.
3. Return the lemma, always a valid word.

The dependency on step 1 is not optional, and skipping it is the most common lemmatization bug in practice. Most library APIs default to assuming the word is a noun if you do not say otherwise, which is why running lemmatized without a POS tag comes back as running — as a noun, that is the lemma — while running tagged as a verb correctly gives run. Someone who reports that "lemmatization did nothing" has almost always forgotten to pass the tag.

WordNet is worth naming carefully, because the exam's most-reported confusable in the adjacent territory is WordNet versus word2vec: WordNet is a hand-built curated lexical ontology of synsets and relations; word2vec is a set of dense vectors learned from co-occurrence statistics. Lemmatization uses the former. That pairing is developed further where embeddings are taught, but the distinction is worth fixing here, since WordNet enters the course as a lemmatizer's dictionary.

Stop-word removal, mechanically. Filter tokens against a list. The subtleties are all in the list:

• Lists are not standard. Different libraries ship different lists of different lengths, so "remove stop words" is not a reproducible instruction without naming the list.
• Removal is destructive and non-invertible. Once dropped, the words are gone.
• Function words carry meaning in specific and important cases. Negation is the headline: strip not and no and "this is not a problem" becomes "this problem." Sentiment analysis, legal text and medical text all break. Phrases are the second case: to be or not to be is almost entirely stop words; so are The Who, Take That, IT and US.
• For n-grams and phrase queries, removing stop words breaks adjacency: president of the united states becomes president united states, and a bigram index built after removal contains bigrams that never occurred in the source.

L3 — The canonical preprocessing pipeline order

The exam asks about the order directly, so learn it as a sequence with reasons attached:

1. lowercase
2. strip special characters / punctuation
3. tokenize
4. remove stop words
5. lemmatize

The order is not arbitrary and each step's position is defensible:

Lowercase first so that The and the are the same string for every step that follows — including stop-word matching, which is a literal string comparison against a list of lowercase words.

Strip special characters before tokenizing so the tokenizer is not splitting on punctuation you intend to discard anyway, which keeps token boundaries clean and avoids empty tokens.

Tokenize third, because every subsequent step operates on tokens rather than on a raw string. This is the boundary between "text cleaning" and "token processing."

Remove stop words before lemmatizing, because it is pure economy: there is no point paying for POS tagging and lexicon lookup on words you are about to delete. The two steps are also independent, so ordering is free to be chosen for efficiency.

Lemmatize last because it is the most expensive step and the most context-dependent, so it should run on the smallest surviving token set. Note that this is the canonical order's choice of lemmatization over stemming — a deliberate signal that lemmatization is the higher-quality option when you can afford it.

Two caveats worth carrying, because a well-written scenario question can turn on them. First, lemmatization needs POS tags, and POS tagging needs the original sentence structure — including punctuation and word order. So a pipeline that strips punctuation and removes stop words before tagging has degraded the tagger's input. In a quality-critical pipeline you tag early, on relatively intact text, then filter. The canonical order above is the exam's canonical order and the one to reproduce when asked; the practical caveat is worth knowing but is not the answer to "what is the canonical order?"

Second, for an LLM you generally run none of steps 1, 2, 4 or 5. More on that in §5, and it is the more consequential of the two caveats for real work.

The comparison table for the module's most heavily reported confusable.

The one-sentence version to have ready: stemming is crude truncation that may produce a non-word; lemmatization is dictionary-based and always produces a valid root, but needs the part of speech. If you can produce that sentence and one example of each — studies→studi versus studies→study — you have what the exam asks for.

Worked contrast on a single word list

Read the caring/cars row pair together, because it is the clearest demonstration of why over-stemming is not a cosmetic complaint. In a search index built on Porter stems, a query about automobiles and a document about compassion share a term. That is a precision failure with a mechanical cause.

Take the sentence:

The researchers were not studying better mice; they studied 3 universities' caring practices!

Walk it through the five canonical steps. The stems below follow Porter-style behaviour and the lemmas follow WordNet-style behaviour; both are constructed illustrations of the rules described in §2 rather than captured tool output, and library versions differ at the margins.

Step 1 — lowercase.

the researchers were not studying better mice; they studied 3 universities' caring practices!

Step 2 — strip special characters. Removing punctuation and the possessive apostrophe. Note that the digit 3 survives unless you also strip digits, which is a separate choice with its own consequences — strip digits and you lose quantities, which matters in medical and financial text.

the researchers were not studying better mice they studied 3 universities caring practices

Step 3 — tokenize (word-level, since this is a classical pipeline — not subword).

["the","researchers","were","not","studying","better","mice","they","studied","3","universities","caring","practices"]

13 tokens.

Step 4 — remove stop words. Using a typical English list containing the, were, not, they:

["researchers","studying","better","mice","studied","3","universities","caring","practices"]

9 tokens. And here is the damage: not is gone. The sentence said the researchers were not studying something. After stop-word removal the representation says they were. For a topic classifier that may be tolerable. For sentiment analysis, a clinical note, or a contract clause it is a correctness failure with no error message attached. This single line is the strongest argument in the lesson for treating stop-word lists as a decision requiring justification.

Step 5 — reduce to roots. Two branches, side by side.

Stemming branch (Porter-style):

["research","studi","better","mice","studi","3","univers","car","practic"]

Observations, all of them exam-relevant:

• studi is not a word — appears twice, correctly collapsing studying and studied
• better is unchanged — under-stemming on an irregular comparative
• mice is unchanged — under-stemming on an irregular plural
• univers — over-stemming; would collide with universal and universe
• car from caring — over-stemming into an unrelated real word, the worst kind of collision
• practic is not a word

Lemmatization branch (with correct POS tags):

["researcher","study","good","mouse","study","3","university","care","practice"]

Observations:

• every output is a real word
• studying and studied both → study, the same collapse stemming achieved but with a valid root
• better → good and mice → mouse: irregular forms handled, which stemming could not do
• university stays distinct from universal: no damaging collapse
• caring → care, not car: the collision is avoided

Score the two branches. Nine tokens each. Stemming produced 3 non-words, 2 under-stemmings and 2 harmful over-stemmings — 7 defects across 9 tokens. Lemmatization produced 9 valid words with the intended collapses and no collisions. Cost: lemmatization required a POS tagger and a lexicon lookup per token; stemming required a rule table and finished in microseconds.

That is the trade-off in full, on one sentence, and it is why the canonical pipeline names lemmatization rather than stemming as its final step while search-engine practice often still uses stemming: quality versus throughput, with the choice depending on whether a human ever sees the output.

Now the counterweight, and the part practitioners get wrong most often. Feed the original sentence to a modern LLM tokenizer instead of the pipeline:

"The researchers were not studying better mice; they studied 3 universities' caring practices!"

A subword tokenizer splits this into pieces and hands them to the model with everything preserved: the capital T, the word not, the semicolon, the possessive apostrophe, the digit 3, the exclamation mark. Every one of those carries information a transformer can use:

• not — the model's attention mechanism handles negation scope directly; deleting it would have destroyed the sentence's meaning
• Capitalisation — signals proper nouns and sentence boundaries
• Punctuation — the semicolon marks clause structure; the apostrophe marks possession
• Inflection — studying versus studied is a tense distinction the model uses to place events in time; collapsing both to study throws away the tense
• Plurality — mice versus mouse is a quantity distinction

Every classical preprocessing step, applied here, removes signal the model would have used. This is the point COURSE-INDEX makes directly in its text-preprocessing entry: modern LLMs need far less of this pipeline than classical NLP did. The reason is structural. Classical models used sparse lexical features where vocabulary fragmentation was fatal, so collapse was necessary. Transformers use contextual subword representations learned from data; they discover morphological and syntactic relationships themselves, and they need the surface form intact to do it.

Decision table — which preprocessing for which pipeline:

The pattern across the table: normalise for lexical matching, preserve for neural understanding. And when normalising for matching, normalise the key and keep the original text, so the destruction is confined to the comparison and never propagates into what you index or show.

Text preprocessing sits under NCA-GENL objectives 2.1 and 2.3 — awareness of extracting insights from large datasets, and conducting data analysis under supervision — and it connects to objectives 1.6 and 1.10 on Python natural-language packages, since spaCy and NLTK are the tools that implement these steps. The blueprint's Data Analysis module places the text-preprocessing pipeline at Tier 1 and calls out "stemming (crude truncation) vs lemmatization (dictionary-based, valid root) — heavily reported" in exactly those words.

That "heavily reported" tag is field calibration from published candidate reports rather than official documentation, so treat it as guidance on study-time allocation rather than a guarantee about any individual question. But two independent lines point the same way: the field reports specifically list lemmatization among the classical-NLP items that punch above their blueprint weight on this exam, alongside BLEU scores, WordNet versus word2vec, lexical diversity versus syntactic complexity, and spaCy. And the drill emphasis for the Data Analysis module names stemming versus lemmatization first. This is a distinction to know cold.

The same reports characterise questions as general-level: know at a high level what each thing is and when to use it. So you will not be asked to reproduce the Porter stemmer's phase-two rules. You will be asked to identify which technique produced a given output, which one guarantees a real word, which one needs part of speech, and which one to choose in a described scenario.

Question phrasings to expect:

• "What is the difference between stemming and lemmatization?" — Stemming is rule-based truncation that may produce a non-word; lemmatization is dictionary-based and returns a valid canonical root using part of speech.
• "Which technique would convert studies to study?" — Lemmatization. Stemming gives studi.
• "Which technique requires part-of-speech information?" — Lemmatization.
• "Which is faster?" — Stemming, because it applies string rules with no lexicon or tagger.
• "What is the canonical order of a text-preprocessing pipeline?" — lowercase → strip special characters → tokenize → remove stop words → lemmatize.
• "Why might stop-word removal harm a sentiment classifier?" — Standard lists contain negations like not and no, whose removal inverts meaning.
• "A team applies lowercasing, stop-word removal and stemming before sending text to a large language model. What is the effect?" — It degrades performance by destroying case, negation and inflection information the model uses. This is the modern-practice question, and it is the one most likely to be missed.
• "Which lexical resource does an English lemmatizer typically consult?" — WordNet.

Distractor families:

Stemming applies fixed suffix-stripping rules and may return a string that is not a word. Lemmatization consults a dictionary and returns the valid canonical form, using the word's part of speech to disambiguate. That is the whole answer, and the exam wants it in about that many words.

The three consequences worth attaching:

1. Validity. Stems can be non-words (studi, practic, wa). Lemmas are always real words, because they came from a lexicon.
2. Irregulars. Lemmatization handles went→go, mice→mouse, better→good. Stemming cannot, because no suffix rule reaches them.
3. Cost and dependencies. Stemming needs a rule table and runs in microseconds. Lemmatization needs a POS tagger and a lexicon, and runs slower.

Canonical example pair to memorise: studies → studi (stemming) versus studies → study (lemmatization).

No. There is no case where it helps and several where it clearly hurts.

Stop-word removal made sense when the representation was a sparse count of terms and function words genuinely contributed noise plus index bloat. A transformer is the opposite kind of model: it reads the sequence in context, and function words are how the sequence's structure is expressed. Prepositions carry relations (the drug for the condition versus the drug from the condition), articles carry definiteness and reference, auxiliaries carry tense and modality, and negation carries truth value.

The negation case alone settles it. not is on essentially every standard stop-word list, and removing it inverts the meaning of the sentence with no error and no warning. If you want a rule: remove stop words only when your representation cannot see word order at all. Bag-of-words and TF-IDF cannot. An LLM can.

lowercase → strip special characters → tokenize → remove stop words → lemmatize

The rationale, briefly: lowercase first so every later string comparison — including stop-word matching — is case-insensitive; strip special characters before tokenizing so the tokenizer produces clean boundaries; tokenize third because everything after operates on tokens; remove stop words before lemmatizing because there is no point paying to lemmatize words you will delete; lemmatize last because it is the most expensive and most context-dependent step and should see the smallest token set.

Two things to note. The canonical order names lemmatize, not stem — a deliberate signal that lemmatization is the preferred final step when affordable. And this is a classical NLP pipeline: for an LLM you run none of steps 1, 2, 4 or 5, and step 3 is done by the model's own subword tokenizer.

spaCy is the one to know by name, because the exam's own objectives list it explicitly and field reports name it directly. spaCy provides tokenization, part-of-speech tagging, named-entity recognition, lemmatization and dependency parsing in one pipeline — and crucially it tags before it lemmatizes, so its lemmas are POS-aware without you wiring that up. It does not ship a classical stemmer, on the design view that lemmatization is the better tool.

NLTK is the classical-NLP toolkit and ships the stemmers: Porter, Snowball and Lancaster, plus a WordNet-based lemmatizer and stop-word lists for many languages. NLTK's WordNet lemmatizer is the one that defaults to noun if you do not pass a POS tag, which is the source of the "lemmatization did nothing" bug.

scikit-learn provides CountVectorizer and TfidfVectorizer, which handle lowercasing, tokenizing and stop-word removal as constructor options but do not stem or lemmatize — you pass a custom analyser or preprocess beforehand. Those two classes are named in the blueprint's text-preprocessing entry and are the subject of 02-06.

No, and expecting it to is a common source of confusion. A word already in canonical form comes back unchanged: study lemmatizes to study, mouse to mouse, run to run. That is correct behaviour, not a failure.

The trap is the POS default. running as a noun has the lemma running — "the running of the machine" is a perfectly good noun phrase. So a lemmatizer called without a POS tag, defaulting to noun, returns running unchanged and looks broken. Tag it as a verb and you get run.

Diagnostic: if lemmatization seems to be a no-op across your corpus, check whether you are passing part-of-speech tags before you check anything else. It is almost always that.

1. The core distinction, cold: stemming is crude rule-based truncation that may produce a non-word; lemmatization is dictionary-based reduction to a valid canonical root that needs the part of speech. studies → studi versus studies → study.
2. Lemmatization handles irregular forms; stemming cannot. went→go, mice→mouse, better→good are unreachable by suffix rules.
3. Stemming is faster and more aggressive. Over-stemming collapses distinct words (caring and cars both → car), which is a precision failure with a mechanical cause.
4. Lemmatization without POS tags looks broken. Most APIs default to noun, so running comes back unchanged. Check the tag first.
5. The canonical pipeline order is lowercase → strip special characters → tokenize → remove stop words → lemmatize, and it names lemmatize rather than stem for a reason.
6. Stop-word removal is destructive and list-dependent. Standard lists contain negations, so removal can invert meaning silently. Never apply it to negation-sensitive tasks.
7. Modern LLM pipelines need far less of this than classical NLP did. Preprocessing before a transformer deletes case, punctuation, negation and inflection that the model uses. Send raw text.
8. Normalise for lexical matching; preserve for neural understanding. When you must normalise, normalise a derived key and keep the original text as source of truth.
9. Apply identical preprocessing to queries and to the index. Asymmetry between the two is a silent retrieval-quality killer.
10. Know the tools by name: spaCy (tokenize, POS, NER, lemmatize, parse — no stemmer), NLTK (the stemmers plus a WordNet lemmatizer), scikit-learn's CountVectorizer and TfidfVectorizer (lowercase, tokenize, stop words; no stemming).

Every technique in this lesson existed to serve a representation that has not yet been described. Collapsing studies and studying to one term, deleting the and of, lowercasing everything — none of it makes sense until you know what consumes the output. The answer is a sparse count vector: a representation where each vocabulary term is a dimension and the value is a count or a weight, with no word order at all.

That representation is why vocabulary fragmentation was fatal and why stop words were noise. It also has an obvious defect once stated — throwing away word order means the dog bit the man and the man bit the dog are identical vectors — and n-grams are the classical patch for it.

Next: 02-06 builds all three: bag-of-words as the base representation, TF-IDF as the weighting that makes rare informative terms count for more than common ones, and n-grams as the way a little word order gets smuggled back in. It closes the module by making the case these methods can still win — small data, interpretable weights, no GPU — and by naming the limitation that motivates everything in 03-01: a sparse count vector cannot know that car and automobile mean the same thing.