Cleaning and Curating a Dataset: Deduplication, Imbalance, and NeMo Curator

Identity statement: cleaning and curating a dataset is the set of corrective operations — handling missing values, deduplicating documents, correcting imbalance, and disciplining exactly which rows any preprocessing statistic is computed from — that turns raw, collected data into something safe to train a model on.

Two words are doing separate work in that identity statement, and conflating them is a common source of confusion. Cleaning is corrective: fixing or removing what is already wrong (a missing field, a duplicate row, an outlier value). Curation is more like editorial judgment applied at scale: deciding which documents belong in the corpus at all, correcting the balance of what kinds of content are represented, and making deliberate choices about coverage rather than accepting whatever was scraped or collected. A pretraining corpus assembled by crawling the open web needs both: cleaning to strip out malformed HTML fragments and exact-duplicate boilerplate, and curation to decide that a corpus that is 40% one domain's marketing copy is not a balanced representation of "general text" no matter how much of it there is.

The reason this comes first in the module, and arguably first in the entire ten-domain exam conceptually, is that every other data-preparation decision — how you format the data, how you tokenize it, what vocabulary size you choose, what your exploratory analysis finds — operates on whatever cleaning and curation left behind. A tokenizer trained on a corpus still full of near-duplicate boilerplate will spend vocabulary entries on that boilerplate's most common phrases. An exploratory pass run on a corpus with undetected label noise will report a label distribution that is wrong in exactly the way the noise biases it. Nothing downstream can subtract a defect it has no way of knowing is a defect.

L1 — Intuition: three different kinds of "this row is broken"

A raw dataset breaks in three recognizably different ways before you ever get to imbalance or leakage. Some rows are missing a field entirely — a metadata tag that never populated, a label a labeler skipped. Some rows are duplicates of other rows, exactly or almost exactly, because a document was scraped from two mirrors or a conversation log was exported twice. Some rows are technically complete but inconsistent in form — inconsistent casing, stray whitespace, mixed encodings, slang or jargon that varies across sources for the same underlying concept. Each failure mode calls for a different fix, and applying the wrong one is its own kind of damage: aggressively "normalizing" text can strip out the very domain jargon a specialized model needs to learn, which is why text normalization has to preserve meaningful domain terms even while it standardizes casing and cleans up noise.

L2 — Mechanism: impute or drop, understand why, then deduplicate

For missing values, the two live options are dropping the incomplete row (or, in the extreme, an entire column that is missing so pervasively no imputation strategy can recover a trustworthy signal) and imputing a value that estimates what the missing entry probably was. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names median imputation as the more robust choice specifically because the median is resistant to the outliers that would otherwise pull a mean-based estimate away from where most of the real data actually sits. The instruction to "understand why they're missing" before deciding matters more than it sounds: missingness that is random and rare is safe to drop, but missingness that correlates with some property of the data — a survey question skipped disproportionately by one demographic, a sensor field that fails specifically under one operating condition — carries information, and dropping those rows silently changes what the remaining dataset represents.

Deduplication is a separate operation entirely, and it has to catch two different kinds of repetition. Exact duplicates are byte-for-byte identical documents, usually easy to catch with a hash comparison. Near-duplicates are documents that differ only in formatting, a boilerplate header or footer, or a handful of substituted words — harder to catch, and more common in web-scraped corpora than exact duplicates are, because the same underlying content gets mirrored, syndicated, and reformatted constantly. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states directly that duplicates "bias training and inflate memorization" — a model trained on a corpus where one document appears fifty times effectively sees that document's phrasing fifty times more often than a document that appears once, which skews what the model treats as common and increases the odds that the model reproduces that specific text verbatim rather than generalizing from it.

Text normalization closes out the corrective pass: standardizing casing, handling typos, and reconciling slang and jargon so the same underlying concept is not represented as a dozen superficially different strings. The qualifier that matters here, and that the source material states explicitly, is to do this "consistently (but preserve meaningful domain terms — see tokenization)" — a normalization pass that lowercases and strips everything indiscriminately can destroy the exact vocabulary a domain-specific model most needs intact, a concern this module's tokenization lessons pick up directly.

L3 — The exam-relevant edge case: why duplicates are worse than they look

The intuitive read of deduplication is "it saves storage space." That is true and also the least important reason to do it. The consequential reason is what duplicates do to a trained model's behavior: a document repeated many times across a corpus effectively gets many times the gradient-update weight of a document that appears once, even though nobody chose to weight it that way — the repetition happened by accident of collection, not by deliberate design. That accidental over-weighting shows up downstream as memorization: a model that has seen the same passage dozens of times is measurably more likely to reproduce that passage's exact wording on a related prompt than to generalize a paraphrase of it, which is both a quality problem (the model looks like it is quoting rather than reasoning) and, for corpora containing anything sensitive, a privacy problem. Treating deduplication as a nice-to-have storage optimization rather than a training-integrity requirement is the specific misreading this section exists to correct.

L1 — Intuition

A dataset's label distribution and feature ranges are not neutral facts about the world; they are what the model will learn to treat as normal. If 95% of the labeled examples belong to one class, a model can score a deceptively high aggregate accuracy by predicting that one class every time and essentially ignoring the other 5% — which is exactly the failure mode class-imbalance correction exists to prevent.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) frames this as analyzing "label balance and feature ranges," with imbalance addressed through oversampling, synthetic augmentation, or reweighting. Oversampling duplicates or resamples examples from the minority class so it contributes proportionally more to training than its raw count would otherwise earn it. Synthetic augmentation generates new, plausible examples in the underrepresented class rather than merely repeating existing ones — for text, this can mean paraphrasing, back-translation, or template-based generation of new minority-class examples rather than verbatim copies, which avoids the exact-duplication problem section 2 just spent time warning against. Reweighting takes a different approach entirely: it leaves the dataset's raw composition untouched and instead adjusts the training loss function so that an error on a minority-class example counts for more than an error on a majority-class example, achieving a similar correction without touching the data itself.

Feature-distribution analysis is the numeric-data sibling of label-balance analysis: understanding the range, skew, and outlier structure of any numeric features attached to a dataset (document length, a metadata score, a timestamp-derived feature) before those features get scaled or fed into a model. A feature with a long right tail of extreme values behaves very differently under a scaler than a feature with a roughly symmetric spread, and knowing which one you have determines which scaling approach in section 4 is actually appropriate.

L3 — The exam-relevant edge case: imbalance is a training-time problem, not an evaluation-time patch

A common but incomplete response to discovering imbalance is to fix the evaluation metric rather than the training data — reporting per-class recall instead of aggregate accuracy, for instance. That is a genuinely good practice for measuring the problem, but it does nothing to fix the underlying training dynamic: a model trained on a 95/5 split will still have seen the minority class fifty-one times less often than the majority class, regardless of which metric later reveals that fact. Oversampling, augmentation, and reweighting are training-time interventions precisely because the problem they solve is a training-time problem — what the model actually learned to weight as common — not a reporting-time problem.

L1 — Intuition

Once numeric features are understood well enough to scale, and once categorical fields are understood well enough to encode, a single procedural rule governs every step that follows: whatever statistic a preprocessing step needs — a minimum and maximum for normalization, a mean and standard deviation for standardization, a category list for a one-hot encoder — must be computed from the training split alone, never from validation or test data, and never from the full dataset before any split has happened.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states the trap directly: "Fit scalers/encoders on train only. Fitting on the full dataset leaks test information and inflates results." The mechanism behind why this is leakage, and not merely a stylistic preference, is worth stating precisely: a scaler's fitted parameters are themselves a summary of whatever data was fed into the fit step. If that data includes validation or test rows, then the transformation later applied to the training data has, in a small but real way, been shaped by values the model is never supposed to have had access to before its final evaluation. The correct procedure splits the data first, fits every scaler and encoder on the training split's rows alone, and then applies that already-fitted transformation — using the training split's own min/max, mean/std, or category list, unchanged — to the validation and test splits without ever refitting on them.

This is not a special rule invented for LLM datasets; it is the same data-leakage discipline that governs any supervised learning pipeline, restated here because it recurs specifically in the context of preparing text for pretraining or fine-tuning, where scaling and encoding show up less obviously than in tabular ML — a length-based feature used for filtering, a metadata score used for sampling weight, a source-domain category used for stratification are all places a fit-on-everything mistake can slip in unnoticed, precisely because the "obvious" scaler-on-numeric-columns case is not the shape the leakage takes in a text pipeline.

L3 — The exam-relevant edge case: leakage does not require touching a label

The detail that separates a shallow understanding of this rule from a correct one is that leakage never requires the model to see a test label. It only requires a test row to have influenced some number — a mean, a max, a category list — that subsequently touches the training pipeline. A scenario item can describe a pipeline that never lets the model see test labels at any point and still be describing a leaky pipeline, because the leak happened one step earlier, at the scaler's fit call, before the model ever entered the picture. Recognizing that distinction — leakage is about information flow through any computed statistic, not specifically about label visibility — is what separates a correct read of this trap from an incomplete one.

⭐ THE EARNED INSIGHT: > Data leakage in preprocessing is never about the model peeking at an answer key — it is about any number computed from data outside the training split quietly influencing anything the training split subsequently touches, and a scaler's fitted mean or an encoder's category list is exactly that kind of number. The fix has one shape every time: split first, fit second, and never let that order reverse, no matter how far downstream from "the model" the actual computation looks.

Everything in sections 2 through 4 is correct as a set of principles regardless of scale, but a pretraining corpus for an LLM is not a spreadsheet with a few thousand rows — it can be terabytes of text, where a CPU-bound deduplication or filtering pass that would take minutes on a small dataset becomes a multi-day bottleneck. [VENDOR SPEC] (Sources/ncp-genl/domain-3-data-preparation.md) names two specific NVIDIA tools built for exactly this scale problem.

The relationship between the two named tools is layered rather than competing: RAPIDS cuDF is the general-purpose GPU dataframe engine — the direct, faster substitute for the pandas operations a data scientist already knows how to write — while NeMo Curator is a purpose-built curation toolkit that specifically targets the deduplication and quality-filtering problem this lesson has been describing, built to operate at the scale an LLM pretraining corpus actually reaches. Neither tool changes what the right operation is; deduplication is still deduplication, and the train-only-fit rule from section 4 still applies no matter which library executes the fit call. What GPU acceleration changes is whether that correct operation finishes in a timeframe a real pretraining pipeline can afford.

The practical reason this distinction is worth holding onto rather than treating both tools as interchangeable "the GPU one for data" answers: a scenario item that describes a specific operation — deduplicating a terabyte-scale web crawl, filtering low-quality documents out of a pretraining corpus, computing quality-score-based sampling weights — is naming a curation-layer task, and NeMo Curator is the tool built for exactly that layer. A scenario item that describes a more generic dataframe operation — grouping, joining, filtering by a column value, computing an aggregate statistic across millions of rows — is naming a dataframe-layer task, and cuDF is the general accelerator underneath it, the same way pandas underlies countless purpose-built data tools on the CPU side. Curator is, in effect, built on the kind of dataframe-acceleration foundation cuDF provides, not a competitor to it.

Treat the following as a constructed scenario built to make the diagnosis legible, not a measurement from any real dataset. A team is fine-tuning a model on a labeled support-ticket dataset of 50,000 examples, with a numeric ticket_length feature they plan to use for stratified sampling.

Reported pipeline (BUGGY):

scaler = StandardScaler()
X_scaled = scaler.fit_transform(ticket_lengths_full)   # fit on ALL 50,000 rows
train, val, test = split(X_scaled, ratios=(0.8, 0.1, 0.1))
model.fit(train)
# validation macro-F1: 0.91
# production macro-F1 (30 days later): 0.74

Walk the diagnosis the way an exam scenario expects: a 17-point gap between validation and production performance, with no architecture change between the two measurements, points at the data pipeline rather than the model. scaler.fit_transform(ticket_lengths_full) runs before the split, on all 50,000 rows — including the 5,000 that will become validation and the 5,000 that will become test. The mean and standard deviation baked into scaler therefore already reflect those held-out rows. When validation is later scored, the transformation being applied to it was shaped, in part, by its own values — a milder version of testing on your own training data. Production traffic, collected after the fact, was never part of that fit call at all, so the artificially favorable validation score does not survive contact with genuinely unseen data.

Corrected pipeline:

train, val, test = split(ticket_lengths_full, ratios=(0.8, 0.1, 0.1))
scaler = StandardScaler()
scaler.fit(train)                        # fit on TRAIN ONLY
train_scaled = scaler.transform(train)
val_scaled   = scaler.transform(val)     # apply, never refit
test_scaled  = scaler.transform(test)    # apply, never refit
model.fit(train_scaled)

This is a constructed scenario — the 0.91/0.74 figures are illustrative, chosen to be a plausible-looking gap rather than measured from a real run — but the mechanism it demonstrates is exactly the one Section 4 names as the domain's central trap, and it generalizes past scalers to any preprocessing statistic computed from data rather than supplied as a fixed rule: an imputation median, an outlier fence, a category list for encoding.

Consider a second constructed scenario, sized to make the arithmetic of duplication concrete rather than to represent any measured corpus. A team assembles a 10-million-document web crawl intended for continued pretraining. Before any cleaning:

Raw corpus:                        10,000,000 documents
Exact duplicates (byte-identical):    620,000 documents  (6.2%)
Near-duplicates (>90% content overlap): 890,000 documents (8.9%)
Corpus after deduplication:          8,490,000 documents  (15.1% removed)

Fifteen percent of the raw crawl disappears in deduplication alone — a large fraction for a step some teams treat as a formality. Now trace what those removed documents would have done if left in. Suppose one syndicated news article, mirrored across 40 near-identical outlet copies, is among the near-duplicates removed.

If retained: the model sees that article's exact phrasing 40 times across training
If deduplicated to 1 copy: the model sees that phrasing once, same as any other single document

Effective over-weighting if retained, relative to a single-copy document: 40x

That 40x is not a deliberate curation choice anyone made; it is an accident of how the article happened to be syndicated across the web, and it is exactly the kind of accidental over-weighting that biases training and inflates memorization, in the source material's own framing. Removing it restores the intended property that each unique piece of content contributes roughly the training signal its content warrants, not the training signal its incidental republication count happens to produce.

Data Preparation is objective cluster 3.1 through 3.5, and cleaning and curation specifically map to objective 3.1. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) frames the domain's 9% weight as punching above its size because "data quality caps everything downstream" — a claim that generalizes across every other domain on the exam, since a fine-tuned, quantized, evaluated, and deployed model is still bounded by whatever the training data was.

Expect this material in a small number of recurring shapes. A scenario naming the fit-before-split leakage pattern directly, usually phrased as a pipeline that scales or encodes the full dataset before splitting into train and test — the keyed answer names data leakage, and the standard distractor offers "nothing is wrong with this" or a vaguer "this will overfit," which sounds plausible but names the wrong mechanism. A scenario naming duplicate content in a corpus and asking what the consequence is — the keyed answer is biased training and inflated memorization, not merely wasted storage. A tool-identification item pairing a described task with the right NVIDIA tool — large-scale deduplication and curation points at NeMo Curator, and GPU-accelerated general dataframe work points at RAPIDS cuDF, with the two occasionally offered as interchangeable distractors for each other despite occupying different layers of the pipeline.

What the distractors typically look like

The standing traps in this lesson's material are: describing "fit the scaler on the full dataset, then split" as an acceptable or even preferable sequence; describing deduplication's benefit purely in terms of storage savings rather than training-integrity; offering mean imputation as the default correct choice on a skewed column, when median is the more robust option the source material names directly; and swapping NeMo Curator and RAPIDS cuDF's actual jobs — attributing curation-specific deduplication logic to cuDF, which is a general dataframe accelerator, or attributing general dataframe operations to Curator, which is a purpose-built curation toolkit layered on top of that kind of infrastructure.

A near-duplicate is a document that is not byte-identical to another but shares enough of its content — typically measured by some form of content-overlap or shingling comparison — that it functions as a repeat for training purposes even though a simple hash comparison would miss it. Near-duplicates matter more in practice than exact duplicates specifically because they are more common in real collected corpora: the same article gets mirrored across outlets with different headers, the same product description gets copied across a dozen retailer pages with a different price footer, and none of those pairs are byte-identical even though their actual content is the same information repeated. A cleaning pipeline that only catches exact duplicates will systematically under-detect this much larger and more consequential category of repetition.

Because leakage is about the flow of information, not specifically about label visibility. A scaler's fitted parameters — a mean, a standard deviation, a minimum and maximum — are statistics computed from whichever rows are fed into the fitting step, and if that step includes validation or test rows, the transformation subsequently applied to the training data has already been shaped, however slightly, by data the model is not supposed to have influenced anything with yet. The model can go the entire pipeline without ever seeing a single test label and the leakage has still occurred, because the leak happened one step earlier, in the preprocessing statistic itself, before the model entered the picture at all.

Not always, and the choice among oversampling, synthetic augmentation, and reweighting depends on how much minority data exists and how expensive it is to generate more. Plain oversampling — resampling the same minority examples so they appear more often — is the simplest option and works well when the minority class has enough genuine diversity that repeating its existing examples does not itself become a mini version of the deduplication problem from section 2: repeat the same handful of minority examples too aggressively, and the model starts memorizing those specific examples rather than generalizing the pattern they represent. Synthetic augmentation avoids that by generating genuinely new minority-class examples rather than copies of existing ones, at the cost of needing a paraphrasing, back-translation, or template pipeline capable of producing plausible new examples rather than noise. Reweighting avoids touching the data at all, which keeps the corpus's actual content honest, but it requires a training setup that supports per-example or per-class loss weighting, which not every fine-tuning framework exposes as a simple option. None of the three is categorically superior; the right choice follows from how much minority data exists, how it was collected, and what the training pipeline actually supports.

• Cleaning and curation are four jobs — missing values, deduplication, imbalance correction, and disciplined preprocessing statistics — and a defect in any one caps what everything trained afterward can achieve.
• Deduplication's real cost is biased training and inflated memorization, not wasted storage — a document repeated by accident of collection gets repeated training weight nobody intended.
• Median imputation, not mean, is the robust default on skewed or outlier-heavy columns.
• Fit every scaler and encoder on the training split only. Fitting on the full dataset leaks test information into training regardless of whether any label was ever visible to the model.
• NeMo Curator handles large-scale curation and deduplication; RAPIDS cuDF is the general GPU dataframe layer beneath it — related tools, different jobs.
• Class imbalance is a training-time problem that oversampling, augmentation, or reweighting address directly; a better evaluation metric only reveals the problem, it does not fix it.

Cleaning and curating a dataset answers what has to be true about a corpus before anything is safely trained on it. It does not yet answer what shape that clean data needs to take for a specific downstream job — a JSONL file of instructions, a prompt/response pair for supervised fine-tuning, or a chunked passage for retrieval are three very different structures built from the same clean underlying content.

Next: M3-02 picks up exactly there — organizing and formatting a cleaned dataset for pretraining, fine-tuning, and RAG — and the clean train/validation/test split discipline established here carries forward directly into how those splits must stay leak-free once the data is reshaped into each format.