Data Quality Problems: Label Noise, Data Leakage, Class Imbalance, and Drift

A data quality problem is a property of a dataset that causes a model trained or evaluated on it to be systematically wrong in a way the metrics do not reveal. The defining characteristic is the second clause. A dataset with a missing column throws an error, so it is not really a quality problem — it is a bug, and bugs announce themselves. A dataset with 4% mislabeled examples trains a model that scores 96% and quietly disagrees with reality on exactly the cases it was taught wrong.

That asymmetry is why the vocabulary matters. You cannot detect what you cannot name, and in multiple-choice form the question is almost always which defect is this? rather than how would you fix it?

Two lists, and the boundary between them:

Data leakage appears on both lists, and it is the only member that does. That is not sloppiness — leakage is simultaneously a record-level artefact (a column that should not be there) and a dataset-level pathology (a train/test boundary that was crossed). It is also the highest-frequency data-quality item on the exam, so if you have limited study time, leakage is where to spend it.

L1 — Intuition: four ways a dataset lies

Strip the jargon and the four pathologies are four different lies:

• Label noise — "here is the right answer" when it is not.
• Leakage — "here is what you'll know at prediction time" when you will not.
• Imbalance — "here is what the world looks like" when 98% of it is one thing and you care about the other 2%.
• Drift — "here is what the world looks like" when that was true last year.

Note that three of the four are lies about representativeness and only one is a lie about correctness. That is why the reflex "clean the data" is usually the wrong reflex: scrubbing individual records does nothing about a dataset that is internally immaculate and collectively unrepresentative.

L2 — The mechanism of each defect

Label noise. The target values are partly incorrect. Sources, in rough order of frequency: annotation guidelines written after labeling began; genuinely ambiguous items where two reasonable people disagree; annotator fatigue on long sessions; automated labeling by heuristic or by another model; and class definitions that overlap. The mechanism of harm is that gradient descent has no way to distinguish a wrong label from a hard example — it optimises toward both — so noisy labels do not merely fail to help, they actively teach wrong behaviour. Two practical consequences: your measured accuracy has a ceiling at the label accuracy (a model cannot score 98% against a 95%-correct test set except by luck), and noise in the test set is worse than noise in the training set, because a model can average out training noise but cannot fix a grader that is wrong. The instrument for detecting it is inter-annotator agreement, in 09-03.

Data leakage. Information about the target reaches the model during training that will not be available when the model runs. Five mechanisms worth naming individually:

The last row is the LLM-specific one and it is why 10-01 exists: when a model is pretrained on a web crawl, any public benchmark published before the crawl may be inside the weights, and a strong benchmark score may be partly recall. It is also the strongest single argument for maintaining a private evaluation set — the one you curated in 08-01.

Class imbalance. One class constitutes the overwhelming majority. The harm is not to the model directly but to the metric: a fraud detector on a dataset that is 99.4% legitimate achieves 99.4% accuracy by predicting "legitimate" always. The named fix is not a rebalancing trick, it is a metric change — precision, recall, F1, and the confusion matrix, per 09-05 — and only then class weighting, stratified splitting (09-08), or resampling. Two rules that get tested: stratify your splits so the minority class appears in every fold, and resample only the training split, because resampling the test split makes your measured performance a statement about a world that does not exist.

Drift. The distribution the model sees in production diverges from the one it was trained on. Three named varieties:

Covariate drift is the cheap one to detect because it needs no ground truth; concept drift is the dangerous one because detecting it requires the labels you do not have yet. For LLM systems drift shows up in specific places: the corpus goes stale, users' phrasing shifts, an upstream model version changes, or your prompt starts receiving inputs it was never tested on. Monitoring is 12-14.

L3 — Sampling bias, and why it is not the same as imbalance

Sampling bias is the fifth pathology and the one most often collapsed into the other four. Class imbalance is a true property of the world that your dataset faithfully reflects. Sampling bias is a property of your collection process that the world does not share. A dataset that is 99% legitimate transactions is imbalanced and correct. A dataset of customer complaints collected only through a web form is biased: it silently excludes every customer who phones, and no amount of resampling recovers them because they were never in the pool.

The distinction has teeth because the fixes are opposite in kind. Imbalance is addressed analytically — change the metric, weight the loss, stratify the split. Bias can only be addressed by changing collection, which is expensive and often organisationally unwelcome. And bias is the pathway by which unfairness enters a model: a dataset that under-represents a group produces a model that performs worse for that group, and the aggregate metric will not say so. That is the thread picked up in 08-04 and closed in 13-04.

There is a fourth relative here worth naming: survivorship bias, where the collection process only retains successful or completed cases. The help-desk example from 08-01 is exactly this — tickets that were never answered tend not to persist as clean records, which is precisely why the unanswerable cases had to be written rather than found.

This is the highest-value table in the module. Read it symptom-first, the way an exam stem presents it.

Two entries in that table are the ones to over-learn, because they are the two most-reported shapes. "Great offline, bad in production, immediately" is leakage. "Good at launch, bad six months later" is drift. The time signature discriminates them completely: leakage is wrong from the first production request; drift is right at first and decays.

And a third discriminator that catches people: overfitting is not drift and not leakage. Overfitting shows as a gap between training and validation performance, visible before you ever deploy, and it is diagnosed from loss curves (12-03). Leakage shows as a gap between validation and production. Drift shows as decay in production over time. Three gaps, three different pairs of numbers.

The numbers here are constructed for illustration to make the diagnostic reasoning visible; they are not measured from a real deployment.

A team builds a classifier that routes support tickets into five queues: Billing, Technical, Account, Refunds, Other. They report:

• Validation accuracy: 94.2%
• Production accuracy over the first week, measured against human re-routing: 71.8%
• Production accuracy in month six: 63.4%

Three numbers, and they encode two separate defects. Work it in order.

Step 1 — the 94.2 → 71.8 drop at launch is a validation-to-production gap, which is the leakage signature. Not overfitting: overfitting would have shown as a train-vs-validation gap during development, and their training accuracy was 95.1%, only 0.9 points above validation. So we look for information present in the training features and absent at prediction time.

They find it. The feature set includes resolution_team, which is populated when the ticket is closed. For every training row it was filled in; for a brand-new ticket it is null. The model had learned to read the answer off the answer key. This is target leakage, mechanism row one from section 2.

Step 2 — quantify. Retraining without resolution_team gives validation accuracy of 74.6%, which is 19.6 points lower and roughly consistent with the 71.8% observed in production. The leak accounted for essentially the entire gap. Notice what happened to the project: its real accuracy was always about 74%, and four months of work were spent celebrating a number that measured nothing.

Step 3 — the 71.8 → 63.4 decay over six months is a different defect. No code changed and no data changed in the model. This is the drift signature. They check the input distribution and find the share of tickets mentioning a product launched in month three has gone from 0% to 22%. Covariate drift — a new input population the training data has no examples of.

Step 4 — a third defect hiding in the aggregate. Per-class recall on the cleaned model:

Refunds is 2% of volume and recall is 0.11 — the model almost never predicts it. Aggregate accuracy barely notices, because getting all 2% wrong costs two points. But refunds are the tickets with a regulatory clock on them. This is class imbalance manifesting as a metric failure, and the fix is not a resampling trick first — it is reporting per-class recall instead of aggregate accuracy, so the failure is visible at all. Then stratified splitting, then class weighting.

Step 5 — check for the fourth. They sample 200 tickets and have two senior agents re-label them independently. The agents agree with each other on 176 (88%) and with the stored label on 181 (90.5%). So roughly 9–10% label noise, concentrated in the Account/Other boundary — a genuinely ambiguous distinction that the guidelines never defined. This caps achievable accuracy near 90% no matter what model they use, and it means the difference between a 74% model and a 78% model is partly unmeasurable with this test set.

The finding, stated as a curator would state it: one dataset, four defects. Leakage cost 20 points of illusory accuracy, drift cost 8 points of decay, imbalance hid a compliance-critical failure entirely, and label noise put a ceiling on everything. None of the four would have been found by looking at aggregate accuracy, and three of the four were found by comparing two numbers that a single metric collapses into one.

Every fix in this area has a failure mode, and the exam likes the fixes that are correct in one context and wrong in another.

The two rows worth memorising as a pair: stratify, don't randomise, when a minority class matters; and resample only the training split. Both are stated in 09-08, and both appear as distractors in the inverted form ("balance the dataset, then split") that quietly guarantees contamination.

This is a heavily represented area with an unusual amount of blueprint support behind it.

Blueprint coverage. The Data Analysis and Visualization domain is 14% of the exam, roughly 8 of 60 questions [OFFICIAL], and data quality touches three of its five objectives: 2.1 (extracting insights from large datasets), 2.3 (conducting data analysis under supervision), and 2.5 (identifying relationships and trends or any factors that could affect the results of research). That last clause is nearly a direct description of this lesson — "factors that could affect the results" is the blueprint's phrase for data defects. Objective 1.2 in the Core ML domain restates 2.1 verbatim, which is the duplicate pair noted in the opening. And 2.2 / 3.2, comparing models with statistical metrics, is where imbalance bites, because a metric chosen without regard to imbalance makes model comparison meaningless.

Calibration. The [FIELD] priority tiers place data-quality handling at Tier 2 — above visualization and feature engineering, below tokenization and NIM [FIELD]. Treat that as a genuine study-weight signal: this material is worth real drilling, and the drilling should be recognition speed on symptom→defect mapping rather than depth on remediation algorithms.

Cross-domain reach. Leakage also appears in the Core ML domain's cross-validation objective (09-08) and in the Experimentation domain's splits-and-holdout material (10-01). It is one of a small number of concepts the blueprint reaches for from three directions, and concepts with that property are over-represented in question banks relative to their nominal weight.

Question phrasings to expect

• "A model achieves 98% accuracy on the test set but performs poorly in production. What is the most likely cause?" → data leakage.
• "A team fits a StandardScaler on the entire dataset and then splits into train and test. What problem has been introduced?" → preprocessing leakage (information from the test set has entered the transform).
• "A fraud model reports 99.2% accuracy on a dataset where 0.8% of transactions are fraudulent. What should be reported instead?" → precision, recall, F1, confusion matrix — the imbalance-metric answer.
• "A deployed model's performance has degraded gradually over eight months with no changes to the code. What is happening?" → drift (covariate or concept).
• "Which of the following is a data-quality defect: missing values, duplicates, outliers, format inconsistencies, or all of these?" → the canonical list is being tested; all of these, plus leakage.
• "Two annotators assign different labels to identical inputs. What is the resulting problem called?" → label noise, measured by inter-annotator agreement.
• "Which step prevents the minority class from being absent from a validation fold?" → stratified splitting.

Distractor families

They produce similar-looking disappointment through opposite mechanisms, and the pair of numbers you compare tells them apart. Overfitting is a model memorising its training data: training performance is high, validation performance is lower, and the gap is visible before deployment. Its fixes are regularisation, more data, simpler models, and early stopping. Leakage is a dataset defect: training and validation performance are both high because both contain the illegitimate information, and the collapse appears only when the model meets real inference-time data. No amount of regularisation fixes it — the model is learning a real pattern that happens not to exist at prediction time. If an exam stem gives you a train/validation gap, answer overfitting; if it gives you a validation/production gap, answer leakage.

Four checks, cheapest first. One: the availability audit. For each feature, ask whether its value exists, and is final, at the moment of prediction. Anything derived from an outcome, a resolution, a close date, or a downstream process fails. Two: the suspiciously-good check. A single feature with implausibly high individual predictive power is the classic signature; a feature-importance ranking dominated by one column deserves suspicion rather than celebration. Three: split hygiene. Confirm transforms are fitted inside the training split, confirm the split is temporal if the data is temporal, and confirm grouped records (same user, same document, same near-duplicate) do not straddle the boundary. Four: the private holdout. Score against data collected after the training cutoff. Temporal separation is the only leakage test that catches mechanisms you did not think to look for, which is why it is worth the cost.

No, and treating it as one is a common over-correction. Imbalance is only a problem when it prevents you from measuring or achieving what you care about. If you care about the minority class — fraud, defects, rare intents, safety violations — imbalance is severe and the first fix is always the metric, not the data: precision, recall, F1, PR curves, and the confusion matrix per 09-05. If both classes matter proportionally and the model performs adequately on each, the imbalance is simply a fact about your domain and needs nothing. What is never optional is stratification — ensuring the minority class is present in every split and fold — because a validation fold containing zero positive examples gives you a number with no meaning. And resampling, when you do it, applies to the training split only, forever.

Because the other three are visible in data you already hold, and drift is by definition about data you do not hold yet. Label noise, leakage, and imbalance can all be found by inspecting the frozen dataset. Drift can only be found by comparing production traffic against the training distribution over time — which requires that you were logging distributions before you needed them. Worse, the most damaging variety, concept drift, changes the relationship between input and output rather than the inputs themselves, so input monitoring will not see it; only fresh labels will, and fresh labels are exactly what production systems lack. The practical posture: monitor input distributions continuously because it is cheap and catches covariate drift, sample and label a small stream of production data continuously because it is the only concept-drift detector, and treat "when do we retrain?" as a scheduled decision rather than an incident response. 12-14 covers the monitoring apparatus.

It matters differently, and in some respects more. A fine-tuned model averages over its training set, so a small fraction of bad examples degrades it gradually. A RAG system retrieves specific documents and puts them verbatim into the context window, so one bad document can produce one confidently wrong answer with a citation attached — arguably a worse failure than a fuzzy one, because provenance makes it credible. The specific RAG-side quality defects are duplicated and boilerplate content dominating retrieval (06-04), stale documents that contradict current ones with no recency signal (07-03), broken chunk boundaries from failed parsing (06-01), and documents a given user should not be allowed to see (07-05). The consolation is symmetric: a bad document in an index can be deleted this afternoon, whereas a bad example baked into weights requires retraining and cannot be audited afterwards.

1. Four dataset-level pathologies, four signature symptoms. Label noise = plateau with arbitrary errors. Leakage = validation-to-production collapse. Imbalance = high accuracy, minority class never predicted. Drift = decay over time with no code change.
2. The time signature separates leakage from drift. Wrong from the first production request → leakage. Right at launch, wrong later → drift.
3. The pair of numbers separates leakage from overfitting. Train-vs-validation gap → overfitting. Validation-vs-production gap → leakage.
4. Learn the canonical record-level list verbatim: missing values, duplicates, outliers, format inconsistencies, data leakage [OFFICIAL]. Leakage is the member that also belongs to the dataset-level list.
5. Leakage has five mechanisms, and preprocessing leakage — fitting a transform before splitting — is the one competent teams still commit.
6. Imbalance is first a metric problem. Change to precision/recall/F1 and the confusion matrix before touching the data; stratify every split; resample the training split only.
7. Sampling bias is not imbalance. Imbalance reflects the world faithfully and has analytic fixes; bias reflects your pipeline and can only be fixed at collection.
8. Concept drift is the hard one because detecting it needs labels that production does not supply. Monitor inputs continuously, sample and label continuously.
9. In RAG, one bad document produces one confident wrong answer with a citation — a sharper failure than a fine-tune's fuzzy degradation, but a far cheaper one to fix.

You can now name every defect. Naming is not detecting — detection requires actually looking at the corpus, and looking at a text corpus is its own discipline with its own instruments and its own trap.

Next: 08-03 runs exploratory data analysis on a text corpus: what to profile, in what order, and why length distributions must be measured in tokens rather than characters, since the character count of a document tells you almost nothing about whether it fits a context window or what it will cost. It is also where the histograms and box plots arrive that 08-04 then teaches you to choose between.