Bias in AI: sources, measurement, and mitigation

Three senses of the word "bias" circulate in machine learning, and questions exploit the ambiguity.

When an exam question is in the Trustworthy AI domain, "bias" means the third sense. When it is in a core ML domain, it usually means the first. The context disambiguates; the words do not.

Within the third sense, be precise about what is unequal, because the mitigation differs:

• Performance bias — the system simply works worse for a group. Accuracy, retrieval quality, refusal rate, or latency differs by slice. This is the most common, most measurable, and most examinable kind.
• Representational bias — the system's outputs depict a group in a stereotyped, diminished, or erased way even when "accuracy" is not at issue. Ask for "a picture of a nurse" or "a story about a scientist" and see what comes back.
• Allocative bias — the system's outputs drive a decision that distributes something valuable (a loan, a job interview, a benefit, a level of scrutiny) unequally. Highest stakes; the domain where a human decision maker acting on model output is part of the harm.

And be precise about the standard: NVIDIA's pillar says minimize, and the exam's objective says minimize. Not eliminate. A model trained on human-generated data reflects a world that is not uniform, and any option promising the elimination of bias, or a guarantee of fairness, is a distractor. What you can promise is a process: enumerate the slices, measure them, document the gaps, gate the release, and monitor for drift.

Five channels. Learn them as a sequence along the pipeline, because that ordering is also the order in which you can intervene.

L1 — The intuition: bias is inherited, then amplified

A model does not invent bias out of nothing. It inherits it from the data, from the people who labelled the data, from the choices about whose data got collected, and from the features it was handed — and then, once deployed, it can amplify what it inherited by feeding its own outputs back into its next training set. Inheritance then amplification. That two-step is the shape of the whole problem.

L2 — The mechanism: the five named channels

Channel 1 — Collection bias. The data reflects who was easy to observe, not who will be affected. A support-ticket corpus contains the language of customers who write tickets, in the channels you logged, in the languages your product shipped in. A medical dataset from three urban teaching hospitals is not a national population. The tell in a question stem: the data came from a source that systematically over-represents some people.

Web-scale pretraining data is a special case worth naming explicitly. It over-represents the languages, dialects, registers, and viewpoints that are abundantly published online — a very specific slice of humanity — and it carries the stereotypes present in that text. This is a fact about every general-purpose LLM you will use, including well-aligned ones, and it is the reason your own evaluation cannot be skipped on the grounds that the vendor aligned the model.

Channel 2 — Labelling bias. Two annotators, one ambiguous example, two different labels. Scaled up, systematic annotator disagreement becomes systematic label skew, and the model learns the skew as if it were signal. Common forms: annotators unfamiliar with a dialect rating it lower for "professionalism"; toxicity annotators marking reclaimed in-group language as slurs; guidelines that never anticipated a case, leaving each annotator to improvise consistently in their own direction. The instruments are the ones 09-03 names — written guidelines, inter-annotator agreement, adjudication of disagreements, and an annotator pool diverse enough that no single perspective silently becomes ground truth. If your labels are biased, your metrics are biased in the same direction, and your evaluation will certify the bias as correct. That is the nastiest property in this entire lesson.

Channel 3 — Sampling bias. You had reasonable data and then chose a subset badly. Deduplication that removes non-English near-duplicates preferentially. Quality filtering whose "quality" classifier was itself trained on a narrow register. Class balancing that discards minority-group examples because they were minority-class examples too. Downsampling for cost that happened to hit a region hardest. This channel is distinct from collection because the world gave you the data and your pipeline threw it away.

Channel 4 — Proxy features. A feature that is not the protected attribute but correlates strongly with it. Postal code proxies for ethnicity and income in many places. Name proxies for gender and ethnicity. Alma mater, employment gaps, device type, browsing hour, writing style — all can carry group membership. Remove the protected attribute and the model reconstructs it from the proxies, which is why "we don't collect that attribute" is not a fairness control. It is worse than not a control: without the attribute you cannot even measure per-slice performance, so you have blinded your audit while leaving the mechanism intact. This is the single most counter-intuitive item in the lesson and a frequent exam target.

For LLM and RAG systems, the proxies are linguistic. Dialect, code-switching, non-native syntax, transliterated names, and formality register all correlate with group membership, and all affect embedding similarity — which means they affect retrieval, which means they affect answer quality before generation is even involved. That mechanism deserves its own mention because newcomers assume bias in an LLM app lives in the model. Often it lives in the retriever.

Channel 5 — Feedback loops. The system's own outputs become its next inputs. A recommender surfaces what it already favours, users can only click what was surfaced, clicks become training data, and the favouritism hardens. An LLM assistant whose accepted answers are logged as preference data learns to produce more of what the users who accept answers prefer — and those users are a slice, not a population. A fraud model that flags a group more often generates more investigations of that group, more confirmed cases in that group, and a next-generation model more confident that the group is fraudulent. Feedback loops are the amplification step, they are invisible to a single-snapshot evaluation, and they are the reason bias monitoring must be continuous rather than a launch-time checkbox. The break is structural: hold out human-labelled ground truth collected independently of the system's own suggestions, and re-measure slices on a schedule.

L3 — Why aggregate metrics are structurally blind

This is the mechanical heart of the lesson, and it is worth working the arithmetic so it stops being a slogan.

Take a constructed example — invented numbers, chosen to make the arithmetic clean.

Constructed illustration (numbers invented for teaching)

Evaluation set: 1,000 items
Slice A: 900 items, accuracy 95%  →  855 correct
Slice B: 100 items, accuracy 60%  →   60 correct
                                     ──────────
Aggregate:                            915 / 1,000 = 91.5%

Now improve slice A by one point and worsen slice B by five:
Slice A: 900 items, accuracy 96%  →  864 correct
Slice B: 100 items, accuracy 55%  →   55 correct
                                     ──────────
Aggregate:                            919 / 1,000 = 91.9%

The aggregate went UP. The disparity got WORSE.

Read the last two lines again. A release that made the system measurably worse for slice B passed an aggregate quality gate, and passed it with an improvement. There is nothing pathological about the numbers; this is just what a weighted average does when one weight is small. And the weights are small precisely for the groups you most need to check, because minority representation in your evaluation set mirrors minority representation in your data.

Three consequences follow.

Consequence one: the slice must be large enough to measure. With 100 items at 60% accuracy, your uncertainty is wide, and a five-point move may be noise. Per-slice evaluation therefore requires deliberately over-sampling small slices in the evaluation set — not in proportion to the population, but to a floor that gives each slice usable statistical power. The sample-size reasoning from 09-09 applies per slice, and it is the reason a fairness evaluation set is built differently from a general one.

Consequence two: you must choose the slices before you look. Slicing after the fact, until a gap appears, is the multiple-comparisons problem in fairness clothing: with enough arbitrary slices, one will look bad by chance. Enumerate the slices that matter for your application, in writing, in advance, and report all of them every time — including the ones that look fine.

Consequence three: you need the attribute to slice on. Which loops back to proxy features. Measuring per-slice performance requires knowing, for each evaluation item, which slice it belongs to. That is a data-collection decision made long before evaluation, and it is why "we don't collect that" forecloses the audit. Note the genuine tension here — collecting sensitive attributes to check for discrimination is itself a privacy exposure, and this is one of the real conflicts between two of NVIDIA's four pillars. The standard resolutions are to collect the attribute only for a consented evaluation sample rather than for all production traffic, to keep it in a separately governed store used only for fairness auditing, or to use a labelled benchmark set constructed for the purpose. Being able to name the tension is worth more on an exam than pretending it does not exist.

The release gate is where per-slice evaluation becomes a control rather than a report. Two gate designs, both defensible:

• Floor gate: no slice may fall below an absolute threshold. Simple; blind to disparity above the floor.
• Gap gate: no slice may fall more than N points below the best slice, or below the aggregate. Directly targets disparity; can be gamed by making the best slice worse, which is why you keep the aggregate gate too.

Run both. Write the numbers down. That written gate, with its per-slice table and its pass/fail history, is your Nondiscrimination artifact — the thing 13-01 says every pillar needs.

A constructed scenario. All numbers are invented for teaching.

The system. A company builds an LLM assistant that reads a job application and produces a structured summary plus a shortlist recommendation (advance / hold / decline) for a human recruiter. It retrieves the job's requirements from an internal job-description store and the company's own hiring criteria. Training for the recommendation component came from three years of historical recruiter decisions.

The launch evaluation. 1,200 held-out applications, agreement with a panel of senior recruiters as the metric. Aggregate agreement: 88%. The team ships.

The complaint. Four months later a hiring manager notices that candidates whose applications were written in a non-native English register almost never reach advance. Nobody had measured it. Here is the audit, step by step, and it is a template you can reuse.

Step 1 — Enumerate slices, in writing, before looking at any results. For this application, defensible slices include: years of experience band; whether the degree is from a domestic or international institution; whether the application text shows non-native English markers; presence of an employment gap; career-changer versus linear path; application channel. Note that none of these is a protected attribute directly. Several are proxies for one, which is exactly why they are the right slices — they are the channels through which disparate treatment would actually flow in this system.

Step 2 — Build enough evaluation data per slice. The original 1,200 items contained 41 international-institution applications. Forty-one items cannot support a claim. The team samples the historical pool to reach a floor of 150 items per slice, deliberately over-sampling small slices, and has the senior panel re-label them so the ground truth is independent of the model.

Step 3 — Measure per slice. The constructed result:

The 88% was real. It was also a weighted average dominated by the largest, best-served slice. Two of the five slices sit fourteen and nineteen points below the best, and the advance rate for the worst slice is roughly a third of the best slice's — a disparity the aggregate agreement number cannot express at all.

Step 4 — Attribute the gap to a channel. This is the step teams skip, and it is the step that determines the fix. The team investigates and finds three distinct contributions:

• Labelling / historical bias. The training signal was three years of human recruiter decisions, which encoded the recruiters' own patterns. The model learned to agree with history, and history was not neutral. Channel: labelling bias (and historical bias in the ground truth itself).
• Retrieval bias. Applications in a non-native register retrieved the relevant job-requirement passages less reliably, because their phrasing was further in embedding space from the internally-written job descriptions. Worse retrieval, thinner context, weaker summary, more conservative recommendation. Channel: proxy features expressed linguistically, biting at the retriever — the mechanism 07-03 describes as a limit of embedding search, showing up here as a fairness defect.
• Feedback loop. Recruiters accepted the assistant's decline recommendations more often than they overrode them, and those accepted decisions were being logged for the next training round. Channel: feedback loop, ready to harden the gap.

Three channels, three different fixes. If the team had assumed "the model is biased" and reached for a single remedy, two thirds of the gap would have survived.

Step 5 — Mitigate per channel.

• Against the labelling channel: stop training on raw historical decisions. Re-label a curated set against written criteria by a diverse panel, with inter-annotator agreement measured and disagreements adjudicated. Where history cannot be repaired, remove the recommendation component entirely and ship only the structured summary — a legitimate and often correct answer.
• Against the retrieval channel: normalize the query representation, add hybrid retrieval so lexical matching backstops embedding similarity (07-06), and add a query-rewriting step. Then re-measure retrieval quality per slice, because retrieval has its own per-slice metrics and they are more diagnostic than end-to-end ones.
• Against the feedback loop: stop logging accepted recommendations as training labels. Collect an independent, human-labelled ground-truth stream on a sample, unconnected to what the assistant suggested.

Step 6 — Gate and monitor. A gap gate: no slice may fall more than five points below the best slice on agreement, and no slice's advance rate may fall below half the best slice's. The per-slice table runs in CI on every release and on a monthly canary. The table itself, with its history, becomes the artifact.

Step 7 — Consider whether the system should exist in this form. The most honest mitigation available in an allocative context is to narrow what the system decides. A summary that helps a human read faster is a very different risk object from a recommendation that shapes the human's decision. "Reduce the scope of automation" is a real answer, and on an exam it is sometimes the keyed one when the stem stresses high consequence.

Read the described symptom, name the channel, name the mitigation, and note which NVIDIA-named technology is in play where one is.

The two rows to memorize for tool-mapping questions: TAO Toolkit is NVIDIA's named tool for curating and validating unbiased datasets, and NeMo Curator is the named tool for large-scale data curation, deduplication, and quality filtering. If a question describes validating that a dataset is not skewed, TAO is the expected answer. If it describes cleaning and filtering a huge corpus, Curator is. Neither is NeMo Guardrails — Guardrails acts at request time on behaviour, not on datasets.

Objective 5.4 — "Describe how to minimize bias in AI systems" — is one of four objectives in a domain worth 10% of the exam, and it is the objective with the clearest right-and-wrong answers, because the correct mitigations are named techniques rather than judgement calls. It also connects outward: bias is the harm behind the Nondiscrimination pillar (13-01), it is a data-quality topic that overlaps the Data Analysis domain's "factors that could affect the results of research", and it is an evaluation topic that overlaps everything Module 9 built.

Question phrasings to expect:

1. "A model achieves high overall accuracy but performs poorly for one demographic group. What evaluation practice would have detected this?" — per-slice / subgroup evaluation. The single most likely bias question on the exam.
2. "Which of the following is a source of bias in machine learning datasets?" — recall from the five channels; expect sampling, labelling, or collection as the key.
3. "An organisation removes gender from its feature set. Has it eliminated gender bias?" — no; proxy features reconstruct it, and removing the attribute also removes your ability to measure the disparity.
4. "Which NVIDIA tool supports curating and validating unbiased datasets?" — TAO Toolkit.
5. "A recommendation system increasingly surfaces the same narrow set of items. What phenomenon is this?" — a feedback loop.
6. "Which trustworthy AI principle does unequal model performance across groups violate?" — Nondiscrimination.
7. "What is the most effective way to reduce bias originating in the training data?" — representative and balanced data collection and curation, not post-hoc output filtering.

Distractor families:

Two calibration notes. First, this exam sits at general depth — you will be asked which practice detects bias, not to compute a specific formal fairness criterion. It is worth knowing that multiple formal fairness definitions exist (equalizing error rates across groups, equalizing selection rates, calibrating within groups) and that they are mathematically in tension so you generally cannot satisfy all of them at once; that fact is examinable at recognition depth as "there is no single fairness metric," but the arithmetic of the impossibility is not. Second, when two options are defensible, the NVIDIA-named one tends to be keyed — which for this objective means TAO Toolkit and NeMo Curator are worth having on instant recall.

No, and the exam's own vocabulary tells you so: the Nondiscrimination pillar says minimize and objective 5.4 says minimize. Three reasons the stronger claim is unavailable.

The data reflects a world that is not uniform. Any corpus of human-produced text carries the distribution of who writes, about what, in which language, with which assumptions. You can rebalance, curate, and filter; you cannot render it neutral, because "neutral" is not a defined point.

Fairness criteria conflict. Equalizing error rates across groups, equalizing selection rates, and calibrating predictions within each group are different requirements, and in realistic conditions they cannot all hold simultaneously. Choosing which to satisfy is a value judgement about the application, not a computation. This is why a system's fairness claim must state which criterion it met.

The pipeline has more than one bias channel. Fix the data and the labels may still skew; fix the labels and the retriever may still skew; fix everything at launch and the feedback loop will re-introduce skew over months.

So what does a defensible answer sound like? "We enumerated eight slices in advance, over-sampled the small ones to 150 items each, measured agreement and outcome rates per slice, closed the largest gap from nineteen points to four by fixing retrieval and re-labelling the training set, gate every release on the worst slice, re-run monthly, and document the residual four-point gap and its known cause in the model card." Every clause is an artifact. That is minimization, evidenced — and it is precisely what the "describe how to minimize bias" objective is asking you to be able to say.

Subgroup evaluation — per-slice evaluation — is reporting your quality metrics separately for each identified group rather than only over the whole set. Building one is five decisions:

Decide the slices. Enumerate the groups whose experience of the system could plausibly differ, in writing, before measuring. Include direct attributes where you legitimately have them, and include the proxy-bearing features through which disparity actually flows in your system: language, dialect, register, region, device, channel, tenure, document type. For a RAG system, add source-language and document-format slices, because those drive retrieval quality.

Decide the metrics. Reuse the metrics you already chose in 09-05, and add the two that expose disparity specifically: outcome rate per slice (how often the system says yes, advances, approves, or answers at all) and refusal or abstention rate per slice. A model that quietly refuses more often for one group is discriminating even if its accuracy-when-it-answers is identical — and accuracy-only reporting misses it entirely.

Size each slice for power. Not population-proportional. Set a floor per slice sufficient to distinguish a gap you would care about from noise. This will make your evaluation set unrepresentative of production traffic by design, which is correct — its job is detection, not simulation.

Get independent ground truth. Labels for the fairness evaluation set should not come from the system's own outputs, from the historical decisions the model was trained on, or from a single annotator. Otherwise you measure agreement with the bias.

Attach a gate and a schedule. A per-slice table nobody can fail is a report. A per-slice table that fails the build is a control. Put it in CI alongside the regression suite from 10-04, and re-run it on a canary schedule as 12-14 describes, because bias appears through drift as well as at launch.

The output is a table of slices by metrics, with gaps, a pass/fail verdict, and a history. That table is the deliverable. It is what you show an auditor, what you paste into a model card, and what you point at when someone asks whether you checked.

Four, with distinct jobs, and the exam expects you to keep them apart:

• TAO Toolkit — NVIDIA's named tool for curating and validating unbiased datasets. When a question describes checking or building a dataset that is not skewed, this is the expected answer.
• NeMo Curator — large-scale data curation: cleaning, deduplication, and quality filtering of LLM datasets. Its bias relevance is twofold: it is how you build a balanced corpus at scale, and it is where a sampling bias can be introduced if you never audit per-slice what your filters removed.
• NeMo Guardrails — request-time rails. It does nothing about a performance gap, but it can suppress overt representational harm in output and, critically, it logs what it suppressed. Behaviour, not data.
• Model Card Generator — automated model cards, which is where the per-slice results, the known gaps, the intended use, and the limitations get documented. Bias you have measured but not closed must be disclosed, and this is the artifact that does it.

Around those sit the non-product practices the same objective covers: representative and balanced data collection, annotation guidelines with inter-annotator agreement, proxy audits, per-slice evaluation with release gates, red-teaming for harms nobody enumerated, human-in-the-loop review on consequential decisions, and continuous monitoring. Objective 5.3 says NVIDIA and other technologies — the "other" is doing real work in that sentence, and the process controls are as examinable as the product names.

• Harmful (societal) bias — systematic performance or treatment differences that disadvantage identifiable groups; distinct from statistical bias and from the bias parameter.
• Performance bias — the system works measurably worse for a group.
• Representational bias — outputs depict a group in a stereotyped, diminished, or erased way.
• Allocative bias — outputs drive unequal distribution of something valuable.
• Collection bias — the data reflects who was easy to observe rather than who will be affected.
• Labelling bias — systematic annotator skew becomes learned signal; also makes your metrics agree with the bias.
• Sampling bias — pipeline choices (deduplication, filtering, balancing, downsampling) discard some groups preferentially.
• Proxy feature — a non-protected feature that correlates with a protected attribute strongly enough to substitute for it.
• Feedback loop — the system's outputs become its next training inputs, hardening existing skew.
• Per-slice (subgroup) evaluation — reporting metrics separately per identified group; the only way performance bias becomes visible.
• Outcome rate / refusal rate — how often the system says yes, or declines, per slice; exposes disparity that accuracy alone hides.
• Floor gate / gap gate — release gates on the worst slice's absolute score, and on its distance from the best slice.
• Statistical power floor — the minimum slice size needed to distinguish a gap you care about from noise.
• Fairness audit — a documented assessment of slices, metrics, gaps, causes, mitigations, and residual risk.
• TAO Toolkit — NVIDIA tooling for curating and validating unbiased datasets.
• NeMo Curator — NVIDIA tooling for large-scale dataset curation, deduplication, and quality filtering.

1. Five channels, by name: collection, labelling, sampling, proxy features, feedback loops. Inheritance then amplification.
2. Aggregate metrics are structurally blind. The worked arithmetic shows an aggregate rising while a slice falls. Per-slice evaluation is the only detector.
3. Removing the protected attribute is not a fairness control. Proxies rebuild it, and you lose your ability to measure.
4. Slices must be enumerated in advance and sized for power, over-sampling small groups deliberately.
5. Measure outcome and refusal rates per slice, not accuracy alone. Quiet disparate refusal is real discrimination.
6. Biased labels make biased metrics. Inter-annotator agreement and a diverse panel are fairness instruments, not just quality instruments.
7. In a RAG system, check the retriever per slice. Disparate retrieval quality precedes and causes disparate answers.
8. Break output-to-label feedback loops and keep an independent human-labelled ground-truth stream.
9. Minimize, never eliminate. Fairness criteria conflict; any option promising elimination or a single fairness score is wrong.
10. Tool map: TAO Toolkit validates unbiased datasets · NeMo Curator curates corpora at scale · NeMo Guardrails constrains runtime behaviour · Model Card Generator discloses the residual gap.
11. A gate, a schedule, and a table make it a control. Without those it is a report, and reports do not stop releases.

Bias is measured on slices of people. The next harm is about individual people — one person's data, one person's consent, and one person's request to have it deleted. That request has a straightforward answer if their data is a row in an index, and a genuinely uncomfortable one if their data was used to train a model, because the parameters do not contain retrievable records to remove.

Next: 13-05 takes up data privacy, consent, and why model weights cannot forget — informed consent and purpose limitation, PII handling and de-identification, retention and the right to withdraw, and the strongest practical argument in this course for retrieval over fine-tuning on sensitive data.