How to Curate a Dataset for an LLM Task (Including the Unanswerable Cases)

Dataset curation is the process of assembling a dataset to match a written task definition, by making explicit decisions about scope, sourcing, filtering, and coverage, and recording those decisions so the dataset can be audited, versioned, and rebuilt. It is the opposite of data collection. Collection asks what can I get? Curation asks what should be in here, and how will I know when it is wrong?

The distinction matters because LLM work has three different dataset shapes and people use the word "dataset" for all three:

They are curated by different rules. A corpus wants breadth, freshness, and deduplication. An SFT set wants label consistency and format uniformity. An evaluation set wants difficulty and representativeness, and above all it wants to be frozen. Confusing the three is a genuine error in production and a genuine distractor family on the exam: an option that proposes fine-tuning on the same examples you evaluate on is describing leakage, which lesson 08-02 names properly.

Curation produces four artefacts, and if you cannot point at all four you have collected rather than curated:

1. A task definition — one paragraph naming the input, the output, the user, and the failure that matters most.
2. A source manifest — where every record came from, with its licence and its collection date.
3. A filter log — what you removed and why, with counts.
4. A coverage matrix — the categories of input the dataset intentionally contains, including the ones that should fail.

L1 — Intuition: curation is a specification, not a scrape

Think of a dataset as the executable form of a specification. When a product manager says "the assistant should answer HR policy questions," that sentence is ambiguous in about nine ways: which policies, which countries, which effective dates, what happens when the policy changed last month, what happens when the question is about someone else's salary. Curation is where each ambiguity gets resolved, because you cannot put a record into a dataset without having implicitly decided all of them.

This is why curation is upstream of everything. A model cannot be better than its data's specification, and a metric cannot be more meaningful than the set it is computed over. If your evaluation set contains only questions your corpus answers well, your faithfulness score is measuring your curation optimism, not your system.

L2 — The six stages of a curation pipeline

Stage 1: Define the task in writing. One paragraph. Input, output, audience, hardest failure. If you cannot write it, you cannot curate for it — and the honest response is to go find out, not to start collecting.

Stage 2: Enumerate the intent taxonomy. List the categories of request the system will receive. Do this before you look at data, then revise it after you look at real logs, because your prior is always wrong in an interesting way. The taxonomy is what makes coverage checkable: with a taxonomy you can count how many records you have per category and see the empty cells. Without one, "we have 5,000 examples" is a number about volume, not about coverage.

Stage 3: Source. Real user logs beat synthetic questions beat questions you invented at your desk, in that order, for representativeness. Real logs also carry the highest privacy burden — lesson 13-05 covers consent and why model weights cannot forget — so sourcing is where legal review belongs, not after annotation.

Stage 4: Filter. Four filters, in this order, because the cheap ones should run first:

Deduplication before quality filtering is a real ordering rule, not stylistic preference. Lesson 06-04 covers why a repeated footer can become the nearest neighbour of every query in a RAG index; the curation-side version of the same point is that duplicates distort every statistic you compute afterwards, including the length distributions you will measure in lesson 08-03.

Stage 5: Cover the gaps — including the unanswerable cases. This is stage five and it is the reason this lesson exists. See section 3.

Stage 6: Version and freeze. A dataset without a version is not a dataset, it is a mood. Assign a version string, record the row count, record the filter counts, and never edit in place. When you later find that model B beat model A, you need to be able to prove they were scored on identical items — which is the whole basis of the comparison discipline in lesson 09-01.

L3 — The coverage matrix and the four kinds of unanswerable

Coverage is the property that a dataset contains examples from every category of input the system will actually see, in proportions you have chosen on purpose. The instrument is a matrix: intent categories down the side, difficulty or variant across the top, counts in the cells. Empty cells are the finding.

Now the part nobody includes. There are four distinct kinds of case that should be in the dataset and produce something other than a confident answer, and they fail differently:

Each row is a different mechanism and needs its own records. A dataset with ten "out of corpus" cases and zero "not permitted" cases is still blind to a category of harm. And note the interaction with retrieval: an unanswerable-case set is the only way to measure whether your system's refusal behaviour survives contact with a retriever that always returns something, because a vector index has no concept of "no results" — it returns the nearest neighbours regardless of whether they are relevant. That property is discussed in 07-03.

The proportion question — how many unanswerable cases? — has no sourced universal answer, and you should distrust anyone who quotes one. The defensible framing is a ratio you choose from your own traffic: if 15% of real user requests are out of scope, an evaluation set with 0% out-of-scope items overstates your system's quality by construction. Set the proportion from observed traffic where you have it, and state the assumption where you do not.

This is the comparison table to memorise, because the exam's data-domain questions frequently hinge on which activity a described scenario is actually describing.

Four notes on the confusables in that table.

Curation contains cleaning; it is not the same as cleaning. Cleaning fixes records. Curation decides which records exist. You can clean a badly scoped dataset to perfection and still have the wrong dataset.

Annotation is downstream of curation and has its own quality regime. Two annotators disagreeing is not a data-cleaning problem, it is an inter-annotator agreement problem, and 09-03 handles it. Label noise arriving from inconsistent annotation is a data-quality defect, which is 08-02.

Augmentation is an accuracy technique, and the exam names it as one. "Use data augmentation to improve model accuracy" is an explicit objective in the official study guide's course-objective list [OFFICIAL], so if a stem asks how to improve accuracy when you cannot collect more labeled data, augmentation is a keyed-answer candidate. Its risks belong in the same breath: augmenting before splitting leaks — a paraphrase of a training item landing in the test set is contamination even though the strings differ — and augmenting text can silently invert labels, since swapping a word for a WordNet synonym can flip the sentiment you were trying to preserve.

Embedding is a curation-adjacent step the blueprint binds to curation. Objective 1.4 says "curate and embed content datasets for RAGs" in one breath, which is why embedding-model choice (03-03) and index freshness (12-12) are part of the same responsibility rather than a separate specialism. Re-embedding the whole corpus when you change embedding models is a curation consequence, not an infrastructure surprise.

Where NeMo Curator sits

NeMo Curator is NVIDIA's data-curation component within the NeMo framework [NVIDIA-DOC] — the stack entry whose job is preparing and curating datasets for LLM training, covering the pipeline stages this lesson describes at corpus scale: downloading and extracting text, language identification, cleaning and unicode fixing, quality filtering, exact and fuzzy deduplication, and PII handling. For the exam, the identity claim is what matters and it is a single sentence: when a question describes preparing or curating a large training dataset and one option names an NVIDIA tool, that tool is NeMo Curator, not NeMo Guardrails (safety rails), not NeMo Retriever (retrieval accuracy), and not TAO Toolkit — although TAO also appears in NVIDIA material in a dataset curation-and-validation role, particularly around producing unbiased datasets, so both can be defensible depending on the stem's framing. Where two NVIDIA tools are both plausible, read the stem for whether it is about text corpora for LLMs (Curator) or curating and validating datasets to reduce bias (TAO is named this way in trustworthy-AI material). Do not memorise a throughput figure for either; no sourced number appears in the material this course is built from, and inventing one is how a confident wrong answer gets learned.

Numbers in this example are constructed for illustration, not measured from a real system. They are chosen to make the arithmetic of coverage visible.

The task definition (stage 1). "Answer employee questions about UK and US HR policy from the published 2024–2025 handbook. Users are non-technical employees. The failure that matters most is stating a policy that does not exist, because an employee may act on it."

That last clause is doing real work. It tells you the metric that matters is groundedness rather than fluency, and it tells you the dataset must contain cases where the correct answer is "the handbook does not cover that."

The intent taxonomy (stage 2). Six categories, drawn from three months of help-desk tickets: leave and holiday, expenses, benefits enrolment, remote-work rules, disciplinary process, payroll dates.

Sourcing (stage 3). 1,400 real help-desk questions exported from the ticket system, de-identified. 60 questions written by two HR partners to cover categories the tickets barely touched.

Filtering (stage 4), with counts:

Look at the near-duplicate row: 402 of 1,460 records, 27.5% of the raw set, were the same handful of questions asked again and again. That is the number that changes decisions. Had we skipped deduplication, "how many holidays do I get" and its 40 variants would have dominated every distribution we later measured, and an evaluation set sampled from this pool would have spent a third of its items re-testing one question. Deduplication is not tidiness; it is the difference between a dataset and a popularity contest.

Coverage (stage 5). Now we sample 250 items against the taxonomy, and we choose the proportions rather than letting the tickets choose them:

60 of 250 items — 24% of the dataset — are cases where a confident answer is the wrong output. No amount of collecting would have produced them, because help-desk tickets that got no answer tend not to survive as clean records, and nobody files a ticket to test their assistant's refusal behaviour. They exist because a curator decided they should.

Versioning (stage 6). policy-eval-v1.0, 250 items, 589-item source pool, filter counts as tabled, frozen. When someone proposes adding items later, that is v1.1 and every historical score gets a footnote.

What the 24% bought. Constructed illustration again: suppose the first pipeline scores 91% on the 190 answerable items and 12% on the 60 unanswerable ones — it answers almost everything correctly and refuses almost nothing correctly. Aggregate accuracy is 72%. If your dataset had contained only the answerable items, you would have shipped a 91% system, and the first employee to ask about a 2026 policy would have received an invented one. The unanswerable cases did not lower the score; they revealed the score. That is the argument, and it is the argument to reach for in any exam stem about why a well-performing model failed in production.

Curation is expensive. Not every situation warrants it, and knowing the exception is part of the judgment this exam tests.

The two rows worth internalising for the exam are the two "no" rows. If a scenario says a team wants the model to learn new, frequently changing facts, curating a fine-tuning dataset is the distractor and building a retrieval corpus is the answer — and the [FIELD] calibration note that a RAG option is usually the keyed one in scenario questions applies here directly. And if the scenario says the information does not exist anywhere in the organisation's data, then neither RAG nor fine-tuning is the answer, and the honest option — acquire the knowledge, or scope the feature out — is the correct one.

Curation earns its place three separate ways, which is unusual.

It is a named objective. Objective 1.4 "Curate and embed content datasets for RAGs" is the only objective in the entire blueprint that names curation as a verb the candidate performs [OFFICIAL]. Objective 2.3 "Conduct data analysis under the supervision of a senior team member" covers the profiling and filtering half. The Data Analysis and Visualization domain is 14% of the exam, roughly 8 of 60 questions, and the job-role frame lists "defining, curating, labeling, and annotating LLM datasets" among the associate's responsibilities [OFFICIAL].

It is where the NVIDIA-tool identity questions live. NeMo Curator is a stack-map item, and the stack map is a Tier-1 drill target. Exam questions that describe a data-preparation problem and offer four NVIDIA product names are testing tool identity, not data science.

It is the setup for every downstream failure question. A large share of scenario questions describe a symptom — the model hallucinates, evaluation scores look great but production is bad, accuracy is high but a subgroup is failing — whose cause is a curation decision. Naming the curation defect is how you find the keyed answer.

Question phrasings to expect

• "A team is preparing a large text corpus for LLM training. Which NVIDIA component is designed for downloading, cleaning, filtering, and deduplicating that corpus?" → NeMo Curator.
• "An assistant answers in-scope questions well but invents answers for questions outside its knowledge base. What was most likely missing from the evaluation dataset?" → out-of-corpus / unanswerable cases.
• "Which step should be performed before quality filtering a corpus?" → deduplication (dedup then score; scoring duplicates wastes compute and skews statistics).
• "A team wants to improve model accuracy but cannot obtain more labeled data." → data augmentation.
• "Curating a dataset for RAG differs from curating one for fine-tuning primarily because…" → the RAG corpus is retrieved from at inference time and can be updated without retraining.

Distractor families

Mistake 8 is the one that catches experienced teams, because adding documents to an index feels additive and is not. A refreshed corpus with duplicated documents changes the neighbourhood structure of the whole index.

There is no sourced universal number, and any specific figure you see quoted for "the minimum fine-tuning dataset size" should be treated as folklore unless it comes with a citation. The defensible answers are structural rather than numeric. For an evaluation set, the constraint is statistical: the set must be large enough that the difference you care about is bigger than the noise, which is the subject of 09-09, and the practical path is to start at the 20-item scale of 01-08 and grow toward the hundred-item scale of 09-01. For an SFT set, consistency dominates size — a few hundred rigorously uniform examples routinely teach a format better than thousands of inconsistent ones, per 11-02. For a RAG corpus, size is set by the knowledge you need to cover, not by a target row count, and adding low-quality documents actively hurts retrieval. The exam-safe formulation: the right size is the size at which coverage of your taxonomy is complete and additional examples stop changing your measurements.

They optimise for opposite properties. A RAG corpus is read at inference time, so it wants factual accuracy, freshness, breadth, clean chunk boundaries, correct metadata, and aggressive deduplication; it does not need input–output pairs at all, and it can be updated tomorrow without touching the model. A fine-tuning set is compiled into weights, so it wants uniformity of format, consistency of labels, and demonstration of the behaviour you want; breadth of facts is close to useless in it, because facts learned in weights cannot be updated, cited, or audited. The decision rule is in 05-06 and fully in 11-08: changing what the model knows is a retrieval problem; changing how the model behaves or formats is a fine-tuning problem.

Both, but for different reasons and never the same items. In the evaluation set they are diagnostic: they measure whether the system refuses correctly, and without them your score is arithmetically inflated in the way section 4 demonstrates. In the fine-tuning set they are instructive: pairs whose target output is a well-formed refusal or a clarifying question are how a model learns that "I do not have that information" is an acceptable response. Keep the two populations disjoint. An unanswerable case that appears in both is leakage, exactly as much as an answerable one would be.

It is the main place bias can be reduced, and also the main place bias enters, which is why it cuts both ways. Sampling that under-represents a group, labels produced by annotators from one background, and proxy features that stand in for protected attributes all enter at curation time and are cheapest to fix there. NVIDIA's trustworthy-AI framing names nondiscrimination — minimising bias so that people have equal opportunity to benefit from AI — as one of its pillars, and names dataset curation and validation tooling as part of the response [NVIDIA-DOC]. But curation cannot prove fairness; only disaggregated measurement can, which is why lesson 08-04 argues that an aggregated chart is structurally incapable of showing group-level harm. Curation gives you the balanced dataset; visualisation and subgroup evaluation are how you find out whether it worked. Lesson 13-04 handles the measurement side in full.

Scale and completeness of the pipeline, not cleverness. The stages themselves — download, extract, language-identify, clean, filter, deduplicate exactly and fuzzily, handle PII — are all implementable by hand, and for a few thousand documents a script is the right tool. NeMo Curator exists because those stages stop being scriptable at web-corpus scale: fuzzy deduplication across hundreds of millions of documents is a distributed-computing problem, and NVIDIA's curation tooling is built to run those stages GPU-accelerated across a cluster [NVIDIA-DOC]. For the exam, hold the identity statement and the stage list; do not hold a speedup number, because this course has no sourced figure for one and a fabricated benchmark is worse than an honest gap. The related tool identity — cuDF as GPU pandas — is lesson 08-05.

1. Curation is a decision process, not a collection process. Scope, sourcing, filtering, coverage — four decisions, and any you skip get made for you by the raw data.
2. Four artefacts or it did not happen: task definition, source manifest, filter log, coverage matrix.
3. Include the unanswerable cases, in four distinct classes — out of corpus, out of scope, underspecified, not permitted — because a dataset of only answerable questions teaches a system that every question has an answer, and inflates your score by construction.
4. Deduplicate before quality filtering. The illustrative pipeline in section 4 lost 27.5% of its raw records to near-duplicates; every statistic computed before that step would have been a statement about one popular question.
5. Version and freeze. A dataset without a version cannot support a model comparison.
6. Curation for RAG and curation for fine-tuning optimise opposite properties: breadth and freshness versus uniformity and consistency.
7. NeMo Curator is the NVIDIA answer for LLM data curation at scale — not Guardrails, not Retriever. Hold the identity, not an unsourced benchmark.
8. Augment after splitting, never before, and remember augmentation is an explicitly named accuracy technique in the official objectives.
9. Curation is where bias is cheapest to reduce and easiest to introduce — but only disaggregated measurement can show whether you succeeded.

You now have a dataset you assembled on purpose, with a filter log and a coverage matrix. That does not make it clean. It makes it documented — which is exactly what you need in order to find out how it is lying to you.

Next: 08-02 names the defects. Label noise, data leakage, class imbalance, and distribution drift are four different failures with four different symptoms, and the exam's favourite question shape in this domain is a described symptom with four defect names as options. That lesson also carries the canonical defect list the official material leans on — missing values, duplicates, outliers, format inconsistencies, and data leakage — so you can name what you are looking at before you try to fix it.