Organizing and Formatting Datasets for Pretraining, Fine-Tuning, and RAG

Identity statement: dataset formatting is the step that takes cleaned, curated content and organizes it into the specific structural shape a downstream training objective or retrieval system requires, because pretraining, fine-tuning, and RAG are not reading the same kind of input from the data.

The reason one format cannot serve all three jobs traces back to what each job's training or query mechanism actually consumes. Pretraining's causal language modeling objective wants a long, continuous stream of text to predict one token at a time — it has no concept of "instruction" and "response" as separate roles, and forcing that distinction onto pretraining data would waste structure the objective cannot use. Supervised fine-tuning's objective is explicitly conditional: given this instruction or prompt, produce this response, which means the training signal only makes sense once prompt and response are labeled as separate, distinguishable fields rather than one undifferentiated stream. RAG's retrieval step is a nearest-neighbor search over embedded units of content, and a nearest-neighbor search over one giant undifferentiated document returns that entire document or nothing useful — it needs content pre-divided into passages small enough that a single passage can be a meaningfully specific answer to a specific query.

None of these differences are about the underlying facts changing. A single well-written paragraph explaining how a product works could plausibly appear, in some form, in a pretraining corpus, as the answer half of a fine-tuning pair, and as one chunk in a RAG index — the content is the same; the packaging is not, and getting the packaging wrong for the job at hand is what this lesson exists to prevent.

L1 — Intuition

JSON Lines (JSONL) is a plain-text format where every line is an independent, complete JSON object. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names JSONL specifically for instruction data — one line, one self-contained training example, with no dependency between lines and no wrapping array to load before you can start reading.

L2 — Mechanism

The line-per-record property is what makes JSONL the practical default for large text corpora rather than a single giant JSON array. A JSON array containing millions of documents has to be parsed as one syntactic unit — malformed content anywhere in that array can make the entire file unparseable, and reading it typically means loading the whole structure into memory before touching a single record. JSONL sidesteps both problems: a pipeline can stream the file one line at a time, a corrupted line fails independently without invalidating every other line in the file, and a training loader can shard, shuffle, or resume from an arbitrary line without ever needing the full file in memory at once. For a pretraining corpus that might run to terabytes of text, that streaming property is not a convenience — it is close to a requirement.

A minimal pretraining-style JSONL record typically carries just the text and whatever metadata a pipeline needs for filtering or provenance:

{"text": "The transformer architecture computes self-attention...", "source": "textbook-corpus", "doc_id": "a1f9"}
{"text": "In distributed training, gradient accumulation lets...", "source": "textbook-corpus", "doc_id": "a1fa"}

An instruction-formatted JSONL record — still one self-contained line, still trivially streamable — carries the same text-plus-metadata shape but adds the fields a fine-tuning objective needs, which section 3 below covers in full.

L3 — The exam-relevant edge case: schema consistency across a JSONL file

The property that actually gets tested is not "what is JSONL" in the abstract, but what breaks when a JSONL file's records are not schema-consistent. Every record in a training-ready JSONL file needs the same set of keys, in a form the loader expects — if 99% of records carry {"text": ..., "source": ...} and a stray 1% instead carry {"content": ..., "src": ...}, most training loaders will not politely skip those records; they will either crash on the first mismatched key or, more dangerously, silently populate a default or empty value for whatever field a permissive loader treats as optional, which can look like a mundane missing-value case when it is really a schema-drift bug from an upstream pipeline change. Consistent schemas are why objective 3.2 groups "correct formats" and "consistent schemas" together rather than treating format choice and schema discipline as separate concerns — a JSONL file with the right idea and an inconsistent schema is functionally as broken as choosing the wrong format outright.

L1 — Intuition

Where pretraining data is undifferentiated text, supervised fine-tuning (SFT) data has to explicitly separate what the model is given from what the model is supposed to produce. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names prompt/response pairs as the correct format for SFT specifically because the training objective needs that separation to compute a loss only over the response the model is supposed to learn to generate.

L2 — Mechanism

A prompt/response pair, in its simplest form, is a JSONL record with (at minimum) two fields — commonly named prompt and response, or instruction and output, depending on the framework — where the prompt is the input the model receives and the response is the target output the loss function is computed against:

{"prompt": "Summarize the following support ticket in one sentence: [ticket text]", "response": "Customer reports login failures after the latest app update."}
{"prompt": "Translate to French: The meeting has been rescheduled to Thursday.", "response": "La réunion a été reportée à jeudi."}

The mechanical reason this separation matters, rather than just concatenating prompt and response into one undifferentiated text field, is how the training loss is computed. In most SFT setups, the loss is masked over the prompt tokens and computed only over the response tokens — the model is not being trained to predict the instruction it was given, only to predict a good response to it. A dataset that merges prompt and response into a single text field with no marker of where one ends and the other begins loses the information a loss-masking step needs to know where to start counting, which either forces a fragile downstream re-parsing step or, worse, trains the model on an unmasked loss that includes the prompt tokens themselves, diluting the training signal that should be concentrated on response quality.

Multi-turn conversational data extends this same shape rather than replacing it: a list of turns, each tagged with a role (user, assistant, and sometimes system), still resolves at training time to the same underlying prompt-context/response-target separation — everything up to the final assistant turn functions as the prompt, and that final turn is the response the loss is computed over.

L3 — The exam-relevant edge case: a prompt/response pair is not "text with an obvious answer in it"

The trap worth naming directly: not every text-with-a-question-and-answer-shaped-substring is a correctly formatted SFT pair. A raw document that happens to contain a question followed by its answer somewhere in flowing prose is pretraining-shaped content, not SFT-shaped content, until it is deliberately extracted, labeled, and separated into explicit prompt and response fields. Treating "the answer appears somewhere in this text" as equivalent to "this is a labeled prompt/response pair" skips the actual formatting work objective 3.2 is testing — the fields have to be explicit and structural, not merely inferable by a human reader skimming the passage.

L1 — Intuition

RAG's retrieval step searches over embedded vector representations of content and returns the top-matching pieces, and [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names chunked passages as the correct format specifically because a nearest-neighbor search over an entire long document is a much cruder, less precise operation than a search over that same document split into focused, retrievable pieces.

L2 — Mechanism

Chunking takes a long source document and divides it into smaller passages, each of which gets its own embedding and its own entry in the retrieval index. The chunk boundary choice is itself a real decision with real consequences: chunk too large, and a single retrieved passage dilutes a specific fact with a large amount of surrounding, possibly irrelevant context, wasting the limited context budget a downstream generation step has to work with; chunk too small, and a single retrieved passage can lose the surrounding context a fact actually needs to be interpreted correctly, or split a single coherent fact across two separate chunks so that neither chunk alone answers the query that depends on it.

A chunked-passage record, formatted for indexing, typically carries the passage text plus enough metadata to trace it back to its source and reassemble context if needed:

{"chunk_text": "The training split must never contribute to preprocessing statistics used on validation or test data...", "doc_id": "m3-01-lesson", "chunk_index": 4, "source_title": "Dataset cleaning and curation"}
{"chunk_text": "Fitting a scaler on the full dataset before splitting is the single most commonly named exam trap in this domain...", "doc_id": "m3-01-lesson", "chunk_index": 5, "source_title": "Dataset cleaning and curation"}

Note the doc_id and chunk_index fields: because a chunk is deliberately a fragment of a larger document, a well-formed RAG dataset preserves enough structure to trace any retrieved chunk back to its parent document and its position within it — useful both for citing a source to a user and for debugging why a particular chunk was or was not retrieved for a given query.

L3 — The exam-relevant edge case: chunking is a formatting decision, not a retrieval-time fix

A common misreading treats chunk size as something to tune at retrieval time — adjusting how many chunks to retrieve, or reranking retrieved chunks — rather than recognizing it as a decision made once, upstream, at formatting time, that constrains everything retrieval can subsequently do. If a corpus was chunked too coarsely at indexing time, no amount of retrieval-time cleverness recovers the precision that a finer chunk size would have preserved, because the index simply does not contain any smaller retrievable unit than what chunking produced. Getting chunk size right belongs squarely in the formatting stage this lesson covers, not in whatever retrieval or reranking logic runs afterward.

Take a single constructed source paragraph — illustrative content, not drawn from any real dataset — and trace it through each of the three formats to see the reshaping directly.

Source content: "NeMo Curator deduplicates large text corpora using both exact-match hashing and near-duplicate detection based on content overlap, and it is designed to run on GPU-accelerated infrastructure so that deduplication of a multi-terabyte corpus completes in hours rather than days."

As a pretraining record, the paragraph goes in largely unchanged, as one line of undifferentiated text:

{"text": "NeMo Curator deduplicates large text corpora using both exact-match hashing and near-duplicate detection based on content overlap, and it is designed to run on GPU-accelerated infrastructure so that deduplication of a multi-terabyte corpus completes in hours rather than days.", "source": "data-prep-notes", "doc_id": "dp-0091"}

As an SFT pair, the same content is deliberately split into an explicit question and an explicit answer — a step that requires a human, or an automated extraction pipeline, to decide what the useful prompt/response boundary actually is:

{"prompt": "What deduplication methods does NeMo Curator use, and why does it run on GPU infrastructure?", "response": "NeMo Curator uses both exact-match hashing and near-duplicate detection based on content overlap. It runs on GPU-accelerated infrastructure so deduplication of a multi-terabyte corpus completes in hours rather than days."}

As a RAG chunk, the same content becomes a retrievable passage, sized to be a complete, self-contained answer to a plausible query on its own — in this case small enough that no further splitting is needed, but tagged with the metadata a larger document's chunks would need to stay traceable:

{"chunk_text": "NeMo Curator deduplicates large text corpora using both exact-match hashing and near-duplicate detection based on content overlap, running on GPU-accelerated infrastructure so a multi-terabyte corpus completes deduplication in hours rather than days.", "doc_id": "dp-0091", "chunk_index": 0, "source_title": "Data prep notes"}

The content did not change across the three versions in any meaningful way. What changed each time was the structural packaging a specific downstream consumer — a causal LM's next-token objective, an SFT loss function, or a retrieval index — actually needs to do its job.

Treat this as a constructed scenario built to make the failure mode legible, not a report from any real incident. A team maintains an SFT dataset of 200,000 prompt/response pairs, collected over several months from two different annotation vendors.

Vendor A's records (140,000 examples):
  {"prompt": "...", "response": "..."}

Vendor B's records (60,000 examples, onboarded three months later):
  {"instruction": "...", "output": "..."}

Both vendors produced genuinely well-written prompt/response content — the annotation quality itself was fine. The problem is purely structural: Vendor A's records use the keys prompt and response; Vendor B's records use instruction and output. A training loader configured to read prompt/response keys will, depending on how permissively it is written, either crash outright on Vendor B's 60,000 records, or — the more dangerous failure — silently treat every missing prompt field as an empty string and every missing response field the same way, quietly training on 60,000 examples that are effectively blank.

Symptom, if the loader fails silently:
  Reported training set size:      200,000 examples
  Examples actually contributing:  140,000 examples (Vendor A only)
  Examples silently reduced to near-empty records: 60,000 examples (30% of the dataset)

Thirty percent of a dataset can silently stop contributing useful training signal without a single error message, purely because two vendors used different key names for functionally identical fields. The fix is the same discipline objective 3.2 names directly: a consistent schema enforced across every source before any file is merged into a single training set, not assumed after the fact because "the content all looks like prompt/response pairs."

A clean, non-leaking train/validation/test split, established on raw documents per M3-01's discipline, is not automatically safe once those documents get reformatted. Reformatting introduces two specific new leakage risks that a document-level split does not by itself prevent.

The first risk is chunk-level split violation for RAG data: if a single long document is chunked into, say, ten passages, and those ten chunks are then randomly assigned to train, validation, and test splits independently of one another, chunks from the same source document can end up on both sides of the split. A retrieval model evaluated on a "held-out" chunk that came from the same document as several training-split chunks has, in effect, seen closely related content during training — an information leak at the chunk level even though the original document-level split looked clean. The fix is to split at the document level before chunking, and carry that split assignment through to every chunk a document produces, so an entire document's chunks land on one side of the split together.

The second risk is near-duplicate leakage across the SFT reformatting step: if the same underlying fact gets extracted into multiple, superficially different prompt/response pairs — a common byproduct of automated extraction pipelines that generate several question phrasings per source paragraph — those near-duplicate pairs can land on opposite sides of a naive random split, again leaking closely related content across the split boundary in a way a document-level dedup pass, run before extraction, would have caught. M3-01's deduplication discipline has to be reapplied, or carried forward carefully, at the level of whatever unit — document, chunk, or extracted pair — the actual split is drawn over.

⭐ THE EARNED INSIGHT: > A train/validation/test split is a property of whatever unit it was drawn over, not a permanent property of the underlying content — and reformatting changes the unit. A split drawn over documents is clean for documents; the moment those documents become chunks or extracted pairs, the split has to be redrawn, or carefully carried forward, over the new unit, because "this document was in training" says nothing on its own about which side of the split any one of its ten derived chunks landed on unless that assignment was deliberately preserved through the reshaping step.

The practical discipline this implies: treat the split assignment as metadata that travels with a document through every reformatting step, rather than as a decision made once on raw documents and then forgotten. A chunking or extraction pipeline that reads a document's already-assigned split label and stamps every derived unit with that same label costs almost nothing to implement and closes both leakage risks above in one move — chunk-level and extracted-pair-level leakage are really the same underlying mistake wearing two different formats.

Objective 3.2 covers dataset organization and formatting directly, and the source material's own framing — correct formats, consistent schemas, clean non-leaking splits — is explicit that this is "unglamorous" material precisely because it is where pipelines silently break rather than loudly fail. Data Preparation sits at 9% of the NCP-GENL blueprint, and formatting is one of five subsections inside that domain, so any individual formatting question is a modest slice of the exam — but the source material's own emphasis that this is where real pipelines break suggests the questions reward recognizing a broken pipeline description over reciting format names from memory.

Expect this material in a few recurring shapes: a scenario naming a specific downstream job (pretraining a base model, fine-tuning on instructions, indexing for retrieval) and asking which format fits it, testing the JSONL/prompt-response/chunked-passage mapping directly; a scenario describing a merged dataset from multiple sources with inconsistent field names or structures, testing whether you catch a schema-consistency failure rather than assuming "the content looks fine so the format must be fine"; and a scenario describing a split performed at the wrong granularity — chunks or extracted pairs split independently of their source document — testing whether you recognize reformatting-introduced leakage as distinct from, but just as real as, the raw-document leakage M3-01 already covered.

What the distractors typically look like

The standing traps here are: offering prompt/response formatting as correct for pretraining data, when pretraining has no prompt/response structure to preserve; offering raw, unchunked documents as acceptable for RAG indexing, when retrieval precision depends on chunk granularity; and describing a document-level train/test split as sufficient protection against leakage in a chunked or extracted dataset, when the actual split needs to happen at, or be carried through consistently to, the unit the reformatting step produced.

Chunk overlap — deliberately repeating a small span of text at the boundary between two adjacent chunks — is a real and commonly used technique, and it exists to solve a specific problem: a fact that happens to straddle a chunk boundary can otherwise be split so that neither chunk alone contains it fully. A modest overlap, typically a small fraction of the chunk size, catches boundary-straddling facts without meaningfully reintroducing the near-duplicate risk M3-01 warns about, because the repeated span is a small fragment of a chunk rather than a fully repeated document. The distinction that keeps this from becoming a deduplication problem is scale: M3-01's concern is whole documents or large spans repeated across a corpus by accident of collection, inflating memorization; a few dozen overlapping tokens at a deliberate, controlled chunk boundary is a formatting decision made on purpose for a specific, bounded reason, not an artifact of duplicate collection. Excessive overlap — approaching the chunk size itself — does start to reintroduce a real cost, bloating the index with mostly-redundant chunks, which is why overlap is typically kept to a modest fraction of chunk size rather than treated as a free way to avoid boundary problems entirely.

Yes, and this is common in practice rather than an edge case — the same cleaned source corpus from M3-01 can be independently formatted into a pretraining JSONL stream, an SFT prompt/response set, and a RAG chunk index, each produced by a separate formatting pipeline that reads the same clean upstream content. What cannot happen safely is skipping the reformatting step and feeding one format directly into a job built for a different one — an SFT loader fed raw pretraining-style JSONL records has no prompt/response fields to find, and a retrieval index built from full, unchunked documents will not retrieve at the granularity a query actually needs.

Because chunking multiplies the number of retrievable or trainable units without necessarily preserving the original split boundary at that finer granularity. A single document assigned to the training split, once chunked into ten passages, is not automatically ten training-split chunks unless the split assignment is explicitly carried through the chunking step — a naive pipeline that reformats first and splits second, treating each chunk as an independent unit to randomly assign, can easily place some of a document's chunks in training and others in validation or test, which leaks closely related content across the split boundary even though the original document-level split, before chunking, looked completely clean.

• Pretraining wants JSONL text streams, SFT wants explicit prompt/response pairs, and RAG wants chunked passages — three structurally different formats built from the same underlying clean content, matched to what each downstream objective or search mechanism actually consumes.
• Schema consistency matters as much as format choice. A dataset in the right format with an inconsistent schema across its sources can silently corrupt a fraction of every training run.
• Chunk size for RAG is a formatting-time decision, not a retrieval-time fix — too coarse dilutes retrieved context, too small loses it, and no amount of reranking recovers precision lost at indexing time.
• A document-level split is not automatically safe after reformatting. Chunking or extraction can multiply a document into many units that need the original split assignment carried through, not reassigned independently.
• This material is unglamorous by design — the source material's own framing is that formatting is where pipelines silently break, which is exactly why a schema-drift or split-leakage scenario, not a bare definition question, is the shape this material tends to take on the exam.

Getting the format and the schema right is what makes a dataset trainable at all. It says nothing yet about the one decision inside this domain that is permanent once training starts: which tokenizer a model uses to turn that correctly formatted text into the integers its embedding layer actually consumes.

Next: M3-03 picks up exactly there — subword tokenization, and the BPE-versus-WordPiece distinction the exam treats as one of this domain's standing distractor pairs.