Preparing Multimodal Data: Data Augmentation and OCR for PDF Extraction

Data augmentation artificially expands a training set by creating modified copies of existing examples, improving accuracy and reducing overfitting, especially when the amount of available labeled data is limited. [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Data augmentation artificially expands the training set by creating modified copies — image flips/crops/rotations/color jitter, audio time-stretch/noise, and text paraphrase/back-translation. It improves accuracy and reduces overfitting, especially with limited data."

The mechanism behind why this works connects directly to Module 1's bias-variance material: a model overfits when it has enough capacity to memorize idiosyncrasies of the specific training examples it saw, rather than learning the general pattern those examples were sampled from. Augmentation attacks this directly by showing the model many variations of the same underlying example — the same dog photographed, in effect, from a flipped angle, cropped differently, or under different lighting — which makes memorizing any one specific pixel arrangement a much less useful strategy than learning the actual visual concept the class label is about. The model still sees the same underlying set of real-world examples; it simply sees each one multiple times, each time slightly and plausibly different, which is functionally similar to having collected more raw data without the cost of collecting more raw data.

L1 — Intuition

Every modality has its own menu of transformations, and the common thread across all of them is the same design constraint: a transformation counts as valid augmentation only if it preserves the label. A flipped photo of a dog is still a photo of a dog; a transformation that would change what a human labeler would call the image is not augmentation, it is a new (and possibly mislabeled) example.

L2 — Mechanism

[GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md) names the specific transformations per modality directly. For images: flips (mirroring horizontally, and sometimes vertically where orientation is not meaningful), crops (taking a sub-region of the original image), rotations (turning the image by some angle), and color jitter (perturbing brightness, contrast, or saturation slightly). Each targets a different kind of invariance the model should learn — flips and rotations teach that an object's identity does not depend on its orientation in the frame; crops teach that an object's identity does not depend on exactly where it sits or how much surrounding context is visible; color jitter teaches that an object's identity does not depend on lighting conditions the camera happened to capture.

For audio: time-stretch (speeding up or slowing down playback without changing pitch) and noise injection (adding background noise at a controlled level). Time-stretch teaches that a spoken word or a sound event's identity does not depend on the exact speaking rate or duration; noise injection teaches robustness to the kind of imperfect recording conditions a deployed system will actually encounter, rather than only the clean studio-quality audio a training set might otherwise be dominated by.

For text: paraphrase (rewriting a sentence to preserve its meaning with different wording) and back-translation (translating a sentence into another language and then back into the original, which typically produces a plausible paraphrase as a side effect of the two translation passes). Both teach that a label or intent does not depend on the exact surface wording used to express it — a sentiment classifier, for instance, should assign the same label to two different phrasings of the identical underlying sentiment.

L3 — The exam-relevant edge case

The label-preservation constraint from L1 has a genuinely tricky edge case worth naming directly: some transformations that are safe for one task are unsafe for another using the identical modality and the identical transformation. A horizontal flip is safe augmentation for a general object classifier (a dog is still a dog mirrored) but is not safe for a task where left/right orientation is part of the label itself — a road-sign classifier distinguishing a left-turn arrow from a right-turn arrow would have its labels silently corrupted by horizontal flipping, because the flipped image now genuinely depicts the other class. The rule "does this transformation preserve the label" cannot be answered by naming the transformation alone; it requires knowing what the label actually depends on for the specific task.

L1 — Intuition

A PDF that started life as a scanned paper document, or as a photograph of a page, contains an image of text — pixels that happen to look like letters to a human eye — not machine-readable text a program can search, copy, or feed into a language model. OCR is the technology that closes that gap.

L2 — Mechanism

OCR (optical character recognition) [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "PDF extraction via OCR turns scanned/image PDFs into machine-readable text so documents can feed NLP/RAG pipelines — a common ingestion step for multimodal document workflows." Mechanically, OCR takes an image (the scanned page, or a photograph of a page) and outputs a sequence of recognized characters and their positions, typically by detecting regions of the image that contain text, segmenting those regions into individual characters or character groups, and classifying each segment against a learned model of what letters and symbols look like. A digitally-created PDF that was authored directly in a word processor or typesetting tool, by contrast, already contains machine-readable text embedded in its file format — no OCR step is needed for that kind of document, because the text was never an image to begin with. Distinguishing which category a given PDF falls into is itself a small but real practical step: attempting to programmatically extract text from a scanned PDF using a tool built for digitally-native PDFs typically returns nothing, or garbage, because there is no embedded text layer to extract — OCR is specifically the fallback (and, for a scanned document, the only route) that works when that embedded layer does not exist.

L3 — The exam-relevant edge case

OCR's output quality is not uniform across every kind of scanned document, and the source material frames this as an ingestion step specifically for "multimodal document workflows" rather than a solved, error-free conversion. A cleanly scanned, single-column, standard-font page produces highly reliable OCR output. A page with unusual fonts, low scan resolution, skewed or rotated scanning, handwriting, or a complex multi-column or table layout produces meaningfully less reliable output — and an OCR error at this ingestion stage propagates downstream into every later step (an NLP pipeline, a RAG retrieval index) exactly the way a data-cleaning defect from M2-01 propagates into a trained model. ⚠️ UNVERIFIED: the domain source material does not give a specific accuracy figure for OCR performance on different document qualities, so treat any specific OCR accuracy percentage you encounter elsewhere as a claim to verify against the specific document type and OCR tool in question, not as a fixed constant.

[GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md) names a further set of steps that apply once text — whether OCR-extracted or natively digital — needs to feed an NLP or RAG pipeline: "tokenization, normalization, stop-word handling, and vectorization (bag-of-words/TF-IDF or learned embeddings)." Tokenization splits text into units (words, subwords, or characters) a model can process individually. Normalization standardizes surface variation that does not carry meaningful signal for most tasks — lowercasing, removing extra whitespace, standardizing punctuation. Stop-word handling decides whether to remove very common, low-information words ("the," "and," "of") before certain kinds of analysis, though modern learned-embedding approaches frequently skip this step because the embedding itself learns to down-weight low-information tokens rather than requiring them to be stripped out beforehand. Vectorization converts tokens into numeric representations a model can compute over — either a classic bag-of-words or TF-IDF representation (counting or weighting word occurrences, ignoring word order) or a learned embedding (a dense vector that captures semantic meaning, the kind CLIP's text encoder and every transformer-based language model produce). [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md) also names three transformer-based tasks this pipeline commonly feeds: "text classification, named-entity recognition (NER), and question-answering during analysis" — three distinct downstream uses of the same cleaned, tokenized, vectorized text. Text classification assigns a document or passage to a category (a support ticket to a department, an email to spam or not-spam); NER identifies and labels named entities within a passage — a person, an organization, a date, a location — which is often the exact step that turns an OCR-extracted contract into a structured record a downstream system can query; question-answering locates or generates an answer to a natural-language question from a passage of text, which is the mechanism underneath a RAG system's final generation step once retrieval has already surfaced the relevant OCR-extracted or natively-digital document.

Treat the following as a constructed scenario built to make the reasoning legible, not a measurement from a real project. A team has 3,000 labeled image-caption pairs for a product-recognition task — a genuinely small dataset by modern standards — and needs to decide an augmentation policy before training a classifier that also uses the caption text as an auxiliary signal.

Step 1: Identify what varies in deployment that does NOT vary enough
  in the 3,000-pair training set.
  Observation: training photos were all taken in a single studio setup
  (consistent lighting, plain background, product centered). Deployment
  photos will be customer-submitted, with varied lighting, backgrounds,
  and framing.

Step 2: Choose image augmentations that target exactly that gap.
  - Color jitter (brightness/contrast/saturation): targets the lighting
    gap directly.
  - Random crop with some margin: targets the framing/centering gap.
  - Rotation within a SMALL range (+/- 15 degrees, not a full 90-degree
    rotation): targets minor camera-angle variation, while avoiding a
    label-breaking transformation for any product whose orientation is
    part of its identity (e.g., distinguishing an "upright" vs. "lying
    down" packaging shot, if that distinction matters to this task).
  - Horizontal flip: SKIPPED for any product with text or an asymmetric
    logo on the packaging, since flipping would make text unreadable in
    a way a real deployment photo never would.

Step 3: Choose text augmentations for the caption field.
  - Back-translation (English -> French -> English) to generate 2
    paraphrased versions of each of the 3,000 captions, tripling the
    effective caption-side training signal without any new photography.

Step 4: State what augmentation does NOT fix.
  If the 3,000 pairs simply never include a product category at all
  (say, a new product line launched after data collection), no amount
  of augmenting the EXISTING pairs manufactures examples of a category
  that was never collected in the first place. Augmentation diversifies
  what you have; it does not substitute for collecting what you don't.

The step 4 caveat is the most exam-relevant part of this worked example: augmentation is described in the source material as improving accuracy and reducing overfitting "especially with limited data," which is a claim about making better use of an existing, if small, dataset — not a claim that augmentation can conjure signal for something the dataset never captured at all.

Treat the following as a constructed scenario. A company wants to build a RAG system over 10,000 internal PDF documents accumulated over 15 years — a mix of digitally authored reports, scanned paper memos, and photographed whiteboard notes that were saved as image-only PDFs.

Step 1: Triage the 10,000 PDFs by type before choosing a pipeline.
  - ~6,000 are digitally-authored (text layer already embedded):
    route directly to text extraction, NO OCR needed.
  - ~3,500 are scanned paper documents (image-only PDF, clean single-
    column text, standard fonts): route through OCR.
  - ~500 are photographed whiteboard notes (handwriting, uneven lighting,
    non-standard layout): route through OCR but FLAG for manual review,
    since handwriting and uneven photography conditions are exactly the
    lower-reliability case named in section 3's L3 tier.

Step 2: Run OCR on the ~4,000 image-based documents (3,500 + 500).
  Output: raw extracted text per document, with a per-document confidence
  score most OCR tools report alongside their output.

Step 3: Apply the text-preprocessing pipeline from section 4 uniformly
  across ALL 10,000 documents (the 6,000 natively-digital ones and the
  4,000 OCR-extracted ones) -- tokenization, normalization, and
  vectorization into embeddings for the RAG retrieval index.

Step 4: Route low-confidence OCR output differently from high-confidence
  output.
  Documents where the OCR tool's own confidence score falls below a
  chosen threshold are indexed with a visible flag ("extracted via OCR,
  lower confidence") rather than silently mixed in with high-confidence
  text, so that a RAG answer citing one of these documents can surface
  the caveat to whoever is reading the answer.

Step 4 connects this ingestion decision directly to a lesson later in this course: a RAG system that retrieves a garbled OCR extraction and presents it with the same confidence as a clean, natively-digital document is manufacturing exactly the kind of over-confident, badly-grounded answer that Module 7's trustworthy-AI material treats as a hallucination-adjacent risk, even though the underlying language model did nothing wrong — the defect entered at ingestion, not at generation.

⭐ THE EARNED INSIGHT

OCR and augmentation both expand what a model can use, but they expand two different things: augmentation expands the training signal from data you already have, while OCR expands the set of documents you can even read from formats a model could not otherwise touch. Confusing the two — treating OCR as a form of data augmentation, or expecting augmentation to fix a document a model literally cannot parse — is a category error the exam is built to catch.

L1 — Intuition

Section 2's augmentation menu is written per modality in isolation, but a genuinely multimodal dataset usually pairs modalities together — an image with its caption, an audio clip with its transcript — and augmenting one half of a pair without the other risks quietly breaking a relationship the model is meant to learn from.

L2 — Mechanism

Consider an image-caption pair where the caption reads "a red car parked on the left side of the driveway." If an image augmentation pipeline applies a horizontal flip to the image but leaves the caption text untouched, the augmented pair now reads "a red car parked on the left side of the driveway" underneath an image where the car is actually on the right. The augmentation was individually valid for a plain image classifier (a flipped car is still a car), but it silently corrupts this particular pair the moment the caption contains a spatial reference the flip invalidates. The general principle: any augmentation applied to one modality in a paired dataset needs to be checked against whether the other modality's content still matches afterward, not just against whether the augmented modality alone still carries its original label.

The practical responses fall into a few patterns. Where the paired modality's content does not reference anything the augmentation would invalidate (a caption like "a red car in a driveway," with no left/right or up/down language), the augmentation is safe to apply to the image alone. Where the pairing does reference something orientation- or position-dependent, the augmentation either needs a matching edit to the other modality (rewriting "left" to "right" in the caption when the image is flipped — mechanically simple for a small, templated vocabulary of spatial terms, considerably harder for open-ended natural language) or needs to be excluded from the augmentation policy for that class of example entirely, the same exclusion this lesson's section 2 already applied to flips for asymmetric packaging. A third, often simpler option used in practice is choosing augmentations that are inherently unlikely to invalidate typical captions in the first place — color jitter and modest cropping rarely interact with caption language the way flips and rotations can, which is part of why they are often the safer default for paired image-text augmentation even before checking any individual caption's content.

L3 — The exam-relevant edge case

This pairing-integrity check is a specific instance of the consistency job M2-01 named as critical when merging multiple modalities or sources, applied at augmentation time rather than at initial cleaning time. A scenario describing an augmented multimodal dataset that produces worse model performance than the unaugmented version — the opposite of what augmentation is supposed to do — is a candidate for exactly this failure mode: check whether the augmentation policy respects the semantic content of the other modality in each pair before assuming the augmentation technique itself is simply unsuited to the task.

Data Analysis is Domain 2 at 10% exam weight, and this pairing is the domain's second explicitly named multimodal-specific addition, alongside attention maps: preparing diverse data, "including PDFs via OCR," for downstream models [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md). [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md) — the domain's own self-check question shows augmentation's expected test shape directly. "Data augmentation is used mainly to:" with the keyed answer "Expand and diversify the training set to improve accuracy and reduce overfitting," against distractors offering "increase model precision by quantization," "replace the need for a validation set," and "convert a PDF into text" — note that the fourth distractor is OCR's own job, deliberately placed as a wrong answer to an augmentation question, which is the exact category-confusion this lesson's earned insight names directly.

A second recognizable question shape names a specific transformation (a horizontal flip, a rotation, a back-translation) and a specific task, and asks whether the transformation is valid augmentation for that task — testing the label-preservation edge case from section 2's L3 tier rather than the bare definition. A third shape describes a document-ingestion scenario (a scanned contract, a photographed whiteboard) and asks what step is needed before the document's content can feed an NLP or RAG pipeline, with OCR as the keyed answer against distractors offering augmentation, direct vectorization, or fine-tuning — each a real technique that presupposes machine-readable text already exists, which a scanned document does not have until OCR produces it.

What the distractors typically look like

The reliable traps mirror the ones the source material calls out directly: offering "convert a PDF into text" (OCR's job) as an answer to an augmentation question, and vice versa, testing whether the two are being confused as one activity; offering a label-breaking transformation (an unrestricted flip or rotation) as valid augmentation for a task where orientation is part of the label; and presenting OCR as producing perfectly reliable text regardless of document quality, when the honest position is that reliability varies with scan quality, font, and layout complexity.

Yes, when the augmentation policy violates the label-preservation constraint from section 2, or when it is applied so aggressively that the augmented examples no longer resemble plausible real-world inputs at all. An overly aggressive color-jitter setting that pushes brightness or saturation far outside any range a real camera would ever produce trains the model on examples that do not resemble deployment data, which can waste model capacity learning to handle an unrealistic case rather than improving on the realistic ones. The general principle worth carrying forward is that augmentation should simulate the plausible range of variation a model will actually encounter, not an arbitrary or extreme range chosen without reference to real deployment conditions.

Modern OCR tools are commonly applied to phone-camera photographs of documents, not only to flatbed scans, though photograph-based OCR is a harder input case than a flatbed scan because of the additional variability a handheld photograph introduces — uneven lighting, perspective distortion (the page not being perfectly perpendicular to the camera), motion blur, and shadows. Many OCR pipelines include a preprocessing step specifically to correct for some of this (deskewing a tilted page, for instance) before the actual character recognition runs, precisely because raw phone-camera input is common enough in practice that handling it well is a real, separate engineering concern rather than an edge case to ignore.

Back-translation is a genuinely useful augmentation technique, and it is also not perfectly meaning-preserving every time — a round-trip through a second language can occasionally shift a sentence's nuance, tone, or, in rarer cases, its literal meaning, especially for idioms, sarcasm, or highly domain-specific terminology that a translation model was not trained on extensively. This is the text-modality version of the same label-preservation concern section 2 raises for images: the augmentation is valuable specifically because it usually preserves meaning while varying surface wording, and the specific cases where it does not (idiomatic or highly technical text) are worth spot-checking before trusting the augmented captions at the same weight as the originals.

Yes — OCR models are trained on text in specific languages and scripts, and their accuracy on a given document depends on how well that document's language, script, and typography match what the OCR system was trained to recognize. A well-supported language like English, using a standard Latin typeface, generally produces the most reliable OCR output; a less common language, a non-Latin script, or a document mixing multiple languages and scripts on the same page is a comparatively harder case, and a pipeline ingesting genuinely multilingual document collections should expect and plan for this variation rather than assuming uniform OCR reliability across every document regardless of its language.

Only to training data, and this follows the same split-discipline reasoning M2-01 established for scalers and imputation statistics. The validation and test splits exist to measure how the model performs on data that looks like real, naturally occurring input — augmenting them would mean measuring performance on artificially modified examples that do not represent what the model will actually encounter in deployment, making the resulting accuracy or F1 number harder to interpret honestly. Augmentation's entire purpose is to make the training process more robust by exposing the model to more variation; the evaluation splits are supposed to stay an unmodified, honest sample of the real distribution throughout.

• Data augmentation expands and diversifies a training set from data you already have; it improves accuracy and reduces overfitting, especially when data is limited.
• Image, audio, and text each have their own augmentation menu — flips/crops/rotations/color jitter, time-stretch/noise, paraphrase/back-translation — chosen from the source material directly.
• The label-preservation constraint is the deciding factor for whether a transformation counts as valid augmentation for a specific task — a flip is safe for a general classifier and unsafe for an orientation-dependent one.
• OCR converts a scanned or image PDF into machine-readable text; it is the standard ingestion step for feeding document content into a multimodal or RAG pipeline.
• OCR reliability varies with scan quality, font, and layout complexity — treat OCR output confidence as information worth tracking, not a uniform guarantee.
• Text preprocessing (tokenization, normalization, stop-word handling, vectorization) applies after OCR extraction exactly as it applies to any other raw text source.
• Augmentation and OCR solve different problems and are not interchangeable — augmentation multiplies labeled examples you already have; OCR converts a document you could not previously read at all into one you can.
• Neither technique fixes a data-quality defect (a leaked feature, a mislabeled example) already present in the underlying dataset — clean first, per M2-01, then augment or extract.

You now have the two multimodal-specific data-preparation techniques this domain names, on top of the classic cleaning, EDA, and charting skills from earlier in this module. The one job left in Module 2 is putting the whole toolkit to use on a genuinely open-ended question, and that is the subject of the next lesson.

Next: M2-06 covers identifying relationships, trends, and confounding factors in a real analysis — the practical discipline of segmenting an aggregate result by subgroup before trusting it, distinguishing real signal from small-sample noise, and reporting uncertainty rather than a single misleadingly confident number.