Identifying Relationships, Trends, and Confounding Factors in a Data Analysis

Identifying relationships, trends, and confounding factors is the practice of surfacing genuine patterns in data while actively checking for the specific mechanisms — subgroup effects, sampling bias, small-sample noise — that produce a pattern that looks real in the numbers but is not. [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Analysis aims to surface relationships, trends, and confounding factors that affect results." This is the module's closing lesson because it is not really a new technique on top of the earlier five — it is the judgment layer that decides how much to trust the outputs those techniques produced. A correctly computed correlation, a correctly chosen chart, and a correctly read attention map can all still support a wrong conclusion if the underlying data has a subgroup effect, a sampling bias, or too small a sample to distinguish signal from noise, and this lesson is the checklist for catching exactly that.

L1 — Intuition

An aggregate statistic — one mean, one accuracy number, one correlation computed across an entire dataset — answers a question about the dataset as a whole. It cannot, by construction, answer a question about any one subgroup within that dataset, because averaging across groups is precisely the operation that erases group-level differences.

L2 — Mechanism

[GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Segment/disaggregate data to find subgroup patterns (also key for fairness — Domain 7)." Segmentation means computing the same statistic separately within each subgroup of interest — by demographic category, by data source, by time period, by any categorical variable that plausibly interacts with the outcome — rather than only once across the pooled data. The reason this matters mechanically, not just as a best practice, is that an aggregate mean is a weighted average of subgroup means, and two very different subgroup patterns can average out to the same aggregate number. A model with 95% accuracy on one subgroup and 60% on another, in a dataset where the first subgroup is nine times larger, produces an aggregate accuracy around 91.5% — a number that looks strong and hides a subgroup performance gap of 35 percentage points entirely.

This is the identical mechanism behind Simpson's paradox, where a trend visible in the aggregate data can literally reverse direction once the same data is segmented by a lurking subgroup variable — a treatment can appear to help patients overall while actually hurting patients within every individual segmented subgroup, if the subgroup sizes and baseline rates combine in the right (or wrong) way. The paradox is not a mathematical trick; it is a direct consequence of how weighted averaging works, and it is the single clearest demonstration of why "look at the aggregate first" is an unsafe default rather than a reasonable starting point.

L3 — The exam-relevant edge case

[GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md) explicitly connects this to fairness, which Module 7's Trustworthy AI material covers in full — the outline's own note for this lesson states the connection directly: "the same discipline that catches subgroup harm in Trustworthy AI." An analysis question about whether a model is "fair" or "unbiased" is, mechanically, a segmentation question: has the aggregate performance number been disaggregated by the subgroups that matter, and does the disaggregated view tell a different story than the aggregate one. Recognizing that a fairness question and a segmentation question are the same underlying operation, applied to different subgroup categories, is a transferable insight this exam rewards across two separate domains.

L1 — Intuition

A pattern found in data can be an accurate reflection of a genuinely biased or imbalanced collection process rather than an accurate reflection of the real-world phenomenon the data is supposed to represent. [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Check for sampling bias and imbalanced classes."

L2 — Mechanism

Sampling bias occurs when the process used to collect data systematically excludes or under-represents part of the population the analysis is meant to generalize to — surveying only smartphone owners about internet usage habits, or collecting training images only from daytime photographs when the deployed system will also see nighttime input. Sampling bias corrupts what the data can honestly claim to represent, and no amount of clever modeling recovers information about the excluded population, because that population was never observed in the first place. Class imbalance is a different, narrower problem: one class or category is genuinely present in the real-world distribution being sampled, but at a much lower rate than another class, so a dataset that accurately reflects the real world still ends up with far fewer examples of the minority class. Fraud detection, rare disease diagnosis, and manufacturing-defect detection are classic imbalanced-class domains where the imbalance is a real, honest fact about the world rather than a collection error.

The two are easy to conflate because both produce a dataset that looks lopsided, but the correct response to each is different. Sampling bias needs to be fixed at collection — going back and gathering the excluded population's data — because no downstream statistical technique substitutes for data that was simply never collected. Class imbalance, when it accurately reflects the real world, is often better addressed at the metric level (per-class precision and recall rather than aggregate accuracy, the same principle section 2 raised for subgroups) or through resampling techniques applied carefully to the training split only, rather than treated as a defect requiring the minority class to be artificially inflated to match the majority.

L3 — The exam-relevant edge case

A scenario describing a lopsided dataset does not tell you, on its own, whether you are looking at sampling bias or class imbalance — that distinction requires asking whether the lopsidedness reflects the real-world population honestly or reflects a flaw in how the data was collected. A dataset with 2% fraud cases out of all transactions, collected by simply logging every transaction that occurred, is class imbalance: fraud really is rare in the underlying population. A dataset with 2% fraud cases collected by sampling only transactions flagged by an existing (imperfect) fraud-detection system is a candidate for sampling bias as well: transactions that no existing system flagged, including fraud that the existing system already misses, are systematically under-represented in ways that go beyond fraud's genuine real-world rarity.

L1 — Intuition

A trend that appears in a small amount of data can be real, or it can be an artifact of how few observations went into computing it — and distinguishing the two is a discipline, not an instinct, because a compelling-looking trend in a small sample feels exactly as convincing as a real one.

L2 — Mechanism

[GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Distinguish signal from noise; a small sample can show a 'trend' that is just variance." The underlying statistical fact is that any finite sample carries sampling variance — even a truly random, patternless process will, by chance, occasionally produce a run of values that looks like a trend when the sample is small. Flip a fair coin 5 times and getting 4 heads is a perfectly ordinary outcome, not evidence the coin is biased; flip it 500 times and getting 400 heads would be a completely different, genuinely alarming signal. The same logic applies to any metric tracked over a small number of observations — five consecutive days of a metric moving in one direction is comparatively weak evidence of a real trend, while the same directional movement sustained over five hundred days is comparatively strong evidence, all else equal.

The practical discipline this implies: before treating an observed pattern as a real trend worth acting on, ask how many independent observations it is based on, and whether a pattern of similar apparent strength could plausibly arise from a random process with that few observations. This is not a formal statistical-significance test — the domain's foundational-level scope does not require computing a p-value — but the underlying intuition (more observations, more confidence; fewer observations, more skepticism) is exactly what the source material is testing.

L3 — The exam-relevant edge case

This connects directly back to M2-02's correlation material: a correlation computed on a very small sample is simultaneously vulnerable to two separate risks — it may be spurious in the coincidence sense M2-02 named, and, independently, the r value itself is a less stable estimate with fewer data points behind it, meaning a slightly different small sample from the same underlying population could easily produce a noticeably different r. Both risks point toward the same practical response: treat a correlation or trend found in a small sample with more caution than one found in a large sample, and seek more data before treating it as a settled fact.

⭐ THE EARNED INSIGHT

A "small sample can show a trend that is just variance" is the same underlying caution as "correlation is not causation," pointed at a different failure mode — the first warns that what looks like a pattern may be noise, the second warns that what looks like a cause may be something else. Both are reminders that the data in front of you is a sample, not the full truth, and every conclusion drawn from it inherits that sample's limitations.

L1 — Intuition

A single number — "accuracy is 87%," "the average is 42.3" — presents a conclusion with a confidence the underlying data may not actually support, because it hides how much that number could plausibly have varied if the sample had been slightly different.

L2 — Mechanism

[GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Report uncertainty (ranges, error bars) rather than a single point estimate." A range or an error bar communicates the plausible spread around a point estimate — "accuracy is 87%, ± 4 percentage points" tells a reader something a bare "87%" does not: that a re-run of the same measurement on a different, equally valid sample could reasonably land anywhere in roughly the 83%-91% window, and a competing model reporting 88% is very plausibly not meaningfully different from this one at all, whereas without the uncertainty range, 87% versus 88% looks like a settled, if narrow, difference.

The practical sources of uncertainty worth naming: sample size (smaller samples produce wider plausible ranges, connecting directly to section 4), measurement noise (any real-world measurement process has some inherent variability), and — specific to model evaluation — the particular split of data a model happened to be evaluated on, since a different random split of the same underlying data would typically produce a somewhat different accuracy number even for the identical trained model.

L3 — The exam-relevant edge case

Reporting uncertainty is not merely a communication nicety; it changes what conclusions are defensible. Two point estimates that differ by 2 percentage points but each carry a ± 5 point uncertainty range overlap substantially, and treating "the higher one is better" as a settled conclusion from those numbers alone overstates what the data supports — the honest framing is that the two are not distinguishable at this level of precision, and a decision between them needs either more data or an additional criterion beyond the metric that motivated the comparison in the first place.

Treat the following as a constructed scenario built to make the reasoning legible, not a measurement from a real deployment. A multimodal captioning model is evaluated across 10,000 test images, with an aggregate caption-accuracy rate of 82%. A stakeholder is ready to approve deployment based on that number.

Step 1: Segment by an available metadata field before approving.
  Segment by image SOURCE (three sources contributed to the test set):
    Source A (7,000 images, professional stock photography): 91% accurate
    Source B (2,500 images, user-submitted smartphone photos): 68% accurate
    Source C (500 images, low-light security-camera stills): 34% accurate

Step 2: Recompute the weighted aggregate to confirm it matches the
  reported 82%.
  (7000*0.91 + 2500*0.68 + 500*0.34) / 10000
  = (6370 + 1700 + 170) / 10000 = 8240 / 10000 = 82.4%  -- matches.

Step 3: Interpret what the aggregate hid.
  The deployment's REAL use case is a consumer mobile app -- meaning
  Source B (user-submitted smartphone photos) is the single most
  representative segment of actual deployment traffic, not Source A.
  The 82% aggregate is dominated by Source A's 91%, purely because
  Source A is 70% of the test set by volume, not because Source A is
  representative of how the model will actually be used.

Step 4: Recommend based on the disaggregated view, not the aggregate.
  The metric that should drive the deployment decision is closer to
  Source B's 68%, since that segment is the actual target population --
  a materially weaker number than the headline 82% suggested, and one
  that changes the deployment recommendation from "approve" to "needs
  further work on smartphone-photo-quality inputs before launch."

The aggregate number in this constructed scenario was computed correctly and was not itself a lie — it is an honest weighted average of three real segments. What made it misleading was presenting it as if it answered the question that actually mattered (how will this perform for real users) when the test set's composition did not match the deployment population's composition, a mismatch invisible in the single 82% figure and immediately visible the moment the data was segmented.

Treat the following as a constructed scenario. A team monitoring a deployed recommendation model observes click-through rate (CTR) over the first five days after a model update: 3.1%, 3.4%, 3.0%, 3.6%, 3.8% — a run that looks like a rising trend, prompting a proposal to declare the update a clear success.

Step 1: Note the sample size behind each day's number.
  Each day's CTR was computed from roughly 2,000 impressions per day --
  a comparatively small sample for a metric this noisy day to day.

Step 2: Compute a rough uncertainty range for a single day's CTR at
  this sample size (illustrative, using a standard-error-style estimate
  for a proportion: sqrt(p(1-p)/n)).
  For p = 0.034 (the mean of the 5 days), n = 2,000:
  SE = sqrt(0.034 * 0.966 / 2000) = sqrt(0.0000164) = 0.0128
  A rough 95% range is roughly +/- 2*SE = +/- 2.6 percentage points
  around any single day's estimate.

Step 3: Compare the day-to-day movement against that uncertainty range.
  The observed day-to-day swing (3.0% to 3.8%, a range of 0.8 points)
  is SMALLER than the +/- 2.6-point uncertainty band computed for a
  single day's estimate at this sample size -- meaning day-to-day
  movement of this size is well within what pure sampling noise alone
  would produce, even if the true underlying CTR had not moved at all.

Step 4: State the honest conclusion.
  Five days of data at ~2,000 impressions/day is not enough to
  distinguish a real upward trend from ordinary day-to-day variance.
  The correct recommendation is to keep monitoring for a longer window
  (more days, more cumulative impressions) before declaring the update
  either a success or a failure -- not to act on five noisy data points
  as if they already answered the question.

This is a constructed illustration — the specific numbers are chosen to be plausible rather than measured — but the reasoning pattern (compare the observed movement against a rough sense of how much noise the sample size alone could produce) is the transferable skill, and it generalizes past click-through rate to any metric tracked over a short observation window: training loss over a handful of steps, validation accuracy across a handful of small evaluation batches, or a business metric tracked for only a few days after a change.

Data Analysis is Domain 2 at 10% exam weight, and this closing subsection is where the domain's individual techniques (cleaning, EDA, chart choice, attention maps, augmentation/OCR) are applied with appropriate skepticism [GROUND TRUTH] (Sources/nca-genm/domain-2-data-analysis.md): "Analysis aims to surface relationships, trends, and confounding factors that affect results," followed by the four named practices this lesson has walked through in full. The question tends to arrive as a scenario presenting a headline conclusion (a strong aggregate metric, an apparent short-term trend, a lopsided-looking dataset) and asking what should be checked, or what is missing, before that conclusion is trusted — testing whether you reach for segmentation, a bias/imbalance check, a signal-versus-noise judgment, or an uncertainty estimate as the appropriate next step for the specific scenario described, rather than testing rote recall of any one technique in isolation.

What the distractors typically look like

The reliable traps mirror the ones the source material calls out directly: accepting a strong aggregate number without asking whether it should be segmented; treating a lopsided dataset as automatically a bias problem when it may honestly reflect real-world class imbalance (or the reverse — treating real bias as if it were merely honest imbalance); treating a short-run pattern as a confirmed trend without considering the sample size behind it; and presenting a single point estimate as decisively different from a competing number without any sense of the uncertainty around either.

The two checks share a family resemblance — both ask "is there a third factor complicating the story a simpler view suggests" — but they answer different questions about different objects. A confounder check, from M2-02, asks whether a correlation between two variables is genuinely causal or is instead explained by a third variable driving both of them; it is a question about the relationship between two specific measured quantities. A subgroup check, from this lesson, asks whether a single aggregate statistic — a mean, an accuracy rate, a trend — actually holds uniformly across the population it was computed over, or whether it is instead an average masking real differences between subpopulations; it is a question about whether one number is representative of every part of the group it summarizes. In practice the two checks are often run together on the same analysis: having found a correlation, you would check both whether a confounder explains it (the M2-02 question) and whether the correlation's strength or direction itself varies by subgroup (this lesson's question) — a correlation can be confounded in the aggregate and also differ in strength across segments, and a thorough analysis checks for both independently rather than treating one check as a substitute for the other.

There is no single fixed number the source material specifies, and treating any round number (30, 100, 1,000) as a universal threshold is itself an oversimplification, because how much data is "enough" depends on how noisy the underlying metric naturally is and how large a difference you are trying to detect. A metric with very low natural variability (say, a controlled lab measurement) can support a confident conclusion from relatively few observations, while a highly variable metric (day-to-day web traffic, click-through rate) needs proportionally more observations before a real trend becomes distinguishable from noise at the same confidence level. The transferable habit from section 4 — comparing the observed movement against a rough estimate of the sampling noise at the current sample size — is the general-purpose substitute for memorizing any single fixed threshold.

It is worth flagging as a consideration, even though it is not a data-collection defect the way sampling bias is. Genuine class imbalance still has real consequences for model training and evaluation — a model can achieve a deceptively high aggregate accuracy by effectively ignoring the rare class, which is exactly M1-03's accuracy-alone-misleads-on-imbalanced-data point resurfacing here — so the correct response is not "fix the data" (there may be nothing to fix; the world really is imbalanced) but "choose an evaluation and, if needed, a training strategy that accounts for the imbalance," such as per-class metrics or careful resampling of the training split.

A multimodal system has more independent axes along which a subgroup effect can hide, not fewer. A single-modality image classifier's aggregate accuracy can already hide a demographic subgroup effect the way section 2's worked example showed; a multimodal captioning or retrieval system adds source-modality quality (a blurry photo versus a clean one, a noisy audio clip versus a clean one), cross-modal pairing quality (a caption written by a careful annotator versus an auto-generated one), and language or script (for text) as additional segmentation axes that a purely visual or purely textual system would not have. A thorough segmentation pass on a multimodal system's evaluation results checks each of these axes, or at minimum the ones most plausibly correlated with the deployment population's actual composition — skipping any one of them reintroduces exactly the aggregate-hides-the-subgroup risk section 2 described, just along an axis specific to combining modalities rather than an axis a single-modality system would also have.

No — the domain's foundational-level scope, consistent with the rest of this module, asks for the habit of considering uncertainty and sample size, not for computing a formal hypothesis test or p-value on every reported number. A rough sense of "how much could this plausibly vary given how much data it is based on" is the testable skill; a fully rigorous confidence interval or significance test is a M3-level experimentation concern for the module that follows this one, where controlled experiment design is covered directly.

• Segment by subgroup before trusting an aggregate trend — an aggregate is a weighted average that can hide, or even reverse (Simpson's paradox), the pattern that matters within any one subgroup.
• Sampling bias (a flawed collection process) and class imbalance (an honest but lopsided real-world distribution) look similar but need different fixes — bias needs new data collection, imbalance needs a better metric or careful resampling.
• A small sample can show a "trend" that is pure variance — compare an observed movement against a rough estimate of the sampling noise at that sample size before declaring it real.
• Report uncertainty (a range or error bar) alongside a point estimate, because two numbers that look different can be statistically indistinguishable once their uncertainty is accounted for.
• This module's segmentation discipline is the same mechanism Module 7 uses to catch subgroup harm in a fairness evaluation — one habit, two domains.
• Every technique earlier in this module (cleaning, EDA, charting, attention maps, augmentation/OCR) can be executed correctly and still support a wrong conclusion if this lesson's four checks are skipped.

Module 2 is complete: you can now clean data without leaking it, describe it honestly, chart it without distorting it, read a model's attention with appropriate skepticism, prepare multimodal input for training and ingestion, and — this lesson's contribution — interrogate a conclusion before trusting it.

Next: Module 3, Experimentation, picks this up directly. It is where the uncertainty-and-small-sample caution from section 4 becomes a formal experiment design, where the correlation-is-not-causation gap from M2-02 finally gets closed with a controlled or randomized test, and where the generative models — diffusion, GANs — that the rest of this course builds toward get their own evaluation metrics and their own version of the "is this result real or noise" discipline this lesson just taught you to apply.