Experiment Design for ML: One Variable at a Time, A/B Testing, and Reproducibility

Identity statement: experiment design is the set of practices that let you attribute a measured difference in outcome to a specific, intended cause rather than to noise, to an unrelated confound, or to an unfair comparison. In a machine-learning context, that means: change one variable, compare on identical data, and keep a fixed baseline you are testing against.

When it matters: any time a scenario describes comparing two model variants, two prompts, two training configurations, or two architectures and asks which produced a "real" improvement.

Four practices make up the discipline, and each one closes off a specific way an experiment can lie to you.

One variable at a time. Hold everything constant except the single factor you are testing, and compare the result against a baseline (control) — the reference configuration every change is measured against. If you change the learning rate and the batch size in the same run and accuracy improves, you cannot say which change — or what combination — produced the gain. The next section works through exactly why this holds even when the temptation to "just try a few things together and see" feels efficient.

A/B testing. Randomly split traffic or data between a control (A) and a treatment (B), so that any systematic difference between the two groups comes only from the one thing you deliberately changed, not from who happened to land in which group. Measure a metric you decided on before looking at the results, and check whether the difference is large enough to be statistically meaningful rather than noise.

Fixed evaluation set. Every variant you compare must be scored on the exact same held-out data. Comparing variant A on one hundred images and variant B on a different hundred images destroys the comparison before a single metric is computed, because any difference in the scores could be a difference in how hard the two evaluation sets happened to be, not a difference in model quality.

Reproducibility. Fix random seeds, pin library and framework versions, and record the full configuration (hyperparameters, data snapshot, hardware) used to produce a result, so that anyone — including you, six weeks later — can rerun the exact experiment and get the same answer. A result nobody can reproduce is a result nobody should trust, because there is no way to distinguish "this really happened" from "this happened once, by chance, under conditions nobody wrote down."

Why "baseline" is not the same as "whatever you had before"

A baseline is a specific, fixed reference point chosen deliberately as the thing every treatment is compared against — not simply "the previous run" or "yesterday's config," which can itself drift if it was never pinned. Treat the baseline the same way you treat the fixed evaluation set: decide it once, keep it fixed for the whole experiment, and only change it deliberately when you are starting a new comparison, not silently mid-experiment.

Suppose a team retrains a text-to-image diffusion model with a new context-embedding scheme and, in the same run, also increases the number of denoising steps and switches to a larger training dataset. Image quality improves. Which change caused it?

There is no way to know from that single run. The improvement could be entirely due to the new embedding scheme, entirely due to the extra denoising steps, entirely due to the larger dataset, or some interaction between two of the three that neither one produces alone. [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names this directly as a common exam trap: "changing several things at once — you cannot attribute the result to any one change." The trap is not that the team did something obviously careless; each individual change is a reasonable thing to try. The trap is that trying them together destroys the ability to learn which one worked, which is usually the entire point of running the experiment.

This matters practically, not just for exam correctness, because the cost of not knowing compounds. If the team ships the combined change and later needs to roll back one piece — say the larger dataset turns out to have licensing problems — they have no evidence about whether removing it will also remove the quality gain, because they never isolated what the dataset contributed on its own.

⭐ THE EARNED INSIGHT A combined-change experiment does not fail to produce a result — it fails to produce a usable one. The measured number is completely real; what is missing is any way to route that number back to a specific cause, which is the only thing that makes an experiment worth running instead of just shipping the bundle and hoping.

The controlled comparison this domain expects you to reconstruct

A scenario question typically describes a multi-variable change and asks you to name the flaw, or describes a single-variable change and asks you to recognize it as sound. The reconstruction is mechanical once you separate the two ingredients: identify how many factors changed between the compared runs, and identify whether a fixed baseline was held constant throughout. Two changed factors against a moving baseline is unsound regardless of how good the result looks; one changed factor against a fixed baseline is sound regardless of how small the result is.

Identity statement: A/B testing is a randomized comparison between a control (A) and a treatment (B), where assignment to A or B is random, the metric being compared is fixed in advance, and the observed difference is checked for statistical significance before being trusted as real.

Three components make the method work, and each is a place a shortcut version quietly breaks it.

Random assignment. If treatment B systematically receives easier inputs, more recent data, or a different user population than control A, any difference you measure is confounded with that systematic difference, not isolated to the treatment itself. Randomization is what spreads confounding factors evenly across both groups so they are, in expectation, not the explanation for what you observe.

A metric decided in advance. Choosing which metric "won" after looking at several candidate metrics invites you to pick whichever one happened to favor the result you wanted, which is not a controlled test, it is a search for a favorable number. Pre-registering the metric before running the comparison is what keeps the test honest.

Statistical significance, not just a bigger number. A treatment that scores 61% against a baseline's 59% is not automatically "better" — it might be well within the range you would see from ordinary run-to-run variance. Section 4 works through exactly why sample size determines whether a gap like that means anything.

A/B testing versus a single-variable comparison: same discipline, different setting

A/B testing is the randomized-traffic-split version of the one-variable-at-a-time principle, applied specifically to comparing a live control against a live treatment rather than comparing two offline configurations on a static dataset. The underlying requirement is identical — one change, a fixed comparison basis, a metric decided up front — but A/B testing adds the randomized-assignment step because "live" settings (a production feature, a user-facing prompt variant) have a population of users or requests that has to be split fairly, where an offline experiment instead has a dataset that can simply be held fixed and reused.

Comparing every candidate model, prompt, or configuration against the same held-out data is what makes any of the comparisons above meaningful at all. [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) states the principle directly: "compare all variants on the same held-out data." Two specific ways this gets violated are worth naming individually, because they produce two different flavors of bad result.

Different evaluation sets across variants. If variant A is scored on one hundred prompts and variant B is scored on a different hundred prompts, any score difference is entangled with how hard each set of prompts happened to be. A model that looks worse than its predecessor might simply have drawn a harder evaluation slice, not have regressed at all.

Evaluating on the training set. [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names this as a distinct trap: "evaluating on the training set." A model that has already seen the exact examples in its evaluation set is not being tested on generalization — it is being tested on memorization, and it will score better than it would on genuinely new data. This is the same discipline as fitting a preprocessing scaler on the training split only rather than the whole dataset before splitting: information from outside the fair comparison has leaked into it, and the resulting number overstates how good the model actually is.

Sample size versus noise: how big is big enough to trust

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names a fifth trap distinct from the first four: "too small a sample makes differences look real when they are just variance." A ten-item evaluation set can easily show one variant "winning" by a wide margin purely because of which ten items happened to be included, with no relationship to which variant is actually better on the underlying task. The fix scales with the size of the effect you are trying to detect: a large, obvious quality difference can be caught on a modest evaluation set, but a small, subtle difference — the kind most real-world experiments actually produce — needs a correspondingly larger fixed evaluation set before the gap is trustworthy rather than noise dressed up as a finding.

A team runs a text-to-image diffusion project and proposes three separate comparisons. Evaluate each one against the four practices from section 1.

Proposal 1
  Baseline: current context-embedding scheme, 50 denoising steps, dataset v3
  Treatment: new context-embedding scheme, 50 denoising steps, dataset v3
  Evaluation: same 500-prompt held-out set, both variants
  Seeds: fixed and recorded for both runs
  -> One variable changed (embedding scheme). Fixed evaluation set. Reproducible.
  -> SOUND. Any measured FID difference can be attributed to the embedding scheme.

Proposal 2
  Baseline: current context-embedding scheme, 50 denoising steps, dataset v3
  Treatment: new context-embedding scheme, 80 denoising steps, dataset v4
  Evaluation: same 500-prompt held-out set, both variants
  Seeds: fixed and recorded for both runs
  -> Three variables changed at once (embedding scheme, step count, dataset version).
  -> UNSOUND. Even with a fixed evaluation set and full reproducibility, a measured
     improvement cannot be attributed to any one of the three changes.

Proposal 3
  Baseline: current context-embedding scheme, 50 denoising steps, dataset v3
  Treatment: new context-embedding scheme, 50 denoising steps, dataset v3
  Evaluation: baseline scored on 500-prompt set A; treatment scored on a newly
              sampled 500-prompt set B drawn from the same pool
  Seeds: fixed and recorded for both runs
  -> One variable changed. Reproducible. But NOT the same evaluation set.
  -> UNSOUND. Any measured difference is entangled with set A vs. set B being
     different prompts, however similar the two samples look on paper.

Reading the three side by side makes the pattern legible: Proposal 1 gets all three structural requirements right and is the only one whose result should be trusted as attributable. Proposal 2 fails on variable count alone, regardless of how carefully everything else was controlled. Proposal 3 fails on the fixed-evaluation-set requirement alone, despite doing the variable-isolation part correctly — a reminder that the four practices in section 1 are independent failure points, and getting three out of four right is still a failed experiment if the fourth is the one that broke.

A product team A/B tests two prompt-template variants for a customer-support assistant, measuring the rate at which users mark a response "helpful."

Variant A (control):    42 helpful out of 60 responses  = 70.0%
Variant B (treatment):  48 helpful out of 60 responses  = 80.0%

Observed gap: 10 percentage points, favoring B.

Ten points looks like a real win. But with only 60 responses per variant, a handful of borderline judgment calls swinging the other way would erase most of the gap — this is exactly the small-sample trap the source material names, applied to a live A/B test rather than an offline benchmark. Contrast the same 10-point gap measured on a properly sized sample:

Variant A (control):    4,200 helpful out of 6,000 responses  = 70.0%
Variant B (treatment):  4,800 helpful out of 6,000 responses  = 80.0%

Same 10-point gap, 100x the sample size.

The percentages are identical, but the second version is far harder to explain away as noise, because a 10-point swing across six thousand responses per group would require a much larger amount of variance than the same swing across sixty. This is a constructed illustration — the exact numbers are invented to make the contrast legible, not a measured result from any real support system — but the shape of the lesson generalizes directly: the same observed gap deserves very different levels of trust depending on how many observations produced it, and a scenario question that reports only a percentage difference without a sample size is deliberately withholding the information you need to judge whether the result is real.

Reproducibility is not a single checkbox; it is a short list of specific things that, left unrecorded, make a result impossible to recreate honestly.

A team that skips recording any one of these rows can still get a correct result once — the trap is not that the original experiment was wrong, it is that nobody, including the original team, can verify it later. [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) frames the whole discipline this way: "fix random seeds, pin versions, and record configs so results can be repeated." Every row above is a literal reading of that one sentence.

Experimentation is the single largest domain on the NCA-GENM blueprint at 25% of the exam, ahead of every other domain individually, and its own scope note frames the domain around exactly two skills: designing and interpreting experiments, and picking the metric that answers the question you are actually asking. Experiment design is the load-bearing half of that pair — every later metric this module covers (FID, BLEU, perplexity, retrieval faithfulness) is only trustworthy inside a comparison that was set up correctly in the first place, so a question that looks like it is testing a metric is very often, underneath, testing whether the comparison producing that metric was sound.

The question tends to arrive in a small number of recognizable shapes.

1. Multi-variable attribution items. A scenario changes several things at once and reports an improvement, then asks what conclusion can be drawn. The keyed answer is that no single cause can be credited; distractors offer one of the changed factors as "the" cause, or claim the combined result proves all the changes together were beneficial (itself a different, narrower claim than any one change working).
2. Evaluation-set integrity items. A scenario compares variants on different held-out sets, or evaluates on the training set, and asks whether the comparison is valid. The keyed answer names the specific integrity failure.
3. A/B design items. A scenario describes an A/B test missing randomization, or choosing a metric after seeing results, and asks what is wrong with the test.
4. Sample-size items. A scenario reports a percentage difference on a small sample and asks whether the difference is meaningful, with the keyed answer flagging the sample size as the missing piece of information.

What the distractors typically look like

The reliable distractor families here are: crediting one arbitrarily chosen factor out of several changed at once, since it is a plausible-sounding "the improvement came from X" answer that simply cannot be verified from the described experiment; treating a bigger raw percentage gap as automatically meaningful regardless of sample size; and describing an experiment that changed one variable but on two different evaluation sets as sound, because "only one thing changed" sounds like it satisfies the whole discipline when it satisfies only one of its four requirements.

A trustworthy A/B test result rests on three things holding together: random assignment between control and treatment so no systematic difference between the groups explains the gap, a metric that was chosen before the results were seen so it was not picked because it happened to favor one side, and a sample size large enough that the observed gap is unlikely to be ordinary variance rather than a real effect. A large raw percentage difference measured on a handful of samples is not automatically trustworthy — the source material names too-small a sample as a named trap precisely because small samples make noise look like a real difference.

You can, and sometimes that is a reasonable first pass when you only care whether the combination as a whole is worth keeping — but you lose the ability to say which of the changes was responsible for any of the improvement, and you cannot safely remove one of the changes later without re-running an experiment to find out what removing it does. If the goal is to learn which specific change works, one variable at a time against a fixed baseline is the only design that produces an answer to that question; a combined trial only answers "is the bundle, taken together, better," which is a narrower and less useful finding than most teams actually want.

• Change one variable at a time against a fixed baseline — the single most tested principle in this domain, because attribution is impossible once more than one factor changes.
• A/B testing isolates one change with a randomized split between control and treatment, a metric decided in advance, and a check for statistical significance.
• Every variant in a comparison must be scored on the identical fixed evaluation set; evaluating on the training set, or on different held-out data per variant, invalidates the comparison.
• Sample size determines whether a difference is real or noise — too small a sample makes ordinary variance look like a meaningful result.
• Reproducibility means fixed seeds, pinned versions, and a recorded configuration for every run, so a result can be verified rather than merely trusted.
• These four practices are independent failure points: a comparison that gets three of the four right is still an unsound experiment if the fourth one breaks.

Work through each item before checking the answer key. Every option is a real claim about some experiment somewhere — the task is matching it to the described scenario, not spotting an obviously fabricated distractor.

1. A team changes the learning rate and the optimizer in the same training run, and validation accuracy improves. What can they conclude?
   • A. The learning rate change caused the improvement.
   • B. The optimizer change caused the improvement.
   • C. No single cause can be credited from this run alone.
   • D. Both changes independently improved accuracy by the same amount.
2. Two model variants are each scored on their own randomly sampled 200-item test set drawn from the same larger pool. Is this a fixed evaluation set?
   • A. Yes, because both sets are the same size.
   • B. Yes, because both sets are drawn from the same pool.
   • C. No, because the two variants were not scored on the identical set of items.
   • D. No, because 200 items is too small a sample regardless of overlap.
3. A support-bot A/B test shows treatment B beating control A by 12 percentage points on 40 conversations per arm. What is the strongest concern?
   • A. The metric was not decided in advance.
   • B. The sample size is small enough that the gap could be noise.
   • C. A/B testing does not apply to conversational systems.
   • D. Treatment B should automatically be shipped.
4. A model scores 94% on its own training set and 71% on a genuinely held-out set. What does the gap most likely indicate?
   • A. The model is exceptionally strong.
   • B. The held-out set was unusually difficult.
   • C. The training-set score reflects memorization, not generalization.
   • D. The two scores should be averaged for a fair estimate.
5. Which of the following best describes reproducibility in an ML experiment?
   • A. Getting a similar-looking result on a rerun.
   • B. Fixing seeds, pinning versions, and recording the full configuration so a rerun matches exactly.
   • C. Publishing the final metric without the configuration that produced it.
   • D. Running the experiment only once, carefully.
6. A team picks whichever of three candidate metrics shows the biggest improvement, after already seeing all three results. What principle does this violate?
   • A. Fixed evaluation sets.
   • B. One variable at a time.
   • C. Deciding the comparison metric in advance.
   • D. Reproducibility.
7. Which comparison is soundly designed, assuming everything not mentioned is held constant and recorded?
   • A. New context embedding, more denoising steps, same evaluation set.
   • B. New context embedding only, same evaluation set, same seed policy.
   • C. New context embedding, same evaluation set, but no seed recorded.
   • D. New context embedding, a different but similarly sized evaluation set.
8. Why does randomization matter in an A/B test's traffic split?
   • A. It makes the sample size larger automatically.
   • B. It guarantees a statistically significant result.
   • C. It spreads confounding factors evenly across control and treatment so they are not the explanation for an observed difference.
   • D. It removes the need for a fixed metric.

Answers

1. C. Two factors changed in the same run, so neither the learning rate nor the optimizer alone can be credited, and there is no basis for claiming an equal split between them — option D asserts a specific attribution the single run cannot support.
2. C. Even drawn from the same pool, two different sampled sets are not the identical evaluation set; any score difference is entangled with which specific items each variant happened to be scored against, regardless of sample size (which is a separate, additional concern raised in D).
3. B. Forty conversations per arm is a small sample; a 12-point gap at that size is exactly the shape of result the source material's named trap warns about — real-looking, but not yet distinguishable from variance. A is a valid general concern but is not what makes this specific result suspect if the metric actually was pre-registered; nothing in the scenario suggests it was not.
4. C. A large gap between training-set and held-out performance is the classic signature of evaluating (or overfitting) on data the model has already seen, which inflates the training-set score without reflecting real generalization.
5. B. Reproducibility is specifically about being able to exactly rerun and match a result via fixed seeds, pinned versions, and full recorded configuration — a "similar-looking" rerun (A) is a weaker, unverifiable claim than reproducibility requires.
6. C. Selecting the metric after seeing which one favors the desired outcome is the opposite of pre-registering a metric; it turns the evaluation into a search for a favorable number rather than a fixed test.
7. B. Exactly one variable changed (the context embedding), the evaluation set is fixed and shared, and the seed policy is accounted for. A changes two variables; C leaves reproducibility incomplete; D breaks the fixed-evaluation-set requirement.
8. C. Randomization is the mechanism that, in expectation, distributes any systematic differences between groups evenly across control and treatment, so the treatment itself remains the most likely explanation for an observed gap — it does not by itself guarantee significance (B) or replace a fixed metric (D).

This module now turns from designing a fair comparison to feeding it the right material. Next: M3-02 covers managing and preprocessing multimodal data pulled from multiple sources — text, images, audio, and more — including the alignment step of pairing a caption with its image before that data ever reaches the augmentation or training pipeline this lesson's experiments would run against.