Hyperparameter Tuning: Grid Search, Random Search, and Bayesian Optimization

Identity statement: a hyperparameter is a value that is set before training begins and controls how training proceeds, as distinct from a parameter (a weight), which is learned during training by gradient descent adjusting it in response to the loss.

When it matters: any time a scenario describes choosing a training-run setting — learning rate, batch size, number of layers, LoRA rank — rather than describing what the model itself learned.

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "Hyperparameters are set before training (not learned): learning rate, batch size, epochs, optimizer, dropout, LoRA rank, number of layers/heads." Each of these controls some aspect of how training happens without itself being adjusted by the training process. Learning rate sets how large a step the optimizer takes at each update. Batch size sets how many examples contribute to each gradient estimate. Epochs sets how many full passes through the training data occur. The optimizer choice (SGD, Adam, and their variants) sets the update rule itself. Dropout sets a regularization rate. LoRA rank sets the size of a parameter-efficient adaptation's low-rank matrices. Number of layers and heads sets the model's own architecture. None of these seven changes during training the way a weight does — they are chosen once, in advance (or occasionally adjusted on a fixed schedule decided in advance, as with a learning-rate schedule), and the training run then unfolds under whatever choice was made.

This distinction matters on the exam because a question can describe a value changing over the course of training and ask whether that value is a hyperparameter or a parameter — a weight changes continuously in response to gradients, while a hyperparameter, even one following a schedule, changes according to a rule decided before training started, not in response to what the loss did. The practical stakes are just as real outside the exam: a wrong hyperparameter choice does not get corrected by training the way a wrong weight does, because nothing in the gradient-descent loop is evaluating whether the learning rate or batch size was a good choice — that evaluation is external, and it is what tuning exists to do.

L1 — Intuition: exhaustive, random, and informed

Imagine trying to find the best combination of two settings, each with several candidate values, without knowing in advance which combination works best. Grid search tries every combination methodically, like checking every square on a checkerboard. Random search samples combinations without any pattern, like throwing darts at the board. Bayesian optimization uses what it has already learned from earlier throws to aim the next dart somewhere more promising, updating its aim after every result.

L2 — Mechanism: how each strategy actually chooses its next configuration

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) names the three strategies directly:

Grid search defines a discrete set of candidate values for each hyperparameter — say, learning rates {0.1, 0.01, 0.001} and batch sizes {32, 64, 128} — and trains a model for every combination in the resulting grid, nine combinations in this example. It is exhaustive within the grid it was given: every combination is tried, so it cannot miss a good setting that happens to fall inside the grid. Its cost grows multiplicatively with both the number of hyperparameters being tuned and the number of candidate values per hyperparameter, which is precisely why [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) describes it as "expensive": tuning five hyperparameters with five candidate values each is 5^5 = 3,125 combinations, each requiring a full or partial training run.

Random search samples configurations from each hyperparameter's range at random, rather than following a fixed grid, and runs a fixed budget of trials — say, 50 random configurations — regardless of how many hyperparameters are involved. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) describes it as "often more efficient in high dimensions," and the reason is a specific, well-documented property: in most real hyperparameter spaces, only a few of the hyperparameters actually matter much for the outcome, and grid search wastes a large share of its combinations varying the unimportant ones in lockstep with the important ones, while random search's independent sampling of each dimension explores the important dimensions' range more thoroughly for the same total number of trials.

Bayesian optimization builds a probabilistic model of how the objective (validation loss or accuracy) depends on the hyperparameters, using the results of trials already run, and uses that model to choose the next configuration to try — typically balancing configurations the model predicts will perform well against configurations the model is still uncertain about, so the search does not get stuck exploiting an early lucky result without checking whether an unexplored region might be better. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) names this "sample-efficient": because each new trial is chosen using everything learned from every previous trial, Bayesian optimization typically needs fewer total trials to reach a good configuration than either grid or random search, at the cost of more computation spent choosing each next point and a search that is inherently sequential rather than trivially parallelizable the way grid and random search are.

L3 — The exam-relevant edge case: why "more efficient" does not mean "always the right choice"

A ranking that reads grid < random < Bayesian in sample efficiency is accurate but incomplete, and the incompleteness is exactly what the exam's scenario items probe. Grid search remains the right choice when the search space is small (one or two hyperparameters with a handful of candidate values each) and trials can run fully in parallel with no need to learn from earlier results — its exhaustiveness is then not wasteful, because there is nothing wasteful about trying nine combinations when nine trials is the whole budget anyway. Random search's advantage specifically requires a higher-dimensional space (several hyperparameters at once) to materialize; in a one-or-two-dimensional space with few candidate values, grid and random search perform comparably. Bayesian optimization's sequential dependency — each trial needs the previous trials' results before it can be chosen — is a genuine cost when trial parallelism matters more than trial count, because a team that can run 100 trials in parallel overnight may reach a better result faster with random search's 100 independent trials than with Bayesian optimization's sequential, one-at-a-time refinement over the same wall-clock budget.

⭐ THE EARNED INSIGHT

"Sample-efficient" describes how few trials a strategy needs, not how fast a search finishes in wall-clock time, and those two things point in opposite directions once parallelism enters the picture. Bayesian optimization wins the first measure and can lose the second, because needing fewer trials is worth nothing if each trial must wait for the last one to finish, while a strategy that needs more trials but can run all of them at once may finish sooner regardless. Choosing a search strategy is therefore a question about the shape of your compute budget — sequential or parallel — as much as it is a question about how many hyperparameters you are tuning.

Constructed scenario, with every figure derived from stated assumptions rather than measured on a real tuning run. A team is tuning three hyperparameters for a multimodal fusion layer: learning rate (5 candidate values), a loss-weighting coefficient between the vision and text loss terms (4 candidate values), and dropout rate (3 candidate values). Each training run to evaluate one configuration costs 2 GPU-hours.

Grid search:
  Total combinations: 5 x 4 x 3 = 60
  Total cost: 60 x 2 GPU-hours = 120 GPU-hours
  Guarantee: every combination in the grid is evaluated.

Random search, budget capped at 20 trials:
  Total cost: 20 x 2 GPU-hours = 40 GPU-hours
  Guarantee: none, but 20 trials sampled across the same three
  ranges typically finds a configuration close to the grid search's
  best result, because — per the source material's own framing —
  usually only one or two of the three hyperparameters dominate the
  outcome, and random sampling explores each dimension's full range
  rather than the grid's fixed checkpoints.

Bayesian optimization, budget capped at 12 trials:
  Total cost: 12 x 2 GPU-hours = 24 GPU-hours, plus the (comparatively
  small) compute cost of fitting the probabilistic model between
  trials.
  Guarantee: none, but each of the 12 trials is chosen using every
  prior trial's result, so a comparable or better result than random
  search's 20 trials is a realistic expectation for this budget.

The pattern to generalize is not the specific ratios — those are illustrative — but the shape: as the search space is held fixed and the strategy moves from grid to random to Bayesian, the number of trials needed to reach a comparably good configuration tends to fall, while the amount of computation and design effort spent choosing each individual trial tends to rise. Grid search's 120 GPU-hours buys an exhaustive guarantee; Bayesian optimization's 24 GPU-hours buys a probabilistic bet that its informed choices found nearly as good a region without checking every point in it.

Notice also what the 60-combination grid actually contains: because the grid is the Cartesian product of all three ranges, most of its 60 trials vary the dropout rate (3 values) while holding the learning rate and loss-weighting coefficient fixed at values that later turn out not to matter much — a pattern that holds whenever, as is typical, only one or two of several tuned hyperparameters dominate the outcome. Random search's 20 trials sample dropout, learning rate, and the loss-weighting coefficient independently and uniformly across their full ranges on every single trial, so the same 20 trials spend proportionally more of their sampling budget on whichever dimension turns out to matter, without needing to know in advance which one that will be. This is the concrete mechanism behind the "often more efficient in high dimensions" claim from section 2's L2 discussion, made visible against one specific search space rather than left as an abstract property.

[GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md): "The learning rate is the most sensitive: too high diverges, too low stalls." Both failure modes have a recognizable shape once plotted, and a scenario question frequently describes the shape rather than naming the learning rate directly.

Run A (learning rate too high):
  Step    100    500   1,000   2,000   5,000
  Loss    4.20   6.80  11.40   NaN     NaN
  -> Loss increases rather than decreases, then diverges to NaN.
     The optimizer's steps are overshooting the loss surface's
     minimum by a growing margin each time.

Run B (learning rate too low):
  Step    100    500   1,000   2,000   5,000
  Loss    4.20   4.05   3.98    3.95    3.93
  -> Loss decreases, but by a vanishingly small amount each step.
     After 5,000 steps the model has barely moved from its starting
     point, and a properly-tuned run would have reached a
     meaningfully lower loss in a fraction of that step count.

Run C (learning rate well-tuned):
  Step    100    500   1,000   2,000   5,000
  Loss    4.20   2.10   1.15    0.62    0.31
  -> Loss falls steadily and substantially, the signature of a
     learning rate that is neither overshooting nor crawling.

Constructed scenario — every loss value here is illustrative, not measured from a real run — but the two failure signatures generalize directly: divergence (loss growing, then NaN) is the signature of too high a learning rate, and near-flatness despite genuine (if tiny) decrease is the signature of too low a learning rate, distinct from the underflow-driven plateau M5-01 describes for a different reason (FP16 gradients rounding to exactly zero) rather than this genuinely-too-small step size. A scenario question describing either signature is testing whether you can name the specific hyperparameter responsible without being told directly which one it is.

A single-modality model's hyperparameter space is already the seven-item list from section 1. A multimodal model adds at least one more dimension worth naming explicitly: the loss-weighting coefficient between its separate loss terms — M1-09's composite-loss material covers why an unweighted sum of a vision loss and a text loss lets the "easier" modality dominate training, and the weighting coefficient that fixes this is itself a hyperparameter, chosen before training and never adjusted by gradient descent. This is inference from the general mechanism of hyperparameter tuning applied to a multimodal composite loss, not a claim traceable to a specific NVIDIA study-guide sentence naming loss-weighting search directly, but it follows the same identity statement from section 1 exactly: a value fixed before training that controls how training proceeds.

Adding this dimension changes which search strategy is worth reaching for, in a way section 2's L3 discussion anticipates directly. A single-modality model tuning only learning rate and batch size is a two-dimensional search, comfortably inside grid search's affordable range. A multimodal model tuning learning rate, batch size, and a loss-weighting coefficient — and, if the vision and text encoders are tuned with separate learning rates rather than one shared rate, a fourth or fifth dimension — crosses into the range where grid search's multiplicative cost growth starts to bite, and where random search's higher-dimensional efficiency advantage, or Bayesian optimization's sample efficiency if each multimodal training run is expensive, becomes the more defensible default. The multimodal-specific lesson is not a new search strategy, it is that multimodal architectures tend to add exactly the kind of extra dimension that pushes a tuning problem out of grid search's comfortable range and into the territory where the choice between random and Bayesian search actually matters.

Performance Optimization is Domain 5 of the NCA-GENM blueprint at 10% weight, and [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) frames tuning as targeting both accuracy and efficiency — throughput, memory, energy — rather than accuracy alone, which is consistent with the domain's guiding question of getting more accuracy per unit of compute, memory, and energy out of a model you already have. Because a poorly-chosen learning rate can waste an entire training run's compute on a model that never converges, tuning is positioned as a performance-optimization technique in its own right, not merely a preliminary step before the "real" optimization work of M5-01 through M5-03 and M5-06.

Questions in this family tend to arrive as a direct identification item naming the three search strategies and asking which one is exhaustive, which samples randomly, and which models the objective — the self-check material's own example asks which hyperparameter is most sensitive, keyed to learning rate — and as a scenario item describing a diverging or stalled loss curve and asking which hyperparameter and which direction of change caused it, in the shape of section 5's worked example. A search-strategy scenario item typically describes a budget and a dimensionality (how many hyperparameters, how much compute, how much parallelism) and asks which strategy suits it, testing the section 2 L3 nuance that "more sample-efficient" is not the same as "always correct."

What the distractors typically look like

Expect an option that describes Bayesian optimization as strictly superior in every situation, omitting its sequential-dependency cost, which is the exact simplification section 2's L3 discussion corrects. Expect grid search's cost described as merely "slower" rather than growing multiplicatively with dimension count, understating how quickly it becomes infeasible past two or three hyperparameters. And expect a hyperparameter offered where a parameter belongs, or vice versa — describing a weight as something "set before training" or a hyperparameter as something "learned by gradient descent" — trading on the fact that both terms sound like technical synonyms for "a setting."

A parameter (a weight) is a value the training process itself adjusts, via gradient descent responding to the loss, and its final value is an output of training. A hyperparameter is a value fixed before training begins — learning rate, batch size, epochs, optimizer choice, dropout, LoRA rank, number of layers or heads — and it is an input to training, not an output of it. M1-05's training loop (loss function, gradient descent, learning rate, backpropagation, optimizer) is the mechanism that adjusts parameters; hyperparameter tuning is the separate, outer-loop process of choosing which values several of those same mechanism's own settings — most obviously the learning rate itself — should use before that inner loop ever runs.

Because both directions of getting it wrong produce a training run that fails to reach a good result, and the failure is often not obvious until well into the run. [GROUND TRUTH] (Sources/nca-genm/domain-5-performance-optimization.md) states the two failure modes directly: too high a learning rate diverges, and too low a learning rate stalls. Other hyperparameters tend to degrade a training run's outcome gradually — an imperfect batch size or a slightly wrong dropout rate typically costs some accuracy without derailing the run entirely — while a badly wrong learning rate can make training fail outright (diverging to NaN) or make it so slow that a run essentially never converges within any reasonable step budget, which is why it is singled out for concentrated tuning attention across grid, random, and Bayesian search alike.

Yes, and doing so is a common practical shortcut, distinct from the question of which search strategy to use. Running each trial of a grid, random, or Bayesian search using M5-01's mixed-precision training reduces the GPU-hours each individual trial costs, which lowers the total budget consumed by whichever search strategy is chosen — it does not change which strategy is preferable, since the relative trial-count and parallelism tradeoffs from section 6's decision table hold regardless of how cheap or expensive an individual trial happens to be. A team might therefore reasonably run a Bayesian search where each trial trains at FP16 rather than FP32, cutting the wall-clock cost of the already-fewer trials Bayesian optimization needs. This is a case where two of this module's techniques compound rather than substitute for each other: tuning decides which hyperparameter values to use, and mixed precision decides how cheaply each candidate value can be tested.

The search-space characteristics matter more directly than the model's identity: how many hyperparameters are being tuned, how expensive each trial is, and how much parallel compute is available are the deciding factors in section 6's decision table, and those characteristics are properties of the tuning budget and infrastructure rather than of the model architecture itself. That said, a very expensive-to-train multimodal model with several separately-tunable components — a learning rate for the vision encoder, a different one for the text encoder, a loss-weighting coefficient between them — tends to push toward Bayesian optimization simply because expensive-to-train models make every wasted trial costlier, which is the same underlying logic as section 6's "each training run is very expensive" row, applied to a multimodal-specific reason a search space might have more dimensions than a single-tower model's tuning problem would.

Yes, and in practice the order matters. A learning rate, batch size, or loss-weighting coefficient found during tuning is chosen for a specific training configuration, and changing that configuration afterward — switching to mixed-precision training (M5-01), for instance — can shift which hyperparameter values actually work best, because a different numeric precision changes how gradients behave near the values loss scaling and FP32 accumulation are protecting. The practical default is to tune hyperparameters within whatever training configuration (precision, in particular) you actually intend to deploy with, rather than tuning under one configuration and assuming the results transfer unchanged to a different one; quantization (M5-02) and pruning (M5-03), by contrast, are applied to an already-trained model and do not interact with the tuning process in the same direct way, since they happen after the hyperparameter-tuned training run is already complete.

Work through each item before checking the answer key. Every option is a real claim about some tuning run somewhere — the task is matching it to the described scenario, not spotting an obviously fabricated distractor.

1. Which of these is a hyperparameter rather than a parameter?
   • A. A specific weight in the fusion layer.
   • B. The learning rate used to train the model.
   • C. A learned attention score.
   • D. A bias term updated during backpropagation.
2. Why is grid search described as expensive?
   • A. It requires more GPU memory per trial than the other two strategies.
   • B. Its trial count grows multiplicatively with the number of hyperparameters and candidate values.
   • C. It cannot be parallelized at all.
   • D. It always requires labeled validation data, unlike random search.
3. A team has a large parallel-compute budget and eight hyperparameters to tune, with no strong reason to believe any single training run is prohibitively expensive. Which strategy best fits?
   • A. Grid search, because it guarantees the global optimum.
   • B. Random search, because its independent trials exploit parallelism and its higher-dimensional efficiency applies here.
   • C. Bayesian optimization, because it is always the most efficient choice regardless of parallelism.
   • D. None of the three — eight hyperparameters is too many to tune at all.
4. A training run's loss grows for several thousand steps and then becomes NaN. What hyperparameter problem does this most likely indicate?
   • A. Batch size too small.
   • B. Learning rate too high.
   • C. Too few epochs.
   • D. Dropout rate too high.
5. Why can Bayesian optimization need fewer trials than random search to reach a comparably good configuration?
   • A. It always trains models faster per trial.
   • B. Each new trial is chosen using a model built from every prior trial's result, rather than sampled independently.
   • C. It restricts the search to a smaller grid than grid search uses.
   • D. It does not require a validation metric.
6. What is the practical downside of Bayesian optimization's sequential dependency?
   • A. It cannot be used for more than three hyperparameters.
   • B. It cannot benefit from a large parallel-compute budget the way grid or random search can.
   • C. It requires more training data than the other two strategies.
   • D. It always produces a worse final result than random search.

Answers

1. B. Learning rate is set before training and is not adjusted by gradient descent; the other three options are all learned parameters.
2. B. This is the multiplicative cost growth section 2's L2 discussion computes directly: five hyperparameters at five values each is 3,125 combinations.
3. B. Large parallel compute and several hyperparameters is exactly the profile where random search's higher-dimensional efficiency and full parallelizability both apply; Bayesian optimization's sequential dependency would waste the available parallelism.
4. B. Growing loss followed by NaN is the diverging signature from section 5's worked example, the specific fingerprint of too high a learning rate overshooting the loss surface's minimum.
5. B. This is the defining mechanism of Bayesian optimization: using a probabilistic model fit to prior results to choose the next, most promising point, rather than sampling blindly.
6. B. Each Bayesian-optimization trial depends on the results of prior trials, so trials cannot all run at once the way grid and random search trials can — the exact tradeoff the earned insight in section 2 names.

• A hyperparameter is set before training and is not learned; a parameter (a weight) is learned by gradient descent during training.
• Grid search is exhaustive but expensive; random search samples and is often more efficient in high dimensions; Bayesian optimization models the objective and is the most sample-efficient of the three.
• Sample efficiency is not the only factor that matters — grid and random search are fully parallelizable, while Bayesian optimization's trials depend sequentially on prior results.
• Learning rate is the most sensitive hyperparameter across every search strategy: too high diverges (loss grows, then NaN), too low stalls (loss barely moves).
• Tuning targets both accuracy and efficiency (throughput, memory, energy), which is why it belongs in a performance-optimization module rather than only in a training-fundamentals one.

This module now turns to a technique that does not touch the model's numbers or its structure at all. Next: M5-05 covers transfer learning for efficiency — why reusing a pretrained encoder is a direct efficiency win in less data, less compute, and lower energy cost, without any of this lesson's search process being required to reach a strong starting point.