Reproducibility in LLM Evaluation: Why Temperature 0 Is Not Deterministic

Reproducibility at temperature 0 means: given the identical prompt, identical model weights, and a deterministic selection rule, you should get the identical output — but in practice you usually only get close to identical, because the numbers the selection rule operates on are not guaranteed to be bit-identical across runs. Temperature 0 removes one source of variation — deliberate random sampling — and leaves several others untouched.

Three words that get used loosely and mean different things here:

Temperature 0 buys you a deterministic rule. It does not automatically buy you a reproducible result, because the rule is only as reproducible as its inputs — the logits — and those logits are the output of billions of floating-point operations whose order is not fully fixed by the model architecture alone. It is fixed by the serving stack: which kernel ran, how requests were batched together, which GPU architecture executed the operation, and which library version compiled the kernel. Change any of those without changing the model or the prompt, and you can change the logits by an amount too small to notice on most tokens and just large enough, occasionally, to flip which token is the argmax.

04-05 established the selection side of this story: greedy decoding, and temperature approaching 0, both collapse to taking the argmax, and that lesson flagged — without unpacking — that "greedy is a deterministic rule, not a guarantee of identical text across runs." This lesson is the unpacking. It matters for evaluation specifically because Module 9's whole premise is that a number should move when, and only when, something real changed. Reproducibility failures are the case where the number moves and nothing real changed at all — a confound that has to be measured and subtracted out before any other conclusion in this module is trustworthy.

L1 — Intuition: the rule is fixed, the arithmetic is not

Think of the model's forward pass as an enormous sum: attention scores, weighted sums over values, layer outputs, all built from additions and multiplications of floating-point numbers. Floating-point numbers have finite precision, so (a + b) + c is not always exactly equal to a + (b + c) — the order in which you add matters, at the level of the last few bits. On a CPU running one operation at a time, that order is fixed and boring. On a GPU running a model in production, that order is not fixed, because a GPU parallelizes reductions (sums across many elements) across thousands of threads, and the order in which partial sums are combined can depend on how the work happened to be scheduled — which in turn can depend on how many other requests were running alongside yours, what batch size the request landed in, and which kernel implementation the runtime chose for that batch shape. Argmax is deterministic given its inputs; the inputs are what move.

L2 — Mechanism: the specific sources, named

Floating-point non-associativity. This is the root cause, and everything else on this list is a way of making it visible. Modern accelerators use reduced or mixed precision (12-01) for speed, and lower precision means fewer bits to absorb rounding error, so the same mathematically-equal computation performed in a different order can land on a measurably different value. It is not a bug; it is a property of finite-precision arithmetic that every numerical computing system has, and GPUs make it more visible than CPUs because they parallelize the reductions that expose it.

Batching effects. Production inference servers batch requests together for throughput — a request rarely runs alone. Which other requests happen to be in your batch, and how large that batch is, can change which kernel the runtime selects and how the reduction is tiled across the hardware. Two calls with the identical prompt, fired a few seconds apart, can land in differently-composed batches and come out with logits that differ in the last few bits. Continuous or in-flight batching — the scheduling strategy used to keep GPUs busy under varying request loads (12-06) — makes this more likely than static batching, precisely because it is designed to change batch composition dynamically as requests arrive and finish.

Kernel and algorithm selection. Libraries like cuBLAS and cuDNN often have several algorithms available for the same mathematical operation, and the runtime can select among them based on input shape, batch size, or a tuning heuristic. Different algorithm, different summation order, different rounding, and — because two implementations of the same operation are not guaranteed to be bit-identical — a different result at the margins.

Hardware and driver differences. A different GPU generation, a different driver version, or a different CUDA/library version can all change kernel implementations, numeric behaviors, or default precision handling, even when the model weights are byte-identical. This is why "we ran it on the same model" is not the same claim as "we ran it on the same stack."

Distributed and parallel execution. When a model is split across multiple devices — tensor or pipeline parallelism (12-04) — results from each device have to be combined, typically through collective operations like AllReduce. The order in which partial results from different devices are combined is itself a reduction, subject to the same non-associativity, and can vary with network conditions, device count, or scheduling.

Quantization. Lower-precision formats (12-01, 12-02) compress the numeric range a model operates in. This does not just add generic rounding noise; it can make the model measurably more sensitive to small perturbations, because there is less headroom between distinct representable values. A model already running near a decision boundary at full precision may cross that boundary more often once quantized.

Provider-side change. For hosted APIs, "the same model" is a claim about a name, not a byte-for-byte weight file. Providers roll out kernel optimizations, serving-stack updates, and sometimes silent model refreshes behind a stable model name. None of this is visible to the caller, and it is the single largest reproducibility risk for anyone building on a hosted API rather than self-hosting pinned weights.

Speculative decoding and other decode-time optimizations. Techniques that use a smaller draft model to propose tokens, verified in bulk by the full model (an inference-optimization technique, not a decoding-parameter choice), introduce an accept/reject step whose numerical comparisons are subject to the same floating-point sensitivity, so enabling or disabling this optimization is itself a variable that can change output even though the "settings" a caller sees look unchanged.

L3 — Depth: how a tiny cause becomes a total divergence

Here is the mechanism that makes this a real evaluation problem rather than an academic footnote: generation is append-only and autoregressive (04-04). Every generated token becomes part of the context for the next one. So if a perturbation of a few parts in a billion happens to flip which of two nearly-tied logits is larger at token 12, the model does not produce "the same sentence with one word swapped." It produces a different token at position 12, which changes the context for position 13, which can change that token's distribution, and the divergence compounds forward through the rest of the generation. Two runs of the identical prompt at temperature 0 can therefore diverge into visibly different completions, even though every single decoding step obeyed the argmax rule perfectly. Small numerical cause, large textual effect — the compounding is a property of autoregression, not of the arithmetic error itself.

This also explains why the effect is intermittent rather than constant. Most tokens in most generations are not close calls — one token dominates the distribution by a wide margin, and a perturbation in the ninth decimal place changes nothing about which one is the argmax. The divergence only shows up at genuine near-ties, which is why identical prompts usually return identical output at temperature 0 and only occasionally do not. That intermittency is exactly what makes it dangerous for evaluation: it does not fail loudly and consistently, it fails rarely and unpredictably, which is the profile of a bug nobody looks for until a regression suite starts flaking for no visible reason.

What actually mitigates it

None of these give you a mathematical guarantee of bit-exact reproducibility on a real serving stack, but each reduces the practical risk:

• Pin everything, not just the model. Model version, library versions, hardware type, and precision mode all belong in your evaluation's recorded configuration. "GPT-model-X" is not a reproducible configuration; "GPT-model-X, provider API version Y, called on date Z" is closer, and self-hosting a pinned weight file on pinned software is closer still.
• Disable dynamic batching for evaluation runs where you can. Running eval calls one at a time, or in a fixed static batch, removes the largest source of run-to-run variance for the price of throughput.
• Use deterministic-algorithm modes where the framework offers them. Some deep learning frameworks expose a flag that forces deterministic kernel selection at a performance cost. It narrows, but does not eliminate, the gap — distributed reduction and provider-side change are outside its reach.
• Measure your noise floor, don't assume it. Run the identical configuration twice and look at the gap between the two runs. 09-09 calls this the run-to-run noise floor, and it is the single highest-value ten minutes in an evaluation project, because it converts "we don't know if this is reproducible" into a number you can compare future changes against.
• Evaluate with tolerance, not equality. Compare distributions of outcomes, or scores within a band, rather than demanding character-for-character identical text between runs. 10-04 treats this as the working assumption for LLM regression testing in CI: pass/fail thresholds and score tolerances, never string equality against a golden output.
• Average over repeated samples when the stakes justify the cost. For a high-stakes comparison, running each configuration multiple times and reporting a mean and spread is more honest than reporting a single run as if it were a fixed number.

These four words are often used as synonyms in casual writing about LLMs, and the exam-relevant distinction is that they are properties of different things.

The exam-relevant sentence to hold onto: "deterministic decoding" describes the rule, not a guarantee about the system. A question that offers "greedy decoding at temperature 0 guarantees identical output across runs" as true is testing exactly this confusion, and the correct answer treats it as false in general, true only under additional, usually-unstated conditions (identical hardware, identical batching, identical software stack, no provider-side change).

Constructed numbers throughout — built to make the mechanism visible, not measured from any real model or API call.

Suppose at some generation step two tokens, A and B, have logits that are extremely close — close enough that the eventual choice between them is a near-tie. Illustrative raw logits before any rounding: A = 7.481203 and B = 7.481198, a gap of 0.000005. Under exact arithmetic, A wins by a hair, every time, forever. Now introduce two different summation orders that a real GPU kernel might use to compute the pre-softmax score, each accumulating the same eight partial contributions but in a different sequence:

Contributions to logit A (same eight partial sums, two accumulation orders):

Order 1 (left-to-right):
  ((((((( 1.02 + 0.87 ) + 1.55 ) + 0.91 ) + 1.10 ) + 0.77 ) + 0.64 ) + 1.20 )  → 7.481203

Order 2 (pairwise / tree reduction, as a parallel reducer would do it):
  ( (1.02 + 0.87) + (1.55 + 0.91) ) + ( (1.10 + 0.77) + (0.64 + 1.20) )         → 7.481187

Both orders sum the identical eight numbers. Under exact real-number arithmetic they are equal. Under finite-precision floating point, rounding happens at each intermediate step, and the accumulated rounding differs between the two orders — here, illustratively, by 0.000016, which is well within the kind of gap real floating-point reordering produces at the precision levels used for inference. That is now large enough to matter, because it lands right on top of the original A-vs-B gap of 0.000005:

Order 1 result for A:  7.481203   vs   B:  7.481198   →  A wins by 0.000005
Order 2 result for A:  7.481187   vs   B:  7.481198   →  B wins by 0.000011

The ranking flips. Argmax is still a perfectly deterministic rule — given Order 1's numbers, it always picks A; given Order 2's numbers, it always picks B — but which numbers you get depended on an accumulation order chosen by the kernel, not by you, not by the prompt, and not by the model's weights.

Propagating the flip. Say token A was "is" and token B was "was." The sentence continues from whichever one got selected: "The result is significant" versus "The result was significant". Every token generated after that point conditions on a different preceding context, so the two completions can diverge further from there — not because anything about the model changed, but because one three-hundred-thousandths-of-a-point numerical wobble, seven tokens in, propagated through an autoregressive loop that has no mechanism to "correct back" toward the other branch.

What this means for evaluation practice. If your metric is exact string match (09-06) or an automated pass/fail check keyed to specific wording, a single early-token flip like this can turn a "pass" into a "fail" with the model's actual competence completely unchanged. If your metric tolerates paraphrase — human rubric scoring, BERTScore, or an LLM judge scoring for meaning rather than wording — the flip is far more likely to be invisible, because "The result is significant" and "The result was significant" typically score the same on any rubric that isn't checking literal phrasing. This is a genuine, practical argument for preferring meaning-tolerant metrics for anything you plan to re-run more than once — not because they are more accurate in principle, but because they are less sensitive to a class of noise that has nothing to do with model quality.

The generalizable rule: decide, before you run anything, whether the thing you are protecting is a specific output or an aggregate conclusion. Aggregate conclusions — is model B better than model A on this task — can and should be made robust to run-to-run noise using the statistical machinery of 09-09. Specific outputs — this exact decision, this exact generated text, this exact blocked attack — need the output itself preserved, because no amount of statistics recovers a single instance that was never logged.

The Experimentation domain's real scope, recovered from its scope statement and reading list after the printed-objective defect documented across this module (09-09 covers the defect itself in full — objectives 3.1–3.5 as printed are a verbatim duplicate of the Data Analysis 2.1–2.5 objectives, so the domain's actual coverage of model evaluation is derived from its own definition rather than read literally from the objective text), includes evaluating and interpreting experiments correctly. A candidate who believes temperature 0 guarantees identical output will misjudge a regression-test failure, misjudge a benchmark comparison, and misjudge whether a reported improvement is real — all core Experimentation-domain failure modes. This lesson's content also serves objective 4.5 (monitor the functioning of data collection, experiments, and other software processes — a flaky eval pipeline is exactly the kind of process malfunction that objective describes) and 4.7 (write software components or scripts under supervision — an evaluation harness that assumes bit-exact reproducibility is a script written on a false assumption).

Question phrasings to expect:

• "Does temperature 0 guarantee identical output across two runs of the same prompt on the same model?" → No — it guarantees a deterministic selection rule, but the underlying computation is not guaranteed to be bit-identical due to floating-point non-associativity, batching, and hardware/kernel differences.
• "Why can floating-point arithmetic on a GPU produce different results across runs for the same computation?" → Floating-point addition is not associative; parallel reduction order can vary with batching, kernel selection, and hardware, producing tiny numeric differences.
• "A regression test suite for an LLM feature occasionally fails with no code changes. What is the most likely cause?" → Inference-time non-determinism (batching/kernel/hardware effects), not a real regression; the fix is tolerance-based testing, not string equality.
• "What is the difference between a deterministic decoding rule and a reproducible result?" → Deterministic describes the rule (same inputs, same output); reproducible describes the whole pipeline's outcome, which depends on inputs staying the same too.
• "How should you test whether an observed score improvement is real given run-to-run noise?" → Measure the noise floor by re-running the identical baseline configuration, and only trust improvements that clearly exceed it (09-09).
• "Which decoding parameter change removes randomness from sampling?" → Setting temperature to 0 (equivalent to greedy decoding), while noting this does not eliminate all sources of output variation.

Distractor families, and why each is wrong:

Closer to it, but not perfectly, and the cost rises steeply as you approach it. Self-hosting a pinned model on pinned hardware, pinned library versions, with dynamic batching disabled and any deterministic-algorithm flags enabled, removes most of the practical sources described above and gets you very close to reproducible behavior in day-to-day use. What remains resistant even then: distributed-inference reduction order if the model is sharded across devices, any residual kernel non-determinism the framework does not expose a switch for, and the simple fact that "very close to reproducible" is a probabilistic claim, not a proof — you have made near-ties rare, not impossible. The honest framing for an evaluation pipeline is not "our system is deterministic" but "our system's run-to-run noise floor, measured directly, is this small" — a number you can actually defend, unlike a guarantee you cannot.

Because temperature 0 fixes the selection rule (always take the highest-scoring token) without fixing the inputs to that rule (the exact logits), and the inputs can shift by tiny amounts between calls due to floating-point non-associativity interacting with batching, kernel selection, hardware, and — for hosted APIs — provider-side changes you cannot see. Most tokens in most completions are not close calls, so most of the time this shift changes nothing. Occasionally, at a genuine near-tie, it flips the winning token, and because generation is autoregressive and append-only, that single flip can cascade into a visibly different rest-of-completion, even though every individual step still obeyed argmax perfectly.

Treat reproducibility as a measured property of your pipeline, not an assumed one, and build around a small set of habits: freeze and version your evaluation set (09-01); log the full configuration — model version, inference stack, hardware, decoding parameters — with every run, not just the score; measure your run-to-run noise floor directly by re-running an unchanged baseline before trusting any comparison (09-09); prefer tolerance-based or semantic-similarity comparisons over exact string matching wherever the task allows it, because they absorb the harmless variety of surface-level flips this lesson describes; disable dynamic batching for evaluation runs when your infrastructure allows it, trading some throughput for a smaller noise floor; and, for anything with legal, safety, or compliance weight, log the actual generated output alongside the score, because a statistic computed today cannot reconstruct an exact decision made yesterday. None of this makes the system perfectly deterministic. All of it converts an unmeasured risk into a known, bounded, and defensible number — which is the actual goal, because "fully reproducible" was never on offer.

• Deterministic (selection rule) — same inputs always produce the same output; argmax and greedy decoding are deterministic rules.
• Reproducible (result) — re-running an entire pipeline, including its infrastructure, yields the same output; a stronger and harder-won property than rule-level determinism.
• Floating-point non-associativity — the fact that (a+b)+c is not always bit-identical to a+(b+c) under finite-precision arithmetic, so summation order affects the result.
• Batching effects — variation in which other requests share a batch, and how large that batch is, which can change kernel selection and reduction order.
• Continuous / in-flight batching — a dynamic batching strategy that changes batch composition as requests arrive and finish, increasing the practical chance of run-to-run numeric variation.
• Kernel/algorithm selection — a runtime's choice among multiple valid implementations of the same operation, which can differ in numeric behavior at the margins.
• AllReduce / distributed reduction order — the combination of partial results across devices in distributed inference, itself a non-associative summation and a source of run-to-run variance.
• Provider-side change — an update to a hosted API's serving stack or underlying model behind a stable model name, invisible to the caller.
• Run-to-run noise floor — the score difference observed when the identical configuration is evaluated twice; the empirical baseline against which any claimed improvement must be judged.
• Stability (of a metric) — small input perturbations produce small score changes rather than a cliff.
• Consistency (of an outcome) — agreement on the conclusion (label, decision, answer) even when exact wording varies.

1. Determinism is a property of the selection rule; reproducibility is a property of the whole result. Temperature 0 buys you the first, not automatically the second.
2. Floating-point non-associativity is the root cause. Summation order affects the result at the margins, and GPUs make that order visible through parallel reduction.
3. Batching, kernel selection, hardware, distributed reduction, quantization, and provider-side updates are the concrete channels through which non-associativity turns into an observable output difference.
4. Generation is autoregressive and append-only, so a single early near-tie flip can cascade into a visibly different full completion, even though every step obeyed argmax exactly.
5. The effect is intermittent, not constant — most tokens are not close calls — which is exactly what makes it dangerous: it fails rarely and unpredictably rather than loudly and consistently.
6. Measure your noise floor directly. Re-run the identical configuration and treat the observed gap as the smallest difference you can trust (09-09).
7. Evaluate with tolerance, not equality. Exact-string-match regression tests are brittle against this class of noise; threshold- and meaning-based comparisons are not (10-04).
8. Log full configuration and, for high-stakes cases, full output — a score alone cannot reconstruct a specific past decision.
9. You can get close to deterministic, never perfectly. Pinning weights, hardware, and software, and disabling dynamic batching, shrinks the noise floor; it does not zero it.
10. Decide upfront whether you are protecting an aggregate conclusion or a specific output. Aggregates should be made robust with statistics; specific outputs must be logged, because they cannot be statistically recovered later.

Reproducibility failures produce a wrong-seeming score even when the model's underlying competence has not changed — noise masquerading as signal. The next lesson covers the opposite and much larger problem: outputs that are wrong in substance, stated with the same fluent confidence as a correct answer, for reasons that have nothing to do with floating-point arithmetic and everything to do with how a language model generates text in the first place.

Next: 09-12 covers why LLMs hallucinate, the distinct types of hallucination worth naming separately, and the mitigation ladder — grounding, citation, constrained decoding, guardrails, and human review — that turns "the model made something up" from a mystery into a diagnosable, addressable failure mode.