Temperature 0 Isn't Deterministic: Why Your LLM Still Drifts

Problem Statement

A CI test that passed consistently for weeks suddenly started failing approximately once a week. The conditions were identical: the same prompt, the same model snapshot, temperature=0, a pinned seed, and an assertion checking the output string. Nothing in the code diff touched any of these elements. After looping the exact same request 500 times and comparing the results, twelve distinct outputs emerged. One example classified a refund ticket as billing instead of fraud-a genuine behavioral change from a byte-identical request.

Why Temperature = 0 Isn't Deterministic

Temperature 0 makes sampling greedy (always picking the top token), but it does not guarantee identical logits between runs. The root cause lies in how floating-point arithmetic behaves during inference.

Floating-Point Non-Associativity

Floating-point addition is not associative. Expressions such as (a + b) + c and a + (b + c) can differ in the last bits. On a CPU running a single thread, the reduction order rarely changes, so this effect is invisible. However, on GPUs, reductions (such as summing across a hidden dimension, computing softmax denominators, or applying RMSNorm) are split across multiple blocks and combined later. The way the tensor is tiled depends on the batch shape, which itself varies based on concurrent traffic.

Batch Invariance

Batch invariance means a single request produces the same output regardless of other requests batched alongside it. Most production inference kernels do not possess this property-it is a deliberate performance trade-off. Serving stacks batch aggressively, so the same request at 3 pm might land in a batch of 48 others, while the same request at 4 am could land in a batch of 3. Each different batch size triggers a different tiling strategy, split‑K choice within the matrix multiplication, reduction order, and final last‑bit rounding of the logits. Consequently, even though each individual run is fully deterministic given its specific batch, the batch composition is outside the developer's control and cannot be observed.

Factors That Compound the Flakiness

Several additional mechanisms amplify the small differences introduced by variable batch sizes:

• Mixture-of-experts routing: Capacity limits mean which tokens reach which expert depend on neighboring tokens competing for the same expert within the same batch. Your token's neighbors are essentially strangers.
• Prefix caching: Whether your prompt hits a cached key-value prefix changes the compute boundary, which alters the reduction grouping for subsequent tokens.
• Speculative decoding: Although designed to be output‑equivalent, the verification path follows a different numeric route, and equivalence holds only up to the same last‑bit tolerances.
• Silent fleet heterogeneity: Two different GPU generations behind a single endpoint may select different kernel variants for identical input. The seed parameter on major APIs is documented as "best effort" precisely because it pins the sampler's RNG but does not pin the arithmetic.

Where the Divergence Happens

Most positions in a generated sequence are far from ties-for example, the phrase "the capital of France is Paris" has a massive gap between the top token (Paris) and the runner‑up (Lyon), making a 1e‑7 perturbation ineffective. Those tokens are effectively locked. However, a small subset of positions consists of near‑ties: "however" vs “but”, { vs [, or fraud vs billing on an ambiguous ticket. At these positions the top‑2 gap is smaller than typical numeric noise, and the argmax genuinely coin‑flips. Once the model commits to one option-such as choosing “however”-it conditions every subsequent token on that choice, producing a completely different second half of the answer.

This explains why failures cluster on the most difficult inputs. Ambiguous, low‑confidence prompts are exactly where near‑ties reside, causing the occasional flaky test that reads like a haunting.

Making LLM Output Reproducible

On a hosted API, true bitwise reproducibility is largely impossible without owning the entire stack. Instead, focus on robustness and measurable confidence:

• Stop asserting on strings. Parse the output and verify the relevant fields. For example, the fraud test should check parsed.label in ALLOWED and parsed.label == "fraud" rather than relying on the surrounding prose.
• Pin the dated model snapshot, not the alias. Aliases can silently rotate, introducing another source of drift that is mistaken for the above issue.
• Shrink the output space for critical assertions. A classifier that emits a single enum token has far fewer near‑tie positions than a model that embeds the label deep within a long paragraph.
• Measure the flip rate instead of pretending it is zero. Run your evaluation set multiple times, record the disagreement rate per item, and treat items that flip as low‑confidence rather than passing or failing outright.
• Use K‑of‑N for important decisions. If three independent runs disagree on a refund classification, treat it as a routing signal for human review rather than attempting to retry until success.
• Log the response fingerprint (e.g., system_fingerprint or equivalent). This lets you distinguish backend changes from actual near‑tie artifacts.

If you require absolute bitwise reproducibility, run locally with batch size 1, fixed library and driver versions, deterministic kernel flags, and a single GPU model. Even then, reproducibility applies only to that specific environment. Batch‑invariant kernels are emerging in some serving stacks but incur throughput costs, so you should assume they are not available unless explicitly provided by a vendor.

Summary

Temperature 0 controls the sampler, not the arithmetic, and the arithmetic depends on server load. The common misconception is that the model is "still being creative"-no, the model is not creative. The variation comes from floating‑point non‑associativity, GPU kernel reduction strategies that vary with batch shape, and the lack of batch invariance in most production serving stacks. Ambiguous prompts that sit near decision boundaries are where these effects manifest. The practical takeaway: design your tests around parsed fields and invariants, pin your model version, reduce output complexity, and accept that on shared endpoints you must work with a measured flip rate rather than seeking perfect determinism.