Temperature 0 is not reproducible. I measured 30 percent of my output changing between identical runs.

I set temperature to 0, ran the same document through the same model twice with the same prompt, and got different answers. Not subtly different. Roughly a third of the extracted records changed. The cause is not sampling. It is batching. | same document, same model, same prompt, temperature 0 | records | byte-identical | Jaccard | |---|---|---|---| --concurrency 1 , run twice | 137 / 137 | yes | 1.0000 | --concurrency 4 , run twice | 160 / 164 | no | 0.705 | | concurrency 1 vs concurrency 4 | 137 / 160 | no | 0.549 | Serving one request at a time is exactly reproducible. Serving four at a time is not, and it is not reproducible against itself either. Why this happens Batched inference does not compute the same arithmetic as unbatched inference. Different batch shapes select different kernels, matrix multiplications get tiled differently, and floating-point addition is not associative. The logits come out fractionally different, and anywhere two tokens were nearly tied, argmax picks the other one. Temperature 0 removes sampling randomness. It does nothing about the arithmetic underneath. This is well known in the abstract. What surprised me is the size: 30 percent of records, not 3. How I found it, which is the embarrassing part I was not looking for this. I was testing whether a new filter reduced false positives, comparing a run with the filter against a run without it. One name showed up as a false positive only in the filtered run. A filter cannot add records. It can only remove them. So either my filter was broken, or the two runs had not seen the same model output at all. It was the second. My A/B test had been comparing two different underlying extractions and attributing the difference to the flag. That experiment was worthless and I nearly published its result. The claim this retires I had a determinism check in the repo already. It ran the same corpus five times and confirmed the final answer was identical each time. It passed. I cited it. It was measuring almost nothing. The pipeline extracts hundreds of records per document and then reduces them in code to one answer. A reduction like "which of these three names appears in the most sections" is extremely tolerant: you can churn a third of the evidence and still land on the same name, because the argmax has a margin. So the final answer was stable while the thing underneath it was not. The check confirmed the margin was wide, and I read it as confirming the system was deterministic. Record-level agreement is the measurement. Answer-level agreement is a consequence, and a weak one. What it did and did not change I re-ran the headline benchmark at concurrency 1, where output is byte-identical. | metric | batched | deterministic | |---|---|---| | the headline metric | 0.980 | 0.980 | | a positional ordering task | 0.942 | 0.904 | | overall | 0.899 | 0.870 | The headline number did not move at all. That is not luck: it counts how many distinct regions of a document a name appears in, and it does not care which particular quoted span proves the name was in a region, only that some span does. Redundant evidence absorbs the churn. The positional tasks moved by up to 0.04, because those depend on the single earliest or latest piece of evidence, and the extremes are exactly where losing 30 percent of records bites. So the honest summary is: the noise is real, it reaches some metrics and not others, and you cannot know which without measuring. What to do Report concurrency with your numbers. It is a experimental condition, not a performance tuning detail. Two papers can differ on it and disagree for no other reason. Measure agreement at the record level, not the answer level. Run the same input twice, diff the intermediate output, report Jaccard. If your pipeline has no intermediate output to diff, that is worth fixing on its own. Run final numbers at concurrency 1. It cost me less than I expected: about 20 minutes to about 23 for a nine-document sweep, because the server was not parallelising as much as the flag implied. Measure the cost before assuming you cannot afford it. Iterate batched, publish serial. Be suspicious of any A/B where the treatment could not have caused the difference you see. That single impossible data point was the only reason I found this. If the filtered run had merely looked better, I would have shipped it. Top comments (0)