Whoever else is in the batch settles a near-tied logit

Turning off the sampler leaves the tokens moving

Two answers to one prompt diverge at a single draw: a sampler picks one token from the probability distribution the model computed, and a different draw picks a different token. Temperature set to zero is supposed to remove the draw entirely. The softmax collapses onto its single largest logit, that token gets a probability of 1, and the sampler has nothing left to choose between. The rule that is left, take whichever token holds the highest score, is called argmax, and it has no dice in it at all.

Send the identical prompt to a hosted model a thousand times at temperature zero, and the identical-argmax rule says the identical completion should come back a thousand times. Thinking Machines Lab ran exactly that experiment in 2025 and got 80 different completions, not one.¹ Argmax did not misbehave. It is a pure function: the same list of numbers in produces the same top entry out, every time. If the winning token changed between runs, the list of numbers changed. Something between the model's fixed weights and the number it hands the sampler is not fixed at all.

Follow that back one step. A token's logit is the last number in a chain of matrix multiplications and normalizations run on the weights and the prompt. Underneath, every one of those operations is a sum: thousands of numbers added together to produce one. If that sum comes out slightly different, the logit comes out slightly different. If two tokens were already close, whichever one now holds the marginally larger number wins the argmax instead of the other. The place to look is not the model. It is the arithmetic.

The order of a sum changes its answer

Ordinary addition does not care what order it happens in: 1 plus 2 plus 3 is 6 whichever pair goes first. The format every number in a neural network is stored in, floating point, does care. Add 0.1, 0.2, and 0.7 as (0.1 + 0.2) + 0.7 on a computer, and it returns exactly 1.0. Add the same three numbers as 0.1 + (0.2 + 0.7), and it returns 0.9999999999999999, one step short of 1.0. Same three numbers, same final operation. Only the order of summation changed, and the two answers land as two different bit patterns in memory: one ending in all zeros, the other ending in all ones.

This is what non-associative means: (a + b) + c does not always equal a + (b + c) once every intermediate result has to be rounded to fit a fixed number of bits. A floating-point number is exactly that, a fixed number of bits.² A study measuring how far this error travels through a deep network found the sensitivity extreme, since the small differences compound across the millions of additions inside every layer.³ One reordered sum, buried inside a single matrix multiplication, is a lever on the final logit.

The kernel's reduction order is not fixed

A single entry in a neural network's output is the sum of thousands of products, one row of weights times one column of input, added together. A GPU does that sum, called a reduction, in parallel: it splits the thousands of terms across many threads and combines their partial sums back into one number. Which partial sums sit next to each other, and in what order they get combined, is a choice made by the kernel, the compiled routine doing the multiplication. That choice depends on the work there is to split. Thinking Machines Lab documents the same normalization kernel switching strategy as batch size moves. At a large batch, "increasing our batch size doesn't affect our reduction strategy; if a batch size of 200 provides sufficient parallelism to our kernel then a batch size of 2000 will definitely provide sufficient parallelism."¹ Shrink the batch and the same kernel switches strategy, because "if we have a small batch size, our data-parallel strategy may no longer have sufficient parallelism to saturate our cores."¹ A kernel tuned to run one request alone splits the work one way. The same kernel handling that request alongside thirty-nine others splits it another way, because the fast tiling for a large batch is not the fast tiling for a small one.

Researchers compared a matrix multiplication run by itself against the identical multiplication run inside a batch. The two invoked different kernels outright, and the resulting error was correlated and directional rather than scattered evenly: for one common number format, close to half the variance in the discrepancy lined up in a structured pattern.⁴ The batch changed which exact computation ran. Floating-point non-associativity and batch-dependent kernel behavior are not in dispute. Both are documented by the frameworks themselves and reproduced independently: this part of the chain is settled engineering.

Who shares your batch decides the order

A production server does not run one request at a time. To keep an expensive GPU busy, it gathers whatever requests arrived in the last few milliseconds into a batch and runs them through the model together. How many other requests land in your batch is a property of the moment your prompt arrived, not of the prompt itself. Thinking Machines Lab states the consequence directly: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load, and thus batch-size, nondeterministically varies."¹ An independent account of the same effect describes a single prompt landing in a batch by itself at some moments, and sharing a batch with dozens of others at other moments. No pattern was visible from outside the server.⁵ Send the identical prompt twice, and each copy can hit a different batch size, triggering a different reduction schedule on the same sum.

Some models route each token to one of several specialist sub-networks, called experts, and batch composition opens a second channel here. Each expert accepts only a limited number of tokens per batch, so which expert a token reaches depends on who else is competing for the same slots. The routing is deterministic at the batch level, one analysis notes. But the token you sent can land on a different expert depending on who else showed up with it, changing the outcome from where the caller sits.⁶

Every step, the arithmetic, the kernel, the routing, is a deterministic function of its inputs. What is not fixed is one of those inputs: which other requests are in the batch with you. Trace the effect down to where it actually lands. Two tokens with logits close enough that the reordering's own numerical noise, a few bits in the last decimal place, is larger than the gap between them: that near-tie is where the chain ends. A different reduction order nudges the trailing token's score just above the leading one, and argmax returns the new winner instead of the old one. Because each new token becomes part of the prompt for the next, one flipped tie at the first position can leave the rest of the answer with nothing in common with the run before it.

Fixing the order costs throughput

Thinking Machines Lab's fix follows directly from the diagnosis: write kernels whose reduction order does not depend on batch composition. Their batch-invariant versions of the three key operations, normalization, matrix multiplication, and attention, use a fixed tiling and a consistent reduction strategy no matter how many requests share the batch.¹ Rerun the earlier experiment, a thousand identical requests at temperature zero, with the batch-invariant kernels switched on, and the completions come back bit for bit identical.¹ The fix removes the variable that was causing the symptom.

That fix has a price. Forcing a fixed tiling regardless of batch size gives up the freedom to pick the fastest tiling for each batch size. Thinking Machines Lab measured its first batch-invariant kernels running with roughly 61.5 percent more overhead than the ordinary kernels they replaced.¹ Later engineering, folding batch-invariant kernels into the inference engine SGLang, brought that down. The SGLang team's own benchmarks report an average slowdown of 34.35 percent, ranging from 25 to 45 percent across workloads, still a real tax on every request the server handles.⁷ Whether a hosted provider pays that tax by default is not settled engineering. It is a live, practical decision, and different systems are answering it differently right now.

This is why a seed parameter on a hosted API is not the guarantee it sounds like. OpenAI's own documentation says that even when the seed and the reported system configuration match between two calls, "there is a small chance that responses differ," because the seed fixes the sampler, not the kernels computing the numbers the sampler reads.⁸ vLLM ships batch-invariant kernels as an option an operator has to turn on, not the default the engine starts with. Even then it guarantees reproducibility only on the same hardware and the same version of the software.⁹ A request at temperature zero removes exactly one source of variation: the draw. The batch a stranger's traffic put you in that millisecond is a second source, and the temperature parameter was never built to reach it.