Two answers to one prompt diverge at a single draw

The model produces scores, not words

Ask a model the same question two mornings running and the two answers rarely match word for word. Nothing about the request changed. The prompt is identical, the model on the server is identical, and no setting changed between the tries. The usual explanation is that the model thinks differently each time. It is a reasonable guess, and it is wrong. The cause is mechanical. It sits at one specific step.

Start with what the model actually produces. It does not produce words. At each position it produces a vector of logits: one real number for every token in its vocabulary, tens of thousands of them, each a raw score for the claim that this token comes next. A larger logit means the model favors that token more, and the scores are unbounded.

A function called softmax turns that list of scores into a probability distribution. It raises the constant e to the power of each logit and divides by the total, so every entry lands between 0 and 1 and the whole set sums to 1. This is the same softmax that turns attention scores into weights. The output is a genuine probability for each token, given this model and this prompt. Holtzman and colleagues write exactly this per-token distribution over the vocabulary, and show that the quality of generated text can change dramatically from the very same trained model depending only on what is done with that distribution next.¹

The draw is where the answer moves

The distribution is fixed. Given the weights and the prompt, the softmax over the vocabulary is a plain calculation with one result; run it a thousand times and the probabilities are identical. So the variation cannot live there. It lives in the next step, where a sampler reads the distribution and picks one token to emit.

There are two ways to pick. The first is to take the token with the highest probability, every time. This is greedy decoding, also called argmax: greedy search selects the word with the highest probability as its next word at each step.² Give it the same distribution and it returns the same token, always.

The second is to sample: to treat the probabilities as odds and let chance choose, so a token the model rates at 20 percent is picked roughly one time in five. Sampling means randomly picking the next word according to its conditional probability distribution.² Two runs of the same distribution now hand back two different tokens, not because the odds changed but because a different roll came up. That draw, at every position, is where two answers to one prompt come apart.

Hold two things apart. One is the distribution: the probabilities themselves, fixed by the weights and the prompt. The other is the draw: the single token chance pulled out of that distribution this time. The weights and the prompt settle the distribution. The sampler and its seed settle the draw. A seed is the starting value for the pseudo-random number generator the sampler uses, and fixing it makes the draw repeatable. OpenAI documents this directly: pass the same seed across requests and the outputs become mostly reproducible, because the same seed reproduces the same draws.³

So the model does not think differently each time. It computes the same distribution each time, and a sampler draws a different token from it. The wrong explanation points at the distribution, which did not move. The cause is the draw, which did.

Temperature reshapes the odds before the draw

If sampling is where variety enters, temperature is the dial that sets how much. Temperature is a single number, written T, that rescales the logits before the softmax sees them: every logit is divided by T before softmax turns the list into probabilities. Hinton, Vinyals, and Dean, who gave this scaled softmax its standard form, state its effect in one line: a higher T produces a softer probability distribution.⁴ At T equal to 1 the division changes nothing, and you get plain softmax.⁴

Watch it on three tokens. Call them A, B, and C, with illustrative logits of 2.0, 1.0, and 0.0. At T equal to 1 the division by 1 leaves the logits alone, and softmax raises e to each: e to the 2.0 is about 7.39, e to the 1.0 is about 2.72, and e to the 0.0 is 1.00. Those three sum to 11.11. Dividing each by that total gives A, B, and C probabilities of 0.665, 0.245, and 0.090. Now set the temperature to 0.5. Dividing each logit by 0.5 doubles it, to 4.0, 2.0, and 0.0, so the exponentials spread further apart: A climbs to 0.867 and C sinks to about 0.016. Set the temperature to 2.0 instead, halving the logits to 1.0, 0.5, and 0.0, and the probabilities even out: A eases to 0.506 while C rises to about 0.186.

The logits never changed. Only T moved, and token C went from a roughly one-in-sixty long shot at T equal to 0.5 to a nearly one-in-five real possibility at T equal to 2.0. Low temperature sharpens the distribution toward its top token. High temperature flattens it toward the field. Holtzman states the same: T below 1 skews the distribution toward high-probability tokens and drains the tail.¹ Push T all the way to zero and the softmax collapses onto its largest logit, giving that token a probability of 1 and every other token 0. That is why very low temperature behaves like greedy decoding: there is nothing left for the sampler to vary.

Temperature is a knob on a real product, and its range is not universal. Anthropic's API runs temperature from 0.0 to 1.0 with a default of 1.0.⁵ Other vendors expose a wider band that reaches above 1.0 for still flatter distributions. The number that matters is not the ceiling but the direction, which is the same everywhere: down sharpens, up flattens.

Cutting the tail: top-k and top-p

Temperature reshapes the whole distribution but never zeroes any token out. Even at a modest setting, the lowest-ranked token keeps a sliver of probability, and the sampler will occasionally draw it. Two more controls forbid that outright, by truncating the distribution before the draw.

Top-k keeps only the k highest-probability tokens and discards the rest, renormalizing what remains. Fan, Lewis, and Dauphin introduced the move for story generation, sampling at each step from the k = 10 most likely candidates.⁶ With k set to 10, the eleventh-ranked token and everything below it cannot be drawn at all, whatever the temperature.

Top-k has a blind spot: a fixed k ignores how peaked or flat the distribution is. When the model is confident, ten candidates may reach into noise; when it is unsure, ten may cut off good options. Top-p, or nucleus sampling, fixes this by cutting on cumulative probability instead of a fixed count. Holtzman and colleagues, who introduced it, keep the smallest set of tokens whose probabilities add up to at least p, then sample from that set.¹ At p equal to 0.9 the model keeps just enough tokens to cover 90 percent of the mass, which is a handful when it is confident and many when it is not. Anthropic exposes this as top_p, computing the cumulative distribution in decreasing order and cutting it off at the chosen probability.⁵

Temperature, top-k, and top-p are settled engineering: each is a specific, well-defined operation on the distribution before the draw, documented in the paper that introduced it and the API that ships it.

Why zero temperature still is not repeatable

One honest complication remains. Set temperature to zero, which should make decoding greedy and therefore deterministic, and hosted models still return different answers on different runs. Anthropic says so in its own reference: even with temperature of 0.0, the results will not be fully deterministic.⁵ OpenAI's seed feature carries the same caveat, that responses can differ even when the seed and the system configuration match, due to the inherent non-determinism of the models.³

The tempting explanation is that GPU floating-point arithmetic is non-associative, so concurrent operations that finish in a different order round to slightly different results. That is a real ingredient, but a recent analysis from Thinking Machines argues it is not the trigger, because a normal forward pass has essentially no race-prone atomic additions to reorder.⁷ Their account puts the cause one level up. The server batches requests together, the batch size shifts with load from other users, and the normalization and matrix-multiply kernels are not batch-invariant, so a change in batch size changes the order the numbers are summed in and therefore the exact result. The primary reason nearly all LLM inference endpoints are nondeterministic, they write, is that the load, and thus the batch size, nondeterministically varies.⁷ Make the kernels batch-invariant, they show, and the runs become bit-identical.

This part is not settled the way softmax is. It is a strong, recent account that others are still testing and adopting. Simon Willison, walking through the same work, flags the batch-size finding as the surprising claim and notes the effect is not tied to any one kind of GPU.⁸ The safe statement is narrow: at temperature zero the sampler stops adding randomness, but the numbers feeding it are not guaranteed identical from run to run on shared hardware, and why is still being worked out.