The same open model returns identical text two to three times faster on one service

One model, one token per pass, is the speed the fast service beats

A hosting service that puts an open model behind an interface can serve the same weights, on the same kind of hardware, at very different speeds, and the difference reaches two to three times.¹ To see how one service pulls ahead without changing the answer, you need two plain terms. A token is the small chunk of text a model reads and writes, usually a word or a piece of one. A forward pass is one sweep of the whole network over the text so far, and it is the step that produces the next token.

A model normally runs one forward pass for every token it writes. Each pass is slow, and a long reply needs hundreds of them in a row.² The pass is slow for a reason worth holding onto: most of its time goes to hauling the model's billions of weights out of memory and into the chip, not to the arithmetic once they arrive.² That fact sets an apparent ceiling. One big-model pass per token looks like the fastest the model can go, and the fast service is somehow beating it.

A small model guesses ahead, and the big model checks every guess in one pass

The trick uses two models. Alongside the big one, the target, the service keeps a small, cheap draft model. The draft runs a few quick forward passes and proposes the next several tokens. Then the target runs once over all of those proposed positions at the same time. A transformer can score a whole stretch of positions in a single forward pass, so checking several guesses costs the target about one pass, the same as producing one token used to.³

The target now compares each guessed token against the token it would have chosen there itself. It keeps the guesses that match, in order, and the moment it hits a guess it disagrees with, it throws that guess and every guess after it away.² At the first disagreement the target supplies its own token instead, so even a run that ends in a rejection still moves at least one token forward.³ This is exactly the loop a serving framework runs: the draft proposes, the target verifies the batch, the accepted tokens are committed, and a rejection triggers a fresh proposal from where the agreement stopped.⁴

When the draft guesses well, several tokens are committed from one target pass, and the reader sees a burst. When it guesses badly, the target spends a pass to move one token and the stream briefly stalls. That alternation is the burstiness you notice on the faster service.

The checked text is the target's own, word for word

The reason quality cannot slip is that the target only ever keeps a token it would have written anyway. When the model is decoding greedily, always taking its single most likely token, a guess is kept only if it equals that most likely token, so the accepted text is identical to what the target alone would have produced.²

Sampling, where the model draws a token at random from its probability distribution rather than always taking the top one, needs more care. The target keeps a guess with probability equal to its own probability for that token divided by the draft's, capped at one. When it rejects, it draws the replacement from an adjusted distribution, the target's probabilities with the draft's already-covered share subtracted off.³ The original paper proves this correction returns exactly the target's own distribution, and the DeepMind team's version states the same guarantee: the scheme preserves the distribution of the target model.⁵ An engineering write-up puts the upshot plainly: the overall output is identical to that of ordinary decoding.⁶

How much faster rides on how often the guess is right

The size of the speedup comes down to one number: the acceptance rate, the chance that any single guess survives the target's check. Write it as α, and write the number of tokens the draft proposes each round as γ. The expected count of tokens the target commits per pass is a short formula.³

Read it through its two ends. When α is near zero the draft is almost always wrong, the sum collapses toward one token per pass, and you are back to plain decoding with a wasted draft. When α is near one the draft is almost always right and the count climbs toward the full γ+1, the whole batch plus one. Everything useful happens in between, and where a workload lands is a property of the text. Predictable, repetitive text like source code is easy to guess, so it accepts more and speeds up more. Framework documentation reports the same shape: two to three times faster in general, and more than that on code.¹ An IBM production team found the same split, tuning three to four guesses per round for prose and six to eight for code.⁶

So separate the settled part from the part that varies. Settled: the output is the target's own, and a transformer verifies a batch of positions for about the cost of generating one, which is what makes the whole thing pay. What varies is the acceptance rate, set by the workload, and the deployment, set by the provider. A service turns speculative decoding on where it helps, which is the memory-bound, moderate-traffic regime, and picks how many tokens to guess.⁴ That is why two services running identical weights hand you the identical answer at different speeds. Newer variants keep pushing the ceiling by replacing the separate draft model with extra prediction heads bolted onto the target itself. Medusa reports 2.3 to 3.6 times.⁷ EAGLE reports 2.7 to 3.5 times on a 70-billion-parameter chat model while still holding the generated text's distribution fixed.⁸