A chatbot reads your whole prompt at once but writes its reply one word at a time

The pause and the stream are two different jobs

Send a model a paragraph and watch what happens in time. First a pause, while nothing appears. Then the reply, arriving one word at a time at a roughly steady pace until it stops. That is not one process running slow and then fast. It is two jobs, and engineers measure each with its own number. The pause is the time to first token: the gap between sending your prompt and seeing the first word come back. The pace of the stream is the inter-token latency, also called the time per output token, the average gap between one streamed word and the next.¹

Both jobs run on the same model, on the same weights, reading the same kind of tokens. What differs is how many tokens each handles at once. The first job, prefill, reads your entire prompt in a single parallel pass. The second, decode, writes the reply one token per step, running the whole model again for each word.²

The pause is the model reading your prompt

The pause scales with how much you typed. A one-line question barely pauses; paste in a long document and the wait before the first word grows with it. That is because prefill does work in proportion to the prompt. It runs every token of your prompt through the model at once, in a single forward pass.² Because all the prompt tokens are known up front, the hardware processes them together as one large matrix multiplication, and a modern GPU is built to do exactly that kind of dense arithmetic fast.²

This makes prefill compute-bound: the time it takes is set by how many arithmetic operations there are, not by how fast the chip can move numbers in and out of memory.³ The prompt has to be long enough to saturate the chip's arithmetic units before that bites. DistServe reports that prefilling a 512-token prompt is already enough to make an A100 GPU compute-bound for a 13-billion-parameter model.⁴ The exact length depends on the model and the GPU, but the direction does not: a longer prompt means more arithmetic, which means a longer pause. The time to first token is, in essence, the duration of this phase.⁴

Prefill does one more thing while it reads. As it passes each prompt token through the attention layers, it produces two vectors for that token, a key and a value, and it saves them. That saved set is the cache the next job depends on.

What the pass leaves behind is a cache of keys and values

To see why that cache matters, look at what the next word would cost without it. Attention, the mechanism at the core of the model, works by having each token compare itself against every earlier token and pull information from them. For that comparison it needs, for every earlier token, two vectors computed back when that token was processed: its key and its value.⁵ The attention lesson builds up what these are and how the comparison works. Here we need only that they exist. And a past token's key and value never change once computed, because the token itself does not change.⁵

So the model saves them. This is the key-value cache, usually just the KV cache: the per-token keys and values, held in GPU memory, so that no step ever has to recompute them.² Without it, generating each new word would mean recomputing the keys and values for the entire prompt and everything written so far, from scratch, at every step. The output would be identical; the cost would be enormous. The cache trades memory for that repeated work.

Memory is the right word, because the cache is large. For OPT-13B, a 13-billion-parameter model, storing the keys and values for a single token takes about 800 kilobytes.² Hold a full 2,048-token sequence and that one request's cache can reach 1.6 gigabytes.² Both figures are specific to that model and to 16-bit numbers; a bigger model or a longer context makes them bigger. The point that carries is the shape: the cache grows with every token in the conversation, and it lives in the same scarce GPU memory the model's own weights already fill.

The stream is slow because each word rereads the whole cache

Now the second job. Decode writes the reply one token at a time. To produce the next word, the model runs a full forward pass, but over just one new token, the last word it wrote. Everything else it needs, the keys and values of every token before, it reads out of the cache.⁶ Then it appends the new word's own key and value to the cache and repeats. Each step waits for the one before it, because the next word depends on the word just written. This is the sequential generation the autoregressive-generation lesson covers.

Here is the twist that makes the stream slow. For that single new token the model does very little arithmetic, but it has to read the entire cache back from memory, along with all the model's weights.⁶ The work is almost all memory traffic and almost no math. That makes decode memory-bandwidth-bound: its speed is set by how fast the hardware can move the cache and the weights out of memory, not by how fast it can compute.³ A faster arithmetic unit barely helps, because the tokens still arrive at the pace memory can feed them. This is the standard way inference engineers describe the two phases, and it is why they watch memory bandwidth more closely than peak compute when they predict token speed.⁷

The cost gap between the two jobs is stark, and it runs the opposite way from intuition. Reading a token during prefill is nearly free, because the chip amortizes one load of the weights across the whole prompt at once. Writing a token during decode reloads those weights for that one token alone. On a Mistral-7B model on a single A100, Sarathi-Serve measures the cost of the main computation for one decode token as close to the cost of 128 prefill tokens.⁸ That ratio is specific to that model and that GPU, but the asymmetry it names is general. The long prompt you took minutes to type is the cheap part. The short reply the model writes back is the slow, expensive part. It is also why output tokens are priced higher than input tokens, though the dollars are a separate lesson.

Where to run the two phases is still unsettled

Everything so far is settled engineering. Every primary systems paper that characterizes the two phases agrees on the direction: prefill is compute-bound and parallel, decode is memory-bandwidth-bound and sequential, and the KV cache is what ties them together.⁶ What is not settled is what to do about the mismatch, because the two jobs want different things from hardware. Prefill wants raw arithmetic and gains little from being batched with other requests; decode wants memory bandwidth and gets far more efficient when many requests are batched together.⁸ Run both on the same GPU and each phase interferes with the other's ideal schedule.

Two camps of researchers, each building working systems, disagree on the fix. One camp splits the phases apart. Splitwise and DistServe run prefill and decode on separate machines, each tuned for its own job, and pass the KV cache between them. Splitwise reports about 1.4 times the throughput at roughly 20 percent lower cost from that separation.⁶ The other camp fuses them more tightly instead. Sarathi-Serve keeps both on the same GPU and breaks each prompt's prefill into small chunks, interleaving those chunks with ongoing decode steps so that neither stalls the other.⁸ Both sides argue from measured results on real hardware. Which organization wins, or whether the answer depends on the workload, is a live question, not a solved one.