The wait before a chatbot's first word barely tracks the prompt's length

The wait between send and stream has four parts

Type a question into a chatbot, hit send, and count. On a typical Claude session over a home connection a few hundred milliseconds usually pass before the first word appears. Then the words arrive one at a time, quickly enough that they read almost as a stream. That first stretch has an industry name. Together AI calls it time to first token, or TTFT: the wait between sending a request and the first word of the reply appearing.¹ Anthropic's own documentation defines it the same way, as the time to the first token of the response measured from when the prompt was sent.² The wire format underneath it is an SSE stream whose first payload event carries that first token as a text_delta.³

The pause is not the model "thinking harder." It is the model running one parallel forward pass over the prompt, producing the first output token, and leaving behind the cache of keys and values the streaming loop then rereads at every step. The mechanism is worked out in the earlier lesson on prefill and decode. The pause itself has parts. Some of it is the pure compute the prompt demands. Some of it is time spent waiting behind other users' work on the same GPU. Some of it is skipped when the serving system has already computed the prompt's cache from an earlier call. Some of it is just the round trip between the user and the datacenter. Those pieces have very different sizes.

One prompt token barely moves the pause

The story most people carry is that a chatbot pauses because attention scales with the square of the input length: double the prompt, quadruple the attention work. That scaling is real, but attention is one term in the model's per-token cost, and at everyday prompt lengths it is not the term that dominates. Vaswani et al.'s Table 1 gives the per-layer cost of self-attention as O(n^{2} \cdot d), where n is the sequence length and d is the model's representation dimension. That table compares layer types: it sets self-attention against a recurrent layer at O(n \cdot d^{2}), and it has no row for the feed-forward block.⁴ The linear-in-n cost of the rest of the model comes from a separate accounting. Kaplan et al.'s per-token FLOP table shows that the feed-forward layers and the query, key, and value projections each cost on the order of d^{2} per token, with no context length in them. Over an n-token prompt that is O(n \cdot d^{2}). Only the attention-score term carries n, growing as O(n^{2} \cdot d).⁵ Kaplan also fixes the point where the quadratic term overtakes the linear one: the context-dependent cost stays a small fraction of the total as long as d > n/12. For a model whose representation dimension runs to a few thousand, that puts the crossover in the tens of thousands of tokens. Below it the feed-forward and projection term dominates the arithmetic. The n-squared term is present and growing, but it is being outweighed.

The Anyscale team measured what that looks like end-to-end. Kadous et al. ran a regression on Llama 2 70b served through Anyscale Endpoints. Each additional input token added about 0.3 to 0.7 milliseconds to end-to-end latency. Each additional output token added 30 to 60 milliseconds.⁶ That is roughly a hundred-to-one ratio. "The practical implication of this is that reducing input tokens has very little effect on latency compared to reducing output tokens," they write. A scatterplot in the same post, at five concurrent requests, shows no discernible relationship between input tokens and TTFT across the 250-to-800-token range. The signal from input length is swamped by the run's other sources of latency.

The mechanism the intuition rests on is real. It just applies at a length most everyday chats never reach. Below that length something else is doing most of the waiting.

Past a few tens of thousands, the pause bends upward

Push the prompt into the tens of thousands of tokens and the naive quadratic story starts to describe reality. Jim Allen Wallace, writing for Redis, cites an NVIDIA developer benchmark of Llama 3.1 70B on GH200 NVL32 hardware. A 32,768-token prompt produces a TTFT of about 472 milliseconds. A 122,880-token prompt, roughly 3.8 times longer, produces one of about 2.2 seconds. That is 4.7 times longer.⁷ "TTFT grew faster than the prompt itself," Wallace observes, "and that gap widens as contexts get longer." The measurement is NVIDIA's. The Redis post is the relay this article relies on.

That is the regime in which the n^{2} attention term outweighs the n \cdot d^{2} feed-forward term. It shows up on long-context work: a chat loaded with a book, an agent carrying a page of retrieved context, a coding session opened over a large repository. It does not show up on the short turns that make up most conversations. A reader who has watched a chatbot get slower on a long document has watched the crossover happen in real time.

One user's pause reaches other users' streams

A single request in isolation is one picture. A production GPU is serving many requests at once, and the two phases share the hardware. DistServe measured what happens on a colocated system directly. Adding a single new prefill job to a batch of ongoing decodes stretches the time per output token for every user in the batch, and "the slowdown intensifies with a longer prefill."⁸ Sarathi's authors measured why. On their LLaMA-13B run on an A6000, at batch size 1 a single 512-token prompt is enough to saturate the GPU's compute on its own, so the ongoing decodes wait on it. Decode is the far more expensive phase per token at that size: "the decode cost per token can be as high as ~200 times the prefill cost per token."⁹

Serving systems have two levers here, and both matter to the pause a user feels. The first is chunked prefill, the trick behind Sarathi and Sarathi-Serve, which splits a long prompt into equal-sized pieces and piggybacks decode work onto each piece. It softens the stutter for streaming users at a price DistServe names in the same paragraph: chunked prefill "trades TTFT for TPOT."⁸ The user who just hit send waits slightly longer so that the users already watching their reply see less stutter. The second is speculative decoding. Leviathan et al. showed that a small "approximation" model can propose several candidate tokens at once. The large "target" model then verifies them all in one forward pass. The reported speedup on T5-XXL was "an out-of-the-box latency improvement of 2X-3X."¹⁰ That improvement lands entirely on the decode loop. Speculative decoding leaves the initial prefill exactly as it was. A serving stack that ships it makes the words after the pause come faster without making the pause itself any shorter.

A prompt the model has seen before skips the pause

The most dramatic reason the pause is not a function of raw input length is that the pause is paid only for the tokens whose keys and values the serving system has not already cached. PagedAttention, the memory system underneath vLLM, stores the KV cache in fixed-size blocks that different requests can share by reference. Kwon et al. report that in one of their experiments, "the KV cache of the prompt part, which accounts for 12% of the total KV cache memory in our experiment, can be shared to minimize memory usage."¹¹ For a chatbot that reuses a long system prompt across turns, or for two users hitting the same shared prompt at once, prefill on that prefix runs once and never again for its lifetime in the cache. On the second turn of a chat, the pause the reader feels can collapse to whatever the incremental new tokens, plus the queue, plus the network add up to.

The network share of the pause is small in the datacenter and easy to miss, and the party that measures TTFT is the one that brings it out. Artificial Analysis warns in its benchmarking methodology that its TTFT numbers are measured from a specific Google Cloud VM in us-central1-a because "TTFT is sensitive to server location" and "includes network latency."¹² A vendor's stated 200-millisecond TTFT is not the on-GPU prefill compute time an academic paper measures. It is the compute, plus the queue, plus the routing, plus the client's round trip. Cut the round trip in half by moving the user closer to the datacenter and the same GPU serves a visibly faster chatbot.