Groq's 270 tokens a second and Anyscale's 185 are both true

The number Groq quotes, and the number Anyscale measured

Groq's own dashboard reports its chips run Llama 2 70B at 270-plus tokens per second per user, a figure it has used to market its inference engine since 2024.¹ Anyscale tells a different story. It built a tool called LLMPerf to check vendor throughput claims, pointed it at the same Groq service running the same model, and measured a median of 185 tokens per second.² Both are real measurements of the same system.

The instinct is to treat one number as honest and the other as spin. That assumes a chip has a single speed, the way a car has one top speed. It does not. A tokens-per-second figure depends on which phase is timed, how many requests share the chip during the test, whose definition of "a token" is counting, and what hardware, precision, and prompt length produced the run. The Groq-Anyscale gap comes from exactly one of these.

Where the stopwatch starts

A model produces an answer in two phases that behave nothing alike. First it reads the prompt: every prompt token is already known, so the chip processes all of them in one parallel matrix multiplication. This phase is called prefill.³ Then it writes the answer one token at a time, and each new token depends on every token before it. That dependency is why decoding cannot be sped up by computing two tokens at once, the mechanic the earlier lesson on how a model generates one token at a time works through. Decode is the phase that produces most of a request's total latency.³

Two clocks separate the phases. Time to first token, TTFT, measures how long a request waits for the first output token, mostly prefill. Everything after that is decode, and its per-token rate is the number most "tokens per second" claims describe, only for what happens after TTFT, not the request end to end.⁴

This is exactly the gap between Groq's number and Anyscale's. Groq's 270-plus figure starts the clock after the first token and averages over a roughly 1,000-token completion, pure decode speed once the prompt-processing cost is paid.¹ Anyscale's LLMPerf test used shorter completions, about 150 output tokens, and folded prompt-processing time into the same per-token average instead of splitting it out. Median time to first token on Groq's service was 0.22 seconds.² Spread a fixed startup cost over 150 tokens instead of 1,000 and the average falls, roughly the size of the 270-to-185 gap. Groq says as much itself: "if you were to test with 1000 output tokens, the result would be closer to the 270+ tokens/s per user you see on groq.com."¹ Neither side is wrong. They answered different questions with the same words.

More users, less speed each

The default guess is that tokens per second is a property of the chip, the way clock speed is a property of a CPU. It is not. It is a property of how many requests share that chip at once, and the same hardware running the same model reports very different numbers depending only on that count. MLPerf's own audited results make the range visible directly. 20 hardware platforms in the same round report throughput for the identical Llama 2 70B model under three latency policies, and on every one, tightening the policy cuts throughput by roughly a third to two-thirds.⁵

Batching raises the total because it reclaims the efficiency prefill already had: the chip multiplies a whole batch of thin per-token vectors at once, the same weights doing more work per pass. vLLM, the system that popularized this, reports 2 to 4 times the throughput of earlier serving software at the same latency.³

What the total hides is what one person feels, and there the direction reverses. A GPU splits its attention across every request each decode step. More riders mean a longer wait for the next token.

MLPerf sets an explicit floor on that wait. Its Server scenario requires a system to answer randomly timed requests within a latency bound. The bound MLCommons set for Llama 2 70B in 2024 was 200 milliseconds per token, anchored to "approximately 240 words per minute," an average adult's reading speed: about 5 tokens a second per user.⁶⁷ By 2025 MLCommons judged that floor too slow for real chat use. It added a tighter Interactive scenario: 40 milliseconds per token, a floor of 25 tokens per second per user.⁸

One of the platforms in the chart above makes the arithmetic concrete. A system built from eight Nebius B200 GPUs sustains 101,611 tokens per second under the looser Server bound and 59,622.7 under the Interactive bound. A 41 percent drop on identical silicon, with nothing changed but how many tokens per second must reach each user.⁵ Loosen the bound and the system can batch more work into each step, so total output climbs; tighten it and that room shrinks.

The tokenizer, the chip, and the prompt length

A token is not a fixed unit of text. It is whatever chunk a tokenizer cuts the input into, and different models use different tokenizers, the mechanic the earlier lesson on tokenization works through. Artificial Analysis, an independent benchmarking firm, runs every model's output through one shared tokenizer, OpenAI's tiktoken, "because different tokenizers create pricing [and count] incomparability across models."⁹ Two models producing the identical sentence can report different token counts for it. Two tokens-per-second numbers are not comparable unless both sides count the same way, or someone normalizes them.

Even holding the model, the text, and the tokenizer fixed, hardware moves the number just as far. Artificial Analysis's own measurements of Llama 3.1 70B, run through an identical test across providers, found output speeds from 37.9 tokens per second on one DeepInfra configuration to 119.6 on Amazon Bedrock's latency-optimized tier. A threefold spread on the same weights, driven by serving stack and quantization: the reduced precision a provider stores weights in to trade accuracy for speed.¹⁰ Context length is a fourth lever. NVIDIA's own H200 benchmarks report a 1.9x speedup over the prior H100 on a 2,048-input summarization job, and a smaller 1.6x on an 80-input chat exchange. The two examples also differ in model and parallelism, not context length alone, so neither isolates length by itself.¹¹ Even so, the two figures agree on the shape of the effect: speed at 1,000 tokens of context and speed at 100,000 are not the same claim, even from the same chip.

The audited number and the marketing number

MLPerf exists because the industry needed one audited way to settle this. MLCommons, the nonprofit that runs it, publishes its rules and reviews every submission. Its Inference benchmark defines three scenarios for one task. Single-Stream sends one query at a time, latency only, no throughput figure. Offline issues every sample as one giant batch with no latency limit, the largest, most quotable number, and the condition live traffic never meets. Server sits between them: queries arrive at random, and the reported figure is the fastest rate the system sustains inside the latency bound above.⁶ One model, one machine, three legitimate numbers.

NVIDIA supplies a case where a headline figure and an audited figure appear to disagree and, once normalized, do not. Ahead of MLPerf's 2025 round, NVIDIA told reporters a full rack of its GB200 systems, 72 GPUs, reached 869,200 tokens per second on Llama 2 70B. IEEE Spectrum flagged the figure as unverified, distinct from "the officially reported MLPerf benchmarks."¹² NVIDIA's own audited submission, an 8-GPU B200 system reviewed by MLCommons the same round, reported 98,858 tokens per second under Offline.¹³ Divide each by its GPU count and the two land close: about 12,357 tokens per second per GPU audited, versus about 12,072 for the rack. What differed was procedural: one figure went through MLCommons's review and one did not.

Cerebras holds up under the same scrutiny. Its Llama 3.1 70B chips launched at 450 tokens per second per user in August 2024. The company later reported 2,100 for the same model after adding speculative decoding, both figures labeled "output speed per user," a single-stream number, not an aggregate.¹⁴ Cerebras pointed to Artificial Analysis as the party that "rigorously tested" the update, with its own disclaimer: "20% higher or lower... is normal."¹⁵ A peak number can be real and narrow at once, if the vendor names the conditions.

The costlier version of this problem is not one buyer fooled by one company. It is a market comparing hardware on throughput measured at an interactivity level no shipped product would use. SemiAnalysis built an open benchmark, InferenceMAX, for exactly this. A GPU that wins on aggregate throughput at 5 tokens per second per user, a speed the group calls "far too slow to be practical" for a chatbot, is not winning at anything a user would run.¹⁶ MLCommons made the same correction from the standards side, adding the Interactive scenario because the old 5-tokens-per-second floor "no longer represented" real chat use.⁸ Both built new instruments for one reason: the old number was read at a speed nobody would accept.