GPT-4o clears 61% of τ-bench tasks once and under 25% of them eight times in a row

The database state that decides whether the agent passed

τ-bench, spelled with the Greek letter and read aloud as "tau-bench," is a 2024 benchmark from the Sierra researchers Yao, Shinn, Razavi, and Narasimhan.¹ τ-bench does not ask a model to answer a question. It puts the model in the seat of a customer-service agent.¹ The agent gets a set of tools, functions it can call against a mock database such as looking up a reservation or canceling one, and a written policy for the domain.¹ It then holds a multi-turn conversation with a simulated customer.¹ That customer is not a person. It is a second language model, prompted to role-play someone with a specific, private goal the agent has to draw out through conversation.¹

The grading itself is not another model's opinion. After the conversation ends, τ-bench checks two things: whether the database landed in the state a human annotator decided was correct, and whether the agent's messages to the customer conveyed the specific information the task required.¹ Both have to hold. An agent that fixes the database but never tells the customer, or one that says the right thing while leaving the database wrong, fails the task.

The paper's own worked example, a task labeled JK9O19, shows what that looks like end to end. A customer wants a reservation canceled. The agent calls a tool, reads the account, and finds a policy that would normally block a refund for that flight. Rather than granting the request outright or refusing flatly, the agent proposes what the policy does allow: canceling the existing reservation and rebooking a comparable one.¹ The customer accepts, and the agent's closing tool call writes the new reservation to the database.¹ That final row, and the summary the agent gave the customer along the way, are what the reward function actually reads. Nothing about how the agent got there, how many turns it took or how it phrased its offer, enters the grade.

One try is not the same as eight in a row

Because a task can be attempted more than once, τ-bench does not settle for a single number. Its authors run every task several times and report two statistics. Pass^1 is the plain one: the share of tasks the agent completes on a given try, averaged across repeated trials.¹ Pass^k is stricter. It is the share of tasks the agent completes on every one of k independent attempts at that same task, not "succeeded at least once in k tries" but succeeded k times running.¹ Sierra's own framing of why it bothered to build the harsher measure states the point directly.² τ-bench "doesn't just test whether an agent can complete a task once; it measures whether it can do so consistently multiple times."²

Run both statistics on the same model and the gap is wide. GPT-4o, the strongest model in the original paper, completed 61.2% of retail tasks on a single try.¹ Made to succeed eight times running on the same retail task, the paper's own headline claim is starker.¹ Its abstract states plainly: "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."¹

GPT-4o also does much worse outside retail: 35.2% pass^1 in the airline domain, per the paper's main results table.¹ Its own policy-ablation table, though, lists 33.2% for what should be the identical baseline condition.¹ The paper never explains the two-point gap. It is worth naming anyway, since a bare number this close to another can travel with a false sense of precision. By March 2025, Sierra's own blog said the best models were "now crossing 80% pass^1 in the easier domain (retail)," while adding in the same breath that performance "degrades as k increases."² Pass^1 climbed. The shape of the collapse did not go away.

The reason the gap matters is what pass^1 leaves out. A business that puts a customer-service agent into production does not run it once. It runs the same kind of task over and over, all day. What it needs to know is not whether the agent can solve a task, but how often the same task comes out right when it runs again. Pass^1 answers the first question. Pass^k answers the second, and on τ-bench's own figures for GPT-4o in retail, the two answers are not close.

The chart that leaves out the reliability rate, and the report that doesn't

A one-try average is not the only thing a bare τ-bench percentage can hide. Which setup produced it is another. When Anthropic announced Claude 3.7 Sonnet in February 2025, it published a chart titled "Agentic tool use — TAU-bench," reporting three models' single-run scores side by side.³ Claude 3.7 Sonnet scored 81.2% retail and 58.4% airline, and Claude 3.5 Sonnet scored 71.5% and 48.8%.³ The chart's third bar, OpenAI's o1, showed 73.5% and 54.2%, a figure Anthropic attributes to OpenAI's own reporting rather than one it verified itself.³ No pass^k figure sits anywhere near that chart. The numbers are a legitimate pass^1-style score, not a mislabeled reliability rate, but they are exactly the kind of bare single-run percentage the paper's own pass^k finding warns against reading as a settled measure of how the agent will hold up in use.

The chart also is not a clean rerun of the paper's own baseline agent. Anthropic's own footnote states the scores came "with a prompt addendum to the Airline Agent Policy instructing Claude to better utilize a 'planning' tool," and with the step limit raised from the paper's default of 30 to 100.³ Give an agent more room to plan and retry inside a friendlier prompt, and the number moves. That is a second thing a bare percentage does not carry with it: the software wrapped around the model, not the model alone, produced the score.

Not every public τ-bench number skips this context. One vendor's May 2026 post on its own agent reported pass^1 through pass^4 side by side, and said plainly what each meant.⁴ "pass^1 measures raw task accuracy. pass^4 measures consistency: the agent must complete the same task correctly across four independent runs."⁴ A bare, uncaveated percentage is often the only thing quoted in public, though not always. The exception matters, because it shows that including the missing context was never technically required, only usually skipped. A reader still has to check which kind of number they are looking at, in the reviewer's own words: "Read every score as a claim, not a fact."⁵

The simulated customer has its own error rate

The reliability gap is not the only caution built into this instrument. τ-bench's reward is a rule, not a model's opinion, but the customer half of the conversation the rule grades is played by one. A separate lesson in this series covers what happens when a language model is the one deciding who passed. τ-bench does not do that: a fixed check of the database renders the verdict. What a model does supply here is the conversation itself, and if that conversation behaves unrealistically, the agent can fail a task for reasons that have nothing to do with its underlying skill.

Sierra's own follow-up benchmark, τ²-bench, put a number on how faithful the original simulated customer actually is. Two human annotators reviewed conversations the retail and airline user simulator produced.⁶ They recorded a 40% and 47% error rate, respectively: moments where the simulated customer did not behave the way a real one would.⁶ Of those errors, 12% and 13% were severe enough to block the task from being completed correctly no matter what the agent did.⁶ A task an agent "fails" under those conditions may be failing an unrealistic customer rather than failing the skill the benchmark means to test.

Sierra's own repository now says as much about the original tasks themselves. Its README carries the warning in bold: "The tasks in this repo are not updated... Please use τ³-bench for the latest fixed tasks and new domains."⁷ The successor repository explains what changed: "a new banking domain, a voice evaluation modality, as well as fixes to the airline and retail domain tasks."⁸ The benchmark's own authors, in other words, no longer stand behind the exact retail and airline tasks that produced GPT-4o's 61.2% and the collapse beneath it, even as the same tasks keep getting quoted.