TL;DR: τ2-bench drops an AI agent into a conversation with a simulated customer, hands it tools and a written policy, and checks the database afterward to see whether the rules held. Of everything in the announcement tables, it is the closest to the question a business actually has: will this thing follow instructions when a customer pushes back?
The one-sentence answer: τ2-bench measures policy adherence under conversation pressure, scored on what actually changed in the system of record rather than on what the transcript sounds like.
Most rows in a model announcement's benchmark table measure something no customer will ever do to you. Nobody calls your office and asks for a competition math proof. This row is different, and it earns its place in our guide to the announcement-table benchmarks for one reason: it simulates the exact failure a business owner fears when they connect an AI agent to real systems.
The fear is not that the agent will be dumb. It is that the agent will be helpful in a way the policy forbids.
What does τ2-bench actually test?
Three parties and a rulebook. An AI agent gets a set of tools (look up a booking, modify an order, issue a credit) and a written domain policy that constrains when those tools may be used. A simulated customer, played by another language model, wants things: some allowed, some not, some allowed only in a specific order. The agent has to complete the task while staying inside the policy.
The original benchmark, τ-bench, was introduced in June 2024 by Yao, Shinn, Razavi and Narasimhan as "a benchmark for tool-agent-user interaction in real-world domains," with retail and airline domains. Its stated purpose was to test capacities most benchmarks skip: "interaction with human users" and the "ability to follow domain-specific rules."
τ2-bench, published in June 2025 by Barres, Dong, Ray, Si and Narasimhan, raised the difficulty in one specific way. The new telecom domain is dual-control: the customer can act on the shared environment too. Think of a support call where the caller is standing at their own router. The agent now has to reason about a system it does not fully control and coordinate another person's actions through conversation.
How is a task scored?
Two mechanics matter, and both are unusual.
First, the ground truth is the database, not the dialogue. A task passes when the final state of the environment is correct: the booking actually changed, the refund actually issued, the forbidden action actually did not happen. An agent that sounds competent while writing the wrong thing to the system fails. This is the right instinct, and it is the same reason our own switching guide tells people to verify record counts instead of trusting a migration's progress bar.
Second, the original paper proposed pass^k: run the same task k times and count it only if the agent succeeds every time. One clean run is a demo. Eight clean runs is reliability, and reliability is the deployable property.
What did the original paper find?
The June 2024 τ-bench paper reported that even strong function-calling agents of that generation, GPT-4o included, succeeded on under 50% of tasks on a single attempt, and that pass^8 fell below 25% in the retail domain. Treat those figures as history rather than a current ranking; they describe models two generations old. The durable finding is the shape, not the numbers: success roughly halved when consistency across eight runs was demanded, on identical tasks.
The τ2 paper added a second durable finding: performance drops significantly when the environment moves from agent-only control to dual control. Its authors built the tasks compositionally, from atomic components with controlled complexity, precisely so they could separate reasoning failures from coordination failures. Agents that can do the thing often cannot talk someone else through doing the thing.
What does a high score not tell you?
The policy in the benchmark is not your policy. Retail, airline, and telecom rulebooks written for evaluation are shorter and cleaner than the accumulated exceptions of a real business. A model that follows a two-page policy at a high rate has demonstrated something real, and still has not demonstrated it will survive your returns policy, your discount matrix, and the undocumented rule your office manager enforces from memory.
The customer is also not your customer. The user is simulated by a language model, which makes the benchmark reproducible and affordable, and also means the difficulty of the customer is itself a modeling choice.
How do the labs use it?
Anthropic's Claude Opus 4.5 announcement discusses τ2-bench in its narrative, including an airline scenario where the model "found an insightful (and legitimate) way to solve the problem" that the task's checker had not anticipated. That anecdote is worth more attention than any percentage, because it captures the evaluation problem from the other side: a rule-following test has to decide what counts as following the rules, and clever-but-legitimate is the hardest case to grade.
Per the series rule, no score gets quoted here without its configuration; when you meet a τ2 number in the wild, the footnote checklist applies in full.
Is it contaminated?
Partly protected, by an unusual mechanism. The tasks are public, so nothing stops them from entering training data, and a memorized flight-change conversation is a real risk on the fixed task set. The defense is the τ2 paper's compositional task generator: because tasks are assembled programmatically from atomic components with controlled complexity, an evaluator can generate fresh tasks the model has never seen and check whether performance holds. A score that survives regeneration is measuring the skill. One that collapses was measuring the memory.
What should this mean if you are buying an AI agent?
Steal the method. The benchmark's two ideas, database-checked outcomes and repeated trials, translate directly into how to run a pilot.
We build customer-facing assistants for client businesses, and the constraint list is where the engineering time goes: what the assistant may never say, which actions require a human, what gets written to the record and what does not. Much of what we engineer into an assistant is what it must not do. τ2-bench is the public evidence for why that emphasis is correct.
And the benchmark is runnable. Sierra maintains the code as an MIT-licensed public repository with retail, airline, and telecom among its included domains and a command-line harness that accepts your choice of agent model, user model, and trial count. Before signing with an agent vendor, running even a small task set with multiple trials against your own policy document tells you more than any announcement row.
Frequently Asked Questions
Is τ2-bench the same thing as "tau-bench"?
Same family, second generation. τ-bench (June 2024) established the format: agent, simulated user, tools, written policy, database-verified outcomes, retail and airline domains. τ2-bench (June 2025) added the dual-control telecom domain where the user acts on the environment too, plus a compositional task generator. When a vendor says "tau-bench," ask which one and which domain, because the telecom dual-control tasks are the harder set.
Why simulate the customer with another AI instead of using people?
Cost and repeatability. Human testers cannot rerun the identical conversation eight times to compute pass^k, and staffing thousands of conversations per evaluation is not practical. The tradeoff is acknowledged in the design: a simulated user is itself a model with quirks, so some recorded failures are really coordination breakdowns between two AIs. The τ2 authors built ablations specifically to separate those cases.
Can we run it against our own agent before buying?
Yes. The repository is public under an MIT license, and the harness takes your agent model, a user model, a domain, and a trial count as command-line arguments. A more useful variant for a buying decision: keep the harness idea, swap in a handful of tasks written from your own policy manual, and require pass^5 before anything touches production.
If you are evaluating an agent for your own operation, this benchmark's design is the template we would hand you anyway: outcomes checked in the system of record, the same task run repeatedly, and the policy document treated as the test. That is how we scope assistant builds before they go live.
See ClickWerxs AI services or get in touch.
Sources
- Yao, Shinn, Razavi, Narasimhan, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" — original benchmark, domains, pass^k metric, single-attempt and pass^8 findings. arxiv.org/abs/2406.12045 (June 2024; retrieved 2026-08-13)
- Barres, Dong, Ray, Si, Narasimhan, "τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment" — dual-control telecom domain, compositional task generation, reasoning-versus-coordination ablations, performance drop under dual control. arxiv.org/abs/2506.07982 (June 2025; retrieved 2026-08-13)
- Anthropic, "Introducing Claude Opus 4.5" — τ2-bench airline-scenario discussion. anthropic.com (retrieved 2026-08-13)
- Sierra Research, tau2-bench repository — MIT license, domains, CLI harness with configurable agent model, user model, and trial count. github.com (retrieved 2026-08-13)
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut