Every AI Benchmark Anthropic and OpenAI Use, Explained

TL;DR: A benchmark score is not a fact about a model. It is a fact about a model running under a specific thinking budget, effort level, tool configuration and trial count, and the labs disclose those settings in footnotes almost nobody reads. Learn to read the footnote and most vendor claims get easier to price.

The one-sentence answer: The benchmarks in AI announcement tables each test one narrow, specific thing, and knowing which thing is the difference between buying capability and buying a headline.

Every few months one of the major labs ships a model, and the announcement carries a table. Eight or twelve rows, each with a name like SWE-bench Verified or τ2-bench or MRCR, each with a percentage next to it, usually with the new model's column bolded.

Most people scan the bolded column and move on. That is a reasonable thing to do with a table you have no way to interpret.

This post is the interpretation. Not the field of AI evaluation, which is enormous, but the specific list of tests Anthropic and OpenAI put in front of buyers when they launch something. That list is finite, it is roughly twenty entries long, and every entry answers a question narrow enough to state in a sentence.

We have a reason to care that is not academic. ClickWerxs builds AI workflows for client businesses, which means picking models, and the gap between what a benchmark measures and what a client needs is where projects go wrong.

What is actually in a model announcement's benchmark table?

A short list of named tests, each measuring one capability, chosen by the lab.

The Claude Opus 4.5 announcement ran seven: SWE-bench Verified, SWE-bench Multilingual, Aider Polyglot, BrowseComp-Plus, Vending-Bench, τ2-bench and Terminal-Bench. Eleven months later the Opus 4.6 announcement ran a longer set including Terminal-Bench 2.0, Humanity's Last Exam, GDPval-AA, BrowseComp, MRCR v2, SWE-bench Verified, MCP Atlas, ARC-AGI-2, CyberGym, OpenRCA, BigLaw Bench and Vending-Bench 2.

OpenAI's tables overlap and diverge: GPQA Diamond, AIME, MMMU, HealthBench, BFCL, MMLU-Pro, SciCode, FrontierMath and GDPval appear across its releases.

Notice what that means. The list is a choice. A lab includes the benchmarks where its model does well and where it thinks buyers are looking, and it drops benchmarks that have stopped separating models. Neither of those is dishonest. Both change what you should infer from the table.

Why does the same benchmark produce different numbers in different places?

Because a benchmark score is conditional on a configuration, and the configuration lives in the footnote.

Here is the methodology note from the Claude Opus 4.5 announcement, quoted in full: "All evals were run with a 64K thinking budget, interleaved scratchpads, 200K context window, default effort (high), default sampling settings (temperature, top_p), and averaged over 5 independent trials." SWE-bench Verified was an exception and ran with no thinking budget at all. Terminal-Bench ran with 128K.

By the Opus 4.6 announcement, SWE-bench Verified was averaged over 25 trials. Humanity's Last Exam, a knowledge benchmark, was run with web search, web fetch, code execution, programmatic tool calling, context compaction and up to 3M total tokens of context.

Read that last one again. A test of what a model knows, scored with the internet open.

That is not a scandal. Running a hard research benchmark with tools is a defensible choice, and Anthropic disclosed it plainly. But it does mean the number measures research capability under tool access, and a buyer who reads it as raw knowledge has learned the wrong thing.

The trial count matters just as much and gets less attention. Averaging over 25 runs produces a different number than averaging over 5, and both differ from a single run, which is what your users get. A model that succeeds 80% of the time averaged across trials fails one task in five for the person sitting in front of it.

The largest variable is not in the footnote at all. It is the scaffold, meaning the code wrapped around the model that lets it read files, run tests and retry. OpenAI published the range in its SWE-bench Verified writeup: GPT-4's score on SWE-bench Lite varied between 2.7% using an early retrieval-based scaffold and 28.3% using CodeR. Same model. Ten times the score.

Sit with that for a second, because it reframes the entire table. A ten-fold swing from engineering that is not the model means a benchmark score is at least as much a measurement of the harness as of the thing being harnessed. The lab builds a good one. Whoever builds your integration may not.

Three questions turn any benchmark number into something you can use. What configuration produced it. How many trials it averages. Whether the same scaffold was applied to every model in the comparison.

What do the coding benchmarks actually test?

Whether a model can resolve a real software issue end to end, which is a different skill from writing code that looks right.

SWE-bench is the anchor and deserves the detail. Introduced in 2023, it contains 2,294 problems drawn from real GitHub issues and their corresponding pull requests across 12 popular Python repositories. The model receives a codebase and an issue description, and it has to edit the codebase so the issue is resolved. Success is determined by running the repository's own tests. There is no partial credit and no human judgment in the loop, which is what makes it credible.

At publication, the best model tested solved 1.96% of the issues. Frontier models now report scores above 80% on the Verified subset. That arc is the single most important fact in AI evaluation and the reason this whole category keeps churning.

SWE-bench Verified is the variant almost every announcement quotes, and the story of why it exists is worth more than the score. OpenAI put 93 Python developers through an annotation campaign on 1,699 random samples from the original test set, three annotators per sample, taking the highest severity rating among them. They found 38.3% of samples had underspecified problem statements and 61.1% had unit tests that could mark valid solutions as incorrect. Filtering on those criteria removed 68.3% of the benchmark, leaving the 500 samples that became SWE-bench Verified.

Two-thirds of the most famous benchmark in AI was not measuring what it claimed to.

Their example makes it concrete. One scikit-learn task asked an agent to fix an ignored copy parameter. To pass, the agent had to raise a deprecation warning the issue never mentioned, matching an exact message string that was settled in pull request discussion the agent cannot see. No amount of capability solves that task.

The effect on scores was large and in the direction you would expect. GPT-4o scored 16% on the original SWE-bench and 33.2% on Verified. Same model, double the score, purely from deleting tasks that were impossible.

That should adjust your confidence in benchmarks generally, and it should also raise your confidence in the people who found it, because the fix was published rather than buried. The rest of the coding family splits by task shape: Terminal-Bench gives an agent a shell and scores on exit codes and file diffs, Aider Polyglot measures editing existing code across languages rather than writing new code, and SWE-bench Pro exists because Verified is small enough to overfit.

What do the agent benchmarks actually test?

Whether a model can stay useful across many turns while obeying rules, which is where most real deployments break.

τ-bench is the one to understand, because it tests the thing businesses actually worry about. A model plays an agent with domain-specific API tools, talking to a simulated user, and it must complete the task while following a written policy. Scoring compares the final database state against an annotated goal state, so the model has to actually do the thing rather than say it did.

The finding in the original paper, published June 2024 and now a legacy benchmark result, is the part worth carrying around. State-of-the-art function-calling agents succeeded on under 50% of tasks, and were inconsistent enough that pass^8 in the retail domain came in under 25%. That metric asks whether the agent succeeds on the same task eight times out of eight. Three quarters of the time, it did not.

Frontier models have improved on that considerably since. The structural point survives: an average score and a consistency score are different numbers, and only one of them is usually printed.

An agent that works most of the time and fails unpredictably is a support ticket generator, and average scores hide that completely.

The rest of the agent family covers different surfaces. BrowseComp measures finding hard-to-locate information online. MCP Atlas and BFCL measure tool-calling correctness. Vending-Bench runs an agent as a business over a long horizon and watches for drift rather than for wrong answers. OSWorld puts an agent in front of a desktop with a mouse and keyboard.

What do the reasoning and knowledge benchmarks actually test?

Whether a model can answer questions that resist lookup, on a difficulty ladder that keeps getting rebuilt.

GPQA is the clean example of good benchmark design. It contains 448 multiple-choice questions in biology, physics and chemistry, written by domain experts and constructed to be "Google-proof." That claim is measured rather than asserted: PhD-level experts in the relevant field scored 65%, while skilled non-experts with unrestricted web access and over 30 minutes per question scored 34%. The Diamond subset is the hardest slice, and it is the one announcements quote.

Humanity's Last Exam was built when MMLU stopped working. Frontier models were clearing 90% on the popular benchmarks, so a benchmark topping out says nothing about which model is better. HLE holds 2,500 questions across more than 100 subjects, contributed by close to 1,000 subject matter experts from over 500 institutions in 50 countries, with a private held-out set to catch overfitting. Questions flagged as searchable during a bug bounty were removed.

ARC-AGI measures something different again: how efficiently a system picks up a skill it has never seen. Its tasks lean on core knowledge rather than education, which is why they are easy for humans and hard for models. It is also the only major benchmark treating cost as part of the result, displaying only systems under $10,000 per run and capping its Kaggle track at $50 for 120 tasks. The reasoning is that buying a score with unlimited compute is not the same as being intelligent, and a benchmark that ignores cost lets you do exactly that.

For a business buyer, that cost axis is the most transferable idea in this entire post.

What do the context, multimodal and domain benchmarks test?

Whether the advertised capability survives contact with a realistic workload.

Long context is the clearest case of a number that misleads. A model's advertised context window is how much text you can send it. MRCR and the retrieval benchmarks measure how much of that the model can actually use, and the multi-needle variants at 1M tokens separate models that quote the same window size. This is the gap that explains why a document-upload feature demos well and disappoints in week three.

MMMU covers multimodal reasoning: 11,500 questions from college exams, quizzes and textbooks across six disciplines, 30 subjects and 183 subfields, spanning charts, diagrams, maps, tables, music sheets and chemical structures. At publication in late 2023, a legacy benchmark result, the strongest models tested reached 56% and 59% against questions a competent undergraduate handles. That gap has closed substantially, which is the pattern this whole post is about.

The domain benchmarks are the ones closest to a purchase decision. HealthBench grades against rubrics written by physicians rather than scoring multiple choice. BigLaw Bench and CyberGym cover legal and security work. LegalBench-RAG evaluates the retrieval half of legal search, which is where most production failures actually start. If you are buying AI for a specific industry, a vertical benchmark predicts your outcome better than any general score in the table.

Why does every benchmark eventually stop being useful?

Because a measure that becomes a target stops being a measure, and the lifecycle is identical every time.

A benchmark gets published and is hard. Labs optimize against it. Scores climb. The benchmark saturates, models cluster within a couple of points, and it no longer separates anything. It gets replaced by a harder successor, and the cycle restarts. MMLU gave way to MMLU-Pro. GSM8K gave way to MATH, then AIME, then FrontierMath. HumanEval gave way to SWE-bench. SWE-bench Verified is showing the same symptoms, which is why SWE-bench Pro exists.

Contamination accelerates it. OpenAI says so plainly about its own preparedness evaluation: because SWE-bench is built from scrapes of public GitHub repositories, large models pre-trained on internet text "are likely to be contaminated on the tasks." A benchmark assembled from public data starts leaking into training sets the day it is published, and the leak is very hard to measure from outside.

The practical consequence is counterintuitive. By the time a benchmark is famous enough for a vendor to quote it at you, it has usually stopped discriminating between the serious options. The benchmarks that would actually inform your decision are the ones you have not heard of yet.

We ran into a version of this measuring our own work. Optimizing this blog for AI citation, we tracked the metric everyone tracks, Google rankings, and it stayed flat. Google organic clicks sat near zero. Meanwhile a single post on Square's Issuing V2 product accumulated 932 Bing AI citations across query variants, and sitewide AI citations went from zero to a peak of 171 per day on May 7, 2026, settling into a 56 to 111 daily range.

The headline metric said nothing was working. The metric that mattered had already moved. We would have drawn the wrong conclusion from the number everybody watches, and the only reason we did not is that we were measuring both.

That is the same failure a benchmark table invites, in a different domain.

Which benchmark should actually change your decision?

Probably none of them on their own, and the two that come closest are the ones measuring economic work rather than capability.

GDPval is OpenAI's attempt at the real question. It grades deliverables, documents, slides, spreadsheets, against work produced by industry professionals averaging roughly 14 years of experience, with a 220-task open gold subset spanning 44 occupations across the nine largest sectors of U.S. GDP. Anthropic now reports GDPval-AA figures too, which makes it the closest thing to a shared cross-lab yardstick. METR's time-horizon work approaches it from the other side, measuring the length of task a model can complete at a 50% success rate, a figure that has been doubling roughly every seven months.

Those two answer "can it do the work" rather than "does it know things." That is a better question.

But the honest answer is that the benchmark that should drive your decision does not exist yet, because nobody has built it for your business. Twenty tasks drawn from your own work, held back so no vendor has seen them, run against two shortlisted models, scored by someone who knows what good looks like in your operation. That takes an afternoon and it will tell you more than the entire announcement table.

The benchmarks in this post are still worth reading. They tell you what a model is likely to be good at, and they let you rule options out cheaply. Just do not let them make the final call.

Frequently Asked Questions

If a lab runs its own evaluations, can the results be trusted at all?

Mostly yes. The benchmarks are public and independently maintained, and results are often reproduced by third parties such as Epoch AI and Vals AI. The useful scepticism is narrower than "are they lying." It is that a lab reports its model under its own well-tuned scaffold, which is a ceiling rather than an expectation. Ask what a neutral harness produces, and whether anyone outside the lab has reproduced the figure.

What is "parallel test-time compute" and why does it appear in footnotes?

Anthropic defines it as a method that aggregates multiple attempts from the model and selects among them. It raises scores because several tries beat one. It matters to a buyer because it costs more per answer and may not be how the product you are buying is configured. When a footnote mentions it, the headline number and the number your users experience are different numbers.

Do benchmark scores predict how a model performs on my actual work?

Weakly, and better when the benchmark resembles your task. Coding benchmarks predict coding outcomes reasonably well because they run real tests. Broad knowledge benchmarks predict very little about a narrow business workflow. Domain benchmarks in your industry sit in between. This is why a held-out set of your own tasks beats any public score.

How often do these tables change enough to matter?

Frequently. Benchmarks get added, dropped and superseded between releases, as the shift from Terminal-Bench to Terminal-Bench 2.0 and from Vending-Bench to Vending-Bench 2 shows. Re-check the current announcement page rather than relying on a comparison written even a few months ago, including this one.

Is there a benchmark for whether an AI system is safe to deploy?

Not a single one, and treat any vendor implying otherwise with suspicion. Capability benchmarks measure what a model can do, not what it will do under adversarial pressure or in your regulatory context. Safety evaluation is a separate discipline covered in lab system cards, and it does not reduce to a percentage in a table.

If you are evaluating an AI vendor and their pitch leans on a benchmark table, the fastest thing you can do is ask which configuration produced each number and how many trials it averages. A vendor who can answer is worth continuing with. Beyond that, the twenty-task exercise above is genuinely an afternoon of work and it is the only evaluation that reflects your operation.

We run AI workflow builds and AI SEO on this basis, measuring the thing that matters rather than the thing that is easy to report. See ClickWerxs AI Services or get in touch.

The benchmarks in this post, at a glance

Each of these gets its own detailed post in this series. Links will be added here as they publish.

Competitor information cited in this post is based on publicly available sources as of the publication date and is subject to change. ClickWerxs is not affiliated with the companies mentioned. All comparative claims are sourced, see source links. This post reflects operator opinion and is not legal, financial, or professional advice.

Sources

1. Anthropic, "Claude Opus 4.5" — benchmark table and evaluation methodology footnotes (64K thinking budget, 5 trials, SWE-bench Verified with no thinking budget, Terminal-Bench at 128K; definition of parallel test-time compute). anthropic.com (retrieved 2026-07-28)
2. Anthropic, "Claude Opus 4.6" — benchmark list and methodology footnotes (SWE-bench Verified averaged over 25 trials; HLE run with web search, web fetch, code execution and up to 3M tokens; ARC-AGI-2 at 120k thinking budget). anthropic.com (retrieved 2026-07-28)
3. Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" — 2,294 tasks, 12 Python repositories, best model at publication solved 1.96%. arxiv.org/abs/2310.06770 (retrieved 2026-07-28)
4. SWE-bench Verified dataset card — 500 human-validated samples. huggingface.co (retrieved 2026-07-28)
5. Rein et al., "GPQA: A Graduate-Level Google-Proof Q&A Benchmark" — 448 questions; experts 65%, non-experts with web access 34% after 30+ minutes. arxiv.org/abs/2311.12022 (retrieved 2026-07-28)
6. "Humanity's Last Exam" — 2,500 questions, 100+ subjects, ~1,000 contributors from 500+ institutions across 50 countries, private held-out set. agi.safe.ai (retrieved 2026-07-28)
7. Yao et al., "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" — under 50% task success for state-of-the-art function-calling agents; pass^8 under 25% in retail. arxiv.org/abs/2406.12045 (retrieved 2026-07-28)
8. Yue et al., "MMMU" — 11.5K questions, six disciplines, 30 subjects, 183 subfields, 30 image types; GPT-4V 56% and Gemini Ultra 59% at publication. arxiv.org/abs/2311.16502 (retrieved 2026-07-28)
9. ARC Prize — skill-acquisition efficiency, core knowledge priors, cost-per-task as an evaluation axis, $10,000 display cap and $50 Kaggle compute budget for 120 tasks. arcprize.org and arcprize.org/leaderboard (retrieved 2026-07-28)
10. OpenAI, "GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks" — 220-task open gold subset, 44 occupations, 9 sectors, professionals averaging ~14 years of experience. arxiv.org/abs/2510.04374 (submitted 2025-10-05, retrieved 2026-07-28)
11. METR, "Measuring AI Ability to Complete Long Tasks" — time horizon defined at 50% success rate, doubling roughly every 7 months. metr.org (retrieved 2026-07-28)
12. OpenAI, "Introducing SWE-bench Verified" (August 2024) — 93 Python developers annotating 1,699 samples, three annotators each with highest-severity ensembling; 38.3% underspecified problem statements, 61.1% unfair unit tests, 68.3% filtered overall; 500 remaining samples; GPT-4o 16% on original vs 33.2% on Verified; GPT-4 on SWE-bench Lite ranging 2.7% to 28.3% across scaffolds; contamination acknowledgement for public-repo scrapes; scikit-learn deprecation-message example. openai.com (retrieved 2026-07-28)

Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut