TL;DR: A benchmark score is not a fact about a model. It is a fact about a model run at a specific thinking budget, effort level, trial count, and tool configuration, and all of that lives in the footnote under the table. Anthropic changed one benchmark's trial count from 5 to 25 between releases. Read the footnote before you quote the number.
The one-sentence answer: Six checks, one escalation question each, and a rule: if you cannot restate a score with its configuration attached, you do not have a number yet.
This is the fine-print companion to our guide to every benchmark in the announcement tables. That post covers what each benchmark tests. This one covers the part almost nobody reads.
Here is the footnote Anthropic published under the Claude Opus 4.5 benchmark table, retrieved from the live announcement:
"All evals were run with a 64K thinking budget, interleaved scratchpads, 200K context window, default effort (high), default sampling settings (temperature, top_p), and averaged over 5 independent trials. Exceptions: SWE-bench Verified (no thinking budget) and Terminal Bench (128K thinking budget)."
One sentence, and it changes what every number in the table means. Two benchmarks in the same table did not even run under the same settings as the rest.
To be clear about the framing: a lab that publishes this sentence is the good case. Anthropic disclosing its configuration is what transparency looks like. The problem is downstream, where the configuration gets stripped off and the bare percentage travels the internet as if it were a property of the model.
The six checks below walk that footnote, and its Opus 4.6 successor, line by line. Each ends with the question to send a vendor who quoted you the number.
What you need before you start
- The primary source: the lab's own announcement or system card, not an aggregator's table. Aggregators strip footnotes first, and some figures in circulation have no primary source at all.
- The exact number your vendor quoted, in writing.
- Ten minutes. That is genuinely all this takes.
Check 1: What was the thinking budget?
Reasoning models can spend tokens thinking before they answer, and the allowance is configurable. Anthropic's Opus 4.5 evals ran with a 64K thinking budget as the default, except Terminal-Bench, which got 128K, and SWE-bench Verified, which ran with none. For Opus 4.6, the ARC-AGI-2 result is footnoted as "max effort and a 120k thinking budget."
Same model, four different thinking configurations, one table. A score at 128K thinking is not evidence about the model you will run at a budget of zero to keep latency down.
Ask the vendor: what thinking budget produced this score, and what budget will our deployment actually run at?
Check 2: How many trials, and what happened to them?
Models are sampled, so the same test run twice gives different scores. Labs run several trials and aggregate. The Opus 4.5 footnote says scores were "averaged over 5 independent trials." The Opus 4.6 announcement says its SWE-bench Verified score "was averaged over 25 trials."
That is the same lab, the same benchmark, and a five-fold change in trial count between consecutive releases. Neither choice is wrong. More trials means a steadier estimate. But an average of 25 runs and an average of 5 runs are two different measurements, and putting them side by side in a comparison chart quietly assumes they are the same kind of number.
Ask the vendor: how many trials, and is that the same count as the number you are comparing it against?
Check 3: Were tools switched on?
The Opus 4.6 announcement describes how Humanity's Last Exam was run: "with web search, web fetch, code execution, programmatic tool calling, context compaction triggered at 50k tokens up to 3M total tokens, max reasoning effort, and adaptive thinking enabled."
Humanity's Last Exam is nominally a knowledge benchmark. Run with search, fetch, and code execution, it stops measuring what the model knows and starts measuring what the model can find and compute. That is a real and valuable capability. It is also a different capability, and the footnote is the only place the difference is recorded.
Ask the vendor: was the score achieved with tools, and does our use case allow the same tools?
Check 4: Was this parallel test-time compute?
Some announcement numbers carry a marker for parallel test-time compute, which Anthropic's Opus 4.5 page defines as "a method that aggregates multiple 'tries' from the model and selects from among them."
Best-of-N is a legitimate technique. It is also N times the inference, plus a selection step. Unless your deployment actually runs multiple attempts and picks a winner, the single-attempt number is the one that describes the product you are buying.
Ask the vendor: is this a single-attempt score, and if not, what does the single-attempt score look like?
Check 5: What were effort, context, and sampling set to?
The rest of the Opus 4.5 footnote: "default effort (high), default sampling settings (temperature, top_p), 200K context window." Effort is now a product dial on frontier models. A benchmark run at high effort describes high-effort behavior, and high effort costs more per request. Sampling settings change output variability. Context size changes what fits.
None of these are exotic. All of them are decisions someone made before the number existed.
Ask the vendor: which effort level, and is that the level our budget assumes?
Check 6: Which variant and whose scaffold?
The benchmark name itself hides a version. SWE-bench alone ships as original, Verified, Multilingual, and Pro, with very different difficulty, and the harness wrapped around the model moves scores as much as the model does. We covered that failure mode in depth in the SWE-bench Verified explainer; the short version is that a quoted score belongs to a variant plus a scaffold, not to a name.
Ask the vendor: which variant, and whose scaffolding ran the eval?
Where this goes wrong, and what to do
The vendor cannot answer. Then you do not have a benchmark result, you have marketing. Treat the claim as unverified and weight it at zero until someone produces the configuration. A vendor who responds with the footnote details is telling you they read their own sources; that answer is worth more than the score.
The number came from an aggregator or a chart on social media. Go to the lab's announcement and find it there. If you cannot find it in a primary source with a configuration attached, the number is unusable. We have watched confidently wrong benchmark figures circulate for weeks because each site copied the last one.
There is no footnote at all. That absence is itself the finding. A lab or vendor publishing bare percentages with no methodology is asking you to trust a measurement they will not describe. The labs that disclose are the ones treating you like a buyer instead of an audience.
How you know you read it right
Restate the number in one sentence with its conditions attached: "Model X scored Y on benchmark Z, variant V, at effort E with a T thinking budget, tools on or off, averaged over N trials, per this URL."
If you can say that sentence, you have a fact you can compare against another fact stated the same way. If you cannot, you have a percentage in a vacuum.
We hold our own numbers to the same rule, because we got the lesson firsthand. Through May 2026, this blog's Google Search Console showed close to zero clicks while Bing's AI query data showed our articles being cited up to 171 times per day. Same site, same weeks, two instruments, two opposite stories. Which number we quoted depended entirely on which measurement we attached to it, and that is exactly what a benchmark footnote is: the record of which instrument produced the number.
Frequently Asked Questions
Are methodology footnotes standardized across labs?
No, and that is the practical problem. Each lab chooses its own thinking budgets, trial counts, and tool configurations, and describes them in its own format. There is no rule requiring the disclosure at all. That is why cross-lab comparison charts should be read as directional at best: the rows share a benchmark name and almost nothing else.
Do more trials make a score better?
They make it steadier, not better. Averaging over 25 trials shrinks the noise band around the estimate; it does not raise the model's underlying ability. The practical effect is on comparability: a 25-trial average will sit closer to the true mean than a 5-trial average, so a small gap between two scores with different trial counts may be sampling noise rather than capability.
Can I just compare two labs' scores on the same benchmark?
Only if the configurations match, and they usually do not. Different thinking budgets, different scaffolds, different variants, and sometimes self-reported versus independently run. Independent leaderboards that run every model through one identical harness are the cleaner comparison, with the tradeoff that their absolute numbers may sit below the labs' announced ones.
Configuration-first evaluation is how we buy on behalf of clients too. When we scope an AI build, the model choice runs against the client's actual tasks at the effort and budget the deployment will really use, because the footnote lesson cuts both ways: the only benchmark configured like your business is one built from your business.
See ClickWerxs AI services or get in touch.
Sources
- Anthropic, "Introducing Claude Opus 4.5" — benchmark methodology footnote quoted in full; parallel test-time compute definition; Terminal-Bench and SWE-bench Verified exceptions. anthropic.com (retrieved 2026-08-13)
- Anthropic, "Introducing Claude Opus 4.6" — Humanity's Last Exam tool configuration; SWE-bench Verified 25-trial averaging; ARC-AGI-2 effort and thinking-budget configuration. anthropic.com (retrieved 2026-08-13)
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut