TL;DR: The same benchmark name on the same model can produce scores twenty points apart depending on the harness. The widely quoted "Claude Fable 5 scores 80.3% on SWE-bench Pro" is real and published by Anthropic, but only inside a table image, in a column covering two models, under a footnote saying the table shows the higher score of the two. Everything that makes the number checkable is the part that does not travel with it.
The one-sentence answer: Before you accept any benchmark number, ask who ran it, on what harness, and where the number is published, because the same benchmark name attached to the same model can produce scores twenty points apart depending on the scaffold, and the figures that circulate most freely are the ones stripped of the conditions that produced them.
Last updated: September 2026. This post was corrected after publication; see the correction note at the end.
Most benchmark numbers you have seen quoted this year travel without the conditions that produced them.
That sounds like a cheap shot. It is a description of what happens when you try to check one.
We went looking for a specific figure in September 2026, for a post in our series on the announcement-table benchmarks. The claim was easy to find: Claude Fable 5 scores 80.3% on SWE-bench Pro. It appears on multiple aggregator sites, stated flatly, no hedging. It is the kind of number that ends up in a slide.
Anthropic's own announcement for Fable 5 does not contain it in any machine-readable form. The page's text names FrontierCode, the Hebbia finance benchmark, IMC trading evaluations, CursorBench, ViBench and FrontierBench, and describes results in words rather than percentages. On coding it says the model "scores highest among frontier models, even at medium effort." The strings "SWE-bench" and "80.3" appear nowhere in the text of the page.
The page does carry a benchmark comparison table, published as an image with no machine-readable values and no descriptive alt text. Open it in a browser and look at the picture, which is what we should have done the first time, and the answer is the first row: "Agentic coding / SWE-Bench Pro," 80.3%.
So the number has an origin. What it does not have, once it leaves that image, is any of the three things that make it mean something.
The column is labelled "Claude Mythos 5 / Fable 5," not Fable 5. The table's methodology note explains the merge: "Reported scores are within a 1-3 percentage point difference for Claude Mythos 5 and Claude Fable 5. This table shows the higher score of the two." So 80.3% is the better of two models, and this table does not break Fable 5 out on its own. The note points to the system card for the per-model detail, which we have not read, so treat that as the open question rather than as an absence. What is settled is narrower and enough: every aggregator that prints "Claude Fable 5 scores 80.3%" is citing a row that does not say it.
And the evidence sits in a format that cannot be indexed, searched, quoted or checked by anyone not looking at the picture. That is how a figure with a real origin circulates as though it had none. Note what this is not: reticence. Anthropic's Opus 4.7 announcement from April 2026 names SWE-bench Verified, Pro and Multilingual in plain text, and adds a footnote most labs would not write, that "our memorization screens flag a subset of problems in these SWE-bench evals" and the improvement holds once those are excluded, a disclosure we put in context here. The company discloses plenty. On this page it chose a picture. The successor announcement for Fable 5.1 publishes a proper HTML table instead, listing Terminal-Bench-Science 0.1, Terminal-Bench 4.0, GDPval-AA v2, OSWorld 2.0, Humanity's Last Exam, AutomationBench and CursorBench 3.2.0. No SWE-bench row appears in it, which is a claim you can now verify yourself in about ten seconds.
So we checked the independent leaderboard.
What does the independent number actually say?
Scale Labs runs the public SWE-bench Pro leaderboard. It evaluates every model through the same SWE-Agent scaffold, most entries with uncapped cost and a 250-turn limit, and it marks a task resolved only when a submitted patch satisfies two conditions in the evaluation environment: issue resolution and no regressions.
At the time of writing the top three are Muse Spark 1.1 at 61.50 ±3.10, gpt-5.4 at xHigh effort on 59.10 ±3.56, and Muse Spark at 55.00 ±3.60. No Claude model appears in the top three. Claude Opus 4.1 sits in the leaderboard's historical comparisons at 23.1%.
Hold those two things side by side. A vendor figure of 80.3%, produced on the vendor's own harness and covering the better of two models, against a standardized leaderboard whose best entry is 61.50% and which lists no Fable model at all.
That is not a vendor lying. Both numbers are real. They were produced by different harnesses under different rules, only one of them is published with its conditions attached, and the version that travels is the one with the conditions removed.
Why do scaffolds move scores so much?
Because the model is only one component of what gets measured.
A benchmark run is a model plus a harness plus a tool configuration plus an effort setting plus a trial count. Change any of those and the number changes, often by more than the gap between two competing models.
Anthropic publishes enough detail to see this directly, which is to its credit and is exactly what reading the footnote is for. The Claude Opus 4.5 methodology footnote reads: "All evals were run with a 64K thinking budget, interleaved scratchpads, 200K context window, default effort (high), default sampling settings (temperature, top_p), and averaged over 5 independent trials. Exceptions: SWE-bench Verified (no thinking budget) and Terminal Bench (128K thinking budget)."
Read what that footnote concedes. One benchmark in that table ran with double the thinking budget of the others. Another ran with none. The scores sit in the same table anyway, formatted identically, and nothing in the visual presentation tells you they were produced under different conditions.
By Opus 4.6, SWE-bench Verified was "averaged over 25 trials" rather than 5. Averaging over more trials is a defensible methodological improvement. It also means the 4.5 number and the 4.6 number are not the same kind of measurement, and any chart that plots them as a trend line is drawing a slope across a definitional change.
Vals.ai, which states plainly that "We run all of our own evaluations and create many of our benchmarks in-house," runs SWE-bench Verified through what it describes as "a minimal bash-tool-only agent harness" chosen for "fair and consistent comparisons." A deliberately minimal harness will produce lower numbers than a rich agentic scaffold. Neither is cheating. They are answering different questions, and only one of them resembles what you will deploy.
What is the strongest argument on the other side?
That labs have legitimate reasons to run their own harness, and this is not a weak argument.
A frontier lab knows things about its own model that no external evaluator does: which prompt formats it responds to, how its thinking budget interacts with tool calls, where its context compaction triggers. Handing that model to a generic scaffold and reporting the result would understate what the model can do in competent hands. If you are going to deploy it with good engineering around it, the vendor-scaffold number may genuinely be the more relevant one.
Anthropic also does something better than most: it publishes the footnote. The 64K budget, the trial count, the exceptions, the harness name. You can disagree with the configuration precisely because they told you what it was. A number you can argue with beats a number you cannot inspect.
The argument fails only at the point of comparison. A vendor-scaffold score is a fair claim about the model. It is not a fair input to a table ranking that model against a competitor measured differently, and the tables get built anyway.
What does this look like in our own work?
We have done this to ourselves, in public, and the record of it reads almost identically.
Writing our local AI SEO guide we cut the statistic the entire topic is built on, and the published post says so in as many words: "the statistic everyone quotes did not survive verification, so we cut it." The figure was that local queries trigger AI Overviews 68% of the time. What we found when we went looking for its primary source was "a pile of AI-generated statistics roundups quoting each other, with trigger-rate numbers ranging from single digits for transactional queries to 90-plus percent for hybrid ones, none traceable to a fetchable methodology."
Change the subject from local search to model evaluation and that paragraph describes how the 80.3% reached us. It does not describe where the number came from, which is what this post said when it was first published and what it got wrong. The 68% had no source. The 80.3% had one, sitting in a picture we did not open.
That is the correction this post carries, and it made the method stricter rather than weaker. "Open the primary source" turned out to be too loose an instruction, because open had quietly come to mean fetch the text. Labs publish their most important tables as images. A page's text cannot establish an absence, so if the claim you are checking is that something is missing, you have to look at the page the way a reader does.
The uncomfortable part is how often it is not there. Checking is cheap, roughly ten minutes per claim, and it changes what you are willing to publish.
What should you actually do with a vendor's number?
Four questions, in this order, and they take one email.
Who ran it? The lab, an independent evaluator, or nobody you can identify. If a vendor quotes a figure, ask for the URL where it is published. A number with no published home is not evidence.
On what harness? Vendor scaffold or standardized leaderboard. The same benchmark name means very little across that boundary.
At what effort and how many trials? Effort level and trial count are configuration choices that move scores. Note that gpt-5.4 appears on Scale's board explicitly labeled "xHigh," which is the leaderboard being honest that effort is part of the result.
Does anything independent corroborate it? Vals.ai, Artificial Analysis and the Scale leaderboards run their own evaluations. Corroboration is not agreement on a number. It is agreement on rank order and rough magnitude, which is all you should have wanted.
If the answer to the first question is "it is on several websites," you have found circulation, not measurement.
Where does this leave a buyer?
Treat benchmark scores as a filter, never as a decision.
They are useful for excluding models that are obviously unsuited and nearly useless for choosing between the top few, because the gaps at the top are usually smaller than the variation introduced by the harness. The decision belongs to a test on your own work, which is a separate post and a longer job.
Here is a falsifiable prediction, offered so this post can be checked rather than believed. Within twelve months, at least one frontier lab will publish a benchmark table that reports the same benchmark twice, once under its own scaffold and once under a standardized third-party harness, as a differentiator. The pressure is already visible: Anthropic credited an independent evaluation to Artificial Analysis on GDPval-AA, and it read as a strength rather than a concession. If no lab does this by September 2027, I was wrong about where the incentives point.
Frequently Asked Questions
Is a vendor-reported score dishonest?
Usually not. It is a real measurement under stated conditions, and the good ones publish those conditions. The failure is in reuse, when a number produced under one configuration gets placed in a comparison table against numbers produced under another.
Why do independent leaderboards report lower numbers?
Because a standardized harness is built for comparability rather than for maximum performance, and it cannot be tuned to any one model's quirks. Expect independent figures to sit below vendor figures on agentic benchmarks, and treat the ordering as more informative than the absolute values.
Should I ignore benchmarks entirely?
No, and the cynical position is as lazy as the credulous one. Benchmarks are the only public, comparable evidence that exists. They just answer a narrower question than the way they are quoted implies.
How do I check a number without technical staff?
Search the exact figure with the model name and see whether any result is a lab announcement, a leaderboard, or a paper. Then open the lab announcement and look at its benchmark table with your eyes, because the table is often an image and searching the page for the figure will find nothing even when the figure is right there. If every result is a blog post citing another blog post, and the lab's own table does not contain the row, you have your answer in about five minutes.
The pattern worth carrying is simple. A benchmark number is a claim about a model plus a harness plus a configuration, and stripping the second and third parts is how a measurement turns into a marketing asset. Ask where it was published before you ask what it means.
We write these because we buy this software and build on it. If you want the same scrutiny applied to a tool someone is selling you, see ClickWerxs AI services or get in touch.
Correction, September 13, 2026. As first published, this post reported that the 80.3% SWE-bench Pro figure for Claude Fable 5 had no published origin we could find, and inferred from the absence of SWE-bench in the announcement's text that Anthropic had not published one. That was wrong. The figure is the first row of the benchmark table on Anthropic's Fable 5 and Mythos 5 announcement, which is published as an image our tooling could not read. The post now reports what the table says, including the methodology note that the column covers both Mythos 5 and Fable 5 and shows the higher score of the two. The argument the post makes is unchanged and the example is stronger, but the original claim was an inference from a failed fetch and should not have been published as a finding.
Sources
- Anthropic, "Claude Fable 5 and Claude Mythos 5" — announcement text names FrontierCode, Hebbia finance benchmark, IMC trading evaluations, CursorBench, ViBench and FrontierBench; coding result stated as "scores highest among frontier models, even at medium effort"; the strings "SWE-bench" and "80.3" do not appear in the page text. The page's benchmark comparison table is published as an image; read in-browser on 2026-09-13, its first row is "Agentic coding / SWE-Bench Pro" with 80.3% in the column headed "Claude Mythos 5 / Fable 5," under the methodology note "Reported scores are within a 1-3 percentage point difference for Claude Mythos 5 and Claude Fable 5. This table shows the higher score of the two." anthropic.com (retrieved 2026-09-08)
- Scale Labs, "SWE-bench Pro (Public) Leaderboard" — SWE-Agent scaffold, most entries uncapped cost with a 250-turn limit; "A task is marked as 'resolved' only if a submitted code patch satisfies two strict conditions within the evaluation environment: Issue Resolution and No Regressions"; top three Muse Spark 1.1 61.50 ±3.10, gpt-5.4 (xHigh) 59.10 ±3.56, Muse Spark 55.00 ±3.60; Claude Opus 4.1 at 23.1% in historical comparison. labs.scale.com (retrieved 2026-09-08)
- Vals.ai — "We run all of our own evaluations and create many of our benchmarks in-house"; SWE-bench Verified run with "a minimal bash-tool-only agent harness" for "fair and consistent comparisons"; SWE-bench Verified board marked archived, last updated 2026-09-01, with Claude Opus 5 at 97.00%, DeepSeek V4 Pro 0813 96.40%, Kimi K3 93.40%, Claude Opus 4.8 88.60%, Grok 4.5 86.60%. vals.ai (retrieved 2026-09-08)
- Anthropic, "Claude Opus 4.5" — "All evals were run with a 64K thinking budget, interleaved scratchpads, 200K context window, default effort (high), default sampling settings (temperature, top_p), and averaged over 5 independent trials. Exceptions: SWE-bench Verified (no thinking budget) and Terminal Bench (128K thinking budget)." anthropic.com (retrieved 2026-09-08)
- Anthropic, "Claude Opus 4.6" — SWE-bench Verified "averaged over 25 trials"; GDPval-AA evaluation credited as run independently by Artificial Analysis. anthropic.com (retrieved 2026-09-08)
- ClickWerxs editorial record — the 68% local-AI-Overview trigger figure cut from our published post
ai-seo-local-businessafter it failed primary-source verification, with the cut and the reasoning disclosed in the post itself. Operator data, on record. - Anthropic, "Introducing Claude Fable 5.1 and Claude Mythos 5.1" — benchmarks reported in a machine-readable HTML table: Terminal-Bench-Science 0.1, Terminal-Bench 4.0, GDPval-AA v2, OSWorld 2.0, Humanity's Last Exam, AutomationBench and CursorBench 3.2.0; no SWE-bench row appears (table read in-browser 2026-09-13). anthropic.com (retrieved 2026-09-08)
- Anthropic, "Introducing Claude Opus 4.7" (April 16, 2026) — names SWE-bench Verified, Pro and Multilingual in the page text, with the footnote "SWE-bench Verified, Pro, and Multilingual: Our memorization screens flag a subset of problems in these SWE-bench evals. Excluding any problems that show signs of memorization, Opus 4.7's margin of improvement over Opus 4.6 holds." anthropic.com (retrieved 2026-09-08)
Benchmark figures cited in this post are snapshots tied to a specific model, date, harness and configuration, and they change with every release. Leaderboard standings quoted here were read on the date shown. ClickWerxs builds on commercial AI models and earns revenue from software built with them. This post is operator opinion, not investment or procurement advice.
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut