TL;DR: SWE-bench Verified is 500 real GitHub issues that a model has to fix by editing an actual codebase, graded by the repository's own tests. It exists because two thirds of the original benchmark turned out to be unsolvable. It is the number every lab quotes, and the number most often misread.
The one-sentence answer: SWE-bench Verified measures whether a model can resolve a real software issue end to end, which is a different skill from writing code that looks correct.
Open any frontier model announcement and one row in the table will be SWE-bench Verified. It is the closest thing the industry has to a shared yardstick for coding ability.
It is also the row people quote without knowing what it counts, which is how a percentage becomes a marketing asset instead of information.
What does SWE-bench actually ask a model to do?
Fix a real bug in a real codebase, with no hints.
The original benchmark, published in 2023, assembled 2,294 problems from genuine GitHub issues and the pull requests that resolved them, drawn from 12 widely used Python repositories. The model receives two things: the codebase, and the text of the issue as a human wrote it. It has to edit the code so the problem goes away.
Nothing else is provided. Not the tests. Not the original fix. Not the discussion on the pull request where the maintainers argued about the approach.
That constraint is what makes the benchmark interesting and what makes it hard.
How is a score decided?
By running the repository's own test suite, with no human judgment involved.
Each task carries two sets of tests. The first set fails before the real fix and passes after it, which proves the issue is resolved. The second set passes both before and after, which proves nothing unrelated got broken. A model's patch has to satisfy both.
There is no partial credit. The patch either makes the tests pass or it does not.
This is the single best feature of SWE-bench and the reason it earned its position. Most AI evaluation involves a judge deciding whether an answer is good. Here, a computer runs the tests. You can disagree with the tasks, but you cannot argue with the result.
Why did the original benchmark need replacing?
Because most of it could not be solved by anyone, no matter how capable.
OpenAI put 93 professional Python developers through an annotation campaign on 1,699 randomly sampled tasks from the original test set, three annotators per task, taking the most severe rating among them. The findings were stark. 38.3% of tasks had problem statements too vague to work from. 61.1% had tests that could reject a perfectly valid solution.
Filtering on those two criteria removed 68.3% of the benchmark.
Their own example makes it concrete. One scikit-learn task asked a model to fix a copy parameter the library was ignoring. To pass, the model had to raise a deprecation warning the issue never mentioned, matching an exact message string that was settled in pull request discussion the model cannot see. No amount of capability solves that.
What remained became SWE-bench Verified: 500 tasks confirmed to be genuinely solvable and fairly graded.
What did that do to the scores?
It roughly doubled them, without any model getting better.
GPT-4o scored 16% on the original benchmark and 33.2% on Verified. Same model, same weights, twice the number, purely because impossible tasks were removed.
That is worth sitting with. A 17-point swing came from fixing the measuring instrument. It is the clearest demonstration available that a benchmark score describes the test as much as the thing being tested.
The longer arc is more dramatic still. At publication in 2023, the best model evaluated solved 1.96% of the original tasks. Frontier models now report figures in the high seventies and above on Verified. Some of that is capability and some is a cleaner benchmark, and the honest position is that both are real.
What does a high score not tell you?
That you will see the same result.
The largest variable is not the model. It is the scaffold, the code wrapped around the model that lets it read files, run tests and retry. OpenAI published the range: GPT-4's score on SWE-bench Lite varied between 2.7% with an early retrieval-based scaffold and 28.3% with CodeR. Same model, ten times the outcome.
A lab reports its model inside a well-tuned harness built by people who know the benchmark. Whoever builds your integration is unlikely to match that on the first attempt. Treat the published figure as a ceiling.
Two further limits. Every task is Python, so the number says nothing directly about your TypeScript or Java codebase. And every task has a known correct answer that a maintainer already wrote, which is not the situation your engineers are in.
Is it contaminated?
Almost certainly, and the benchmark's own users say so.
OpenAI states it plainly about its own preparedness evaluation: because SWE-bench is built from scrapes of public GitHub repositories, large models pre-trained on internet text "are likely to be contaminated on the tasks." The issues, the discussions and the actual fixes were all public before the models were trained.
This does not make the benchmark worthless. It does mean a score includes some unknown quantity of recall alongside reasoning, and that the quantity cannot be measured from outside.
It is also why SWE-bench Pro exists. Built to resist contamination and considerably harder, its pass rates sit far below Verified's, which makes it better at separating models that have converged at the top. Two other variants fill gaps: Multilingual extends the task beyond Python, and Lite is a smaller subset that predates Verified and was made easier rather than fairer.
Which variant is a lab actually quoting?
Check, because the difference is large enough to change your conclusion.
Verified is the one in almost every announcement. Pro produces much lower numbers on harder tasks. Lite produces higher ones. A comparison mixing variants is not a comparison.
The configuration matters as much as the variant. Anthropic's Opus 4.5 announcement states its evaluations ran with a 64K thinking budget averaged over five independent trials, with SWE-bench Verified as a stated exception that ran with no thinking budget at all. By the Opus 4.6 announcement, SWE-bench Verified was averaged over 25 trials.
Averaging over 25 runs is not the same measurement as averaging over five, and neither is what a single user gets. A model that resolves 80% of tasks on average still fails one in five for the person waiting on it.
What should this mean if you are buying?
Use it to narrow the field, not to make the decision.
A high Verified score is real evidence that a model can work through a multi-step code change in an unfamiliar repository. That is a genuinely useful thing to know and it was not true of any model three years ago.
What it will not tell you is whether the model handles your codebase, your language, your conventions, or your definition of finished. For that you need a handful of tasks from your own repository, run under the harness you will actually deploy.
We build our own platform on that principle rather than on published scores. As of the end of July 2026 it carries 1,058 commits since the start of April, 148 written threat models produced before their corresponding code, and 106 test files. The test count is the one that matters here: automated tests are the mechanism that makes the thousandth change as safe as the third, and they are also exactly what SWE-bench uses to decide whether a model succeeded. The benchmark and good engineering practice are measuring the same thing.
Frequently Asked Questions
Can I run SWE-bench against a model myself?
Yes. The dataset is public and there is a containerised evaluation harness built alongside SWE-bench Verified specifically to make runs reproducible. Expect the harness setup to be the hard part rather than the model call, which is itself a useful lesson about where cost sits in AI engineering.
Why do different sites report different SWE-bench numbers for the same model?
Usually the scaffold, occasionally the variant, sometimes the trial count. A vendor-run figure using the vendor's own harness and a neutral leaderboard using one standard harness for every model are two different measurements. When they disagree, neither is lying.
Does a low score mean a model is bad at code?
Not necessarily. SWE-bench rewards a specific shape of work: navigating an unfamiliar repository and making a surgical change. A model can be weak at that and strong at writing new code from a clear specification, which is what many businesses actually need.
Is there an equivalent benchmark for languages other than Python?
SWE-bench Multilingual is the direct extension. Beyond it, coverage thins quickly, which is a real gap given how much production code is not Python. Treat cross-language ability as unmeasured rather than as demonstrated.
How long will SWE-bench Verified stay useful?
Less time than you would hope. Its 500 tasks are few enough to overfit, which is why SWE-bench Pro was built. Expect Verified to follow MMLU and HumanEval into the category of benchmarks everyone scores well on and nobody learns from.
If a vendor is quoting SWE-bench at you, two questions get you most of the way: which variant, and what scaffold produced it. A vendor who can answer both is worth continuing with.
See ClickWerxs AI services or get in touch.
Sources
- Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (2023) — 2,294 tasks from 12 Python repositories; model receives codebase plus issue text; FAIL_TO_PASS and PASS_TO_PASS test structure; best model at publication resolved 1.96%. arxiv.org/abs/2310.06770
- OpenAI, "Introducing SWE-bench Verified" (August 2024) — 93 Python developers annotating 1,699 sampled tasks with three annotators each; 38.3% underspecified problem statements; 61.1% unfair unit tests; 68.3% filtered overall leaving 500 tasks; GPT-4o at 16% on the original versus 33.2% on Verified; GPT-4 ranging 2.7% to 28.3% across scaffolds; contamination acknowledgement for public-repo scrapes; scikit-learn deprecation-message example; containerised evaluation harness. openai.com
- SWE-bench Verified dataset card — 500 human-validated samples, unit-test verification against post-PR behaviour. huggingface.co
- Anthropic, "Claude Opus 4.5" — 64K thinking budget averaged over 5 independent trials, with SWE-bench Verified run without a thinking budget. anthropic.com
- Anthropic, "Claude Opus 4.6" — SWE-bench Verified averaged over 25 trials. anthropic.com
- ClickWerxs platform repository — 1,058 commits, 148 STRIDE threat models, 106 test files as of 29 July 2026. First-party operator data.
Competitor and third-party information cited in this post is based on publicly available sources as of the publication date and is subject to change. ClickWerxs is not affiliated with the companies mentioned. All comparative claims are sourced, see source links. This post reflects operator opinion and is not legal, financial, or professional advice.
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut