TL;DR: Every other benchmark in a model announcement reports capability and treats the bill as a side note. ARC-AGI's public leaderboard plots cost per task on its axis and refuses to list systems that needed more than $10,000 to run. That one design choice makes it the benchmark whose worldview matches how a business actually buys.
The one-sentence answer: ARC-AGI measures whether a system can solve visual reasoning puzzles it has never seen, and reports what each solution cost, on the stated principle that efficiency is part of intelligence.
Most of the benchmarks in the announcement-table guide answer one question: can the model do it? This one insists on a second question the others ignore: what did it cost to do it?
For anyone who pays an inference bill, the second question is not a footnote. It is the purchase decision.
What does ARC-AGI actually test?
Small colored grids. Each task shows a few before-and-after examples that demonstrate some transformation rule, then presents a new grid where the system must infer the rule and apply it. No task description, no instructions, no domain knowledge. The rule has to be extracted from the examples alone.
The point of the format is that every task is unfamiliar by construction. This is a test of picking up a brand-new skill on the spot rather than recalling a practiced one. The maintainers frame the design goal as measuring skill acquisition on unfamiliar tasks, which is the thing memorization cannot fake.
Humans handle these puzzles without any training. For ARC-AGI-2, the maintainers report that every evaluation task was solved by at least two humans within two attempts, calibrated through testing with more than 400 members of the general public. The gap between "ordinary people solve this casually" and "frontier systems find it hard" is the entire message of the benchmark.
Who maintains it, and where did it come from?
The ARC Prize organization maintains the family. The original ARC-AGI dates to 2019 and, in the maintainers' words, "endured five years of global competitions" with only minimal progress until test-time adaptation methods arrived. ARC-AGI-2 followed in 2025, built specifically to stress-test the new generation of reasoning systems after the first version began to crack.
Version 2's structure: 1,000 public training tasks, plus three calibrated evaluation sets of 120 tasks each — public, semi-private for live leaderboards, and private for final scoring. Tasks judged susceptible to brute-force search were removed outright.
How is a task scored, and why does cost count?
Solve the grid, score the point. The interesting part is what the leaderboard does next.
The ARC Prize leaderboard plots each system against cost per task, and it draws a hard boundary, stated in one sentence on the page: "Only systems which required less than $10,000 to run are shown." Spend more than that across the evaluation and the leaderboard simply does not display the result, whatever the accuracy. The maintainers put the principle just as plainly: "True intelligence isn't just about solving problems, but solving them efficiently with minimal resources."
The annual Kaggle competition runs the same idea at production severity: a "$50 compute budget for 120 evaluation tasks," which is roughly 42 cents a task. A method that needs a data center per puzzle is not just expensive there; it is disqualified.
No other benchmark in the announcement tables treats cost this way. Elsewhere, a lab can burn any amount of test-time compute chasing a record, and the resulting percentage carries no trace of the spend. Here the spend is on the chart.
How do the labs configure it?
Per the series rule, configuration before number: Anthropic's Claude Opus 4.6 announcement footnotes its ARC-AGI-2 result as run with "max effort and a 120k thinking budget." For scale, the same lab's Opus 4.5 evals ran a 64K thinking budget as the general default, so this benchmark got roughly double the default thinking allowance at the highest effort setting.
Max effort and six figures of thinking tokens are exactly the settings that raise cost per task, which is why the footnote checklist matters doubly here: on ARC-AGI, the configuration is not just context for the score, it is the other axis of the score.
What does a high score not tell you?
Nobody's business runs on colored grids. A system that infers grid transformations has demonstrated abstract pattern acquisition, and that is a meaningful signal about reasoning. It says nothing directly about whether the same system drafts your proposals or answers your phones well. ARC-AGI sits at the far research end of the announcement table; it forecasts trajectory more than it predicts your workload.
And the cost axis cuts both ways. A headline accuracy achieved at hundreds of dollars per task is a legitimate scientific result and an irrelevant commercial one. The leaderboard's $10,000 display cutoff exists precisely because the maintainers watched capability records become detached from any deployable price point.
Is it contaminated?
Contamination resistance is built into the format rather than bolted on. Novel tasks, a private evaluation set that never ships, and the removal of brute-forceable tasks mean a high score is hard to achieve by having seen the answers. The public training set is fair game for practice; the private 120 are the exam. Among the benchmarks in the series, this is the design the others borrow from when they want to claim rigor.
What should this mean if you are buying?
Adopt the axis. The single most transferable idea in ARC-AGI is not the puzzles; it is the refusal to report capability without its price.
That is how we quote AI work, because it is the only honest way to scope it. When we build an automation for a client business, the estimate is framed per unit of work: what one processed lead, one drafted reply, one summarized document costs to run at the effort level that actually produces acceptable output. Capability and cost are one decision, not a headline and a footnote. A model that is 5% better and 300% more expensive per task is, for most operational workloads, the wrong model.
So when a vendor quotes any benchmark at you, append the ARC question: at what cost per task, and at what effort setting? If they cannot price the capability, they are selling the demo, not the deployment.
Frequently Asked Questions
Why can ordinary people solve these puzzles when frontier AI struggles?
The tasks are designed to expose what the maintainers call Core Knowledge Priors: concepts people apply to new situations without any training. The version 2 design goals name symbolic interpretation, compositional reasoning, and contextual rule application as the specific capabilities under test. Systems trained on historical data excel where the past predicts the future, and each ARC task is deliberately a tiny future the past has not covered. That mismatch, trivial for one kind of intelligence and hard for the other, is the measurement.
Is ARC-AGI an IQ test for AI?
No, and the maintainers do not claim it is. It measures one property, efficient acquisition of unfamiliar skills, under one format, visual grids. IQ analogies oversell both directions: a system can score poorly here while being commercially transformative at language work, and progress here does not certify general intelligence. Treat it as a barometer for a specific hard capability.
Does beating ARC-AGI mean AGI has arrived?
The maintainers' own framing is a challenge target, not a finish line: their stated challenge is building a system that reaches 85% accuracy. The name is an argument as much as a title. The productive reading for a buyer is narrower: when scores rise at falling cost per task, reasoning per dollar is improving, and that trend line, not the milestone, is what shows up in your operating costs.
Cost-per-task is the lens we bring to every AI scoping conversation, because it is the number your P&L will actually meet. If a vendor's pitch has capability with no price axis, we can help you build the missing half of the chart.
See ClickWerxs AI services or get in touch.
Sources
- ARC Prize, "ARC-AGI-2" — task structure, training and evaluation splits, human calibration (400+ testers, all tasks solved pass@2 by at least two humans), memorization-resistance design, 2019 origin and 2025 introduction. arcprize.org/arc-agi/2 (retrieved 2026-08-13)
- ARC Prize, leaderboard — cost-per-task axis; "Only systems which required less than $10,000 to run are shown"; efficiency principle; "$50 compute budget for 120 evaluation tasks" Kaggle constraint. arcprize.org/leaderboard (retrieved 2026-08-13)
- Anthropic, "Introducing Claude Opus 4.6" — ARC-AGI-2 run configuration: max effort, 120k thinking budget. anthropic.com (retrieved 2026-08-13)
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut