TL;DR: Humanity's Last Exam was built from 2,500 expert-written questions selected so they could not be answered by searching the internet. The maintainers actively removed questions that turned out to be searchable. The labs now report scores on it with web search and web fetch switched on, which means the headline figure measures research capability rather than the knowledge the benchmark was designed to isolate.
The one-sentence answer: HLE is a closed-ended academic exam built to resist retrieval, so a score produced with the internet open is answering a different question than the one the benchmark was constructed to ask, and the two numbers should never sit in the same column.
A benchmark's design document and its scoring configuration can disagree. On this one they disagree loudly, and almost nobody mentions it.
Humanity's Last Exam earns its row in our guide to the announcement-table benchmarks because it was the field's most serious attempt at a ceiling. It also earns the closest reading, because how it is run has drifted a long way from why it was made.
What does Humanity's Last Exam actually test?
Expert-level academic knowledge, on questions specifically chosen to be unGoogleable.
The paper describes the selection criterion directly. Questions must have "a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval," and were developed "globally by subject-matter experts."
The motivation was saturation. The paper opens from the observation that "LLMs now achieve over 90% accuracy on popular benchmarks like MMLU," which had stopped discriminating between frontier models. HLE was pitched as "the final closed-ended academic benchmark of its kind with broad subject coverage."
The maintainers describe the scope narrowly and honestly: "HLE tests structured academic problems rather than open-ended research or creative problem-solving abilities, making it a focused measure of technical knowledge and reasoning."
Note what that sentence rules out. This was never meant to measure research. It was meant to measure what a model knows and can reason through, with retrieval deliberately taken off the table.
Who built it and when?
The Center for AI Safety and Scale AI organized it as a global collaboration. Nearly 1,000 subject experts from more than 500 institutions across 50 countries contributed, and the paper carries 1,155 authors, which is itself a signal of how the set was assembled.
The dataset holds 2,500 questions across more than a hundred subjects. It was finalized on April 3, 2025, and published in Nature on January 28, 2026. A rolling variant, HLE-Rolling, launched on October 8, 2025 to give the benchmark a moving target as models improve against the static set.
At publication the result was unambiguous: "State-of-the-art LLMs demonstrate low accuracy and calibration on HLE." Calibration is the quieter half of that finding and the more useful one for a buyer. A model can be wrong; the question is whether it knows it might be.
How is a question scored?
Automatically, against a known answer.
The set is built from "multiple-choice and short-answer questions suitable for automated grading." Each question has one verifiable solution, so scoring needs no rubric, no panel, and no judgment about partial credit. That design choice is what makes the benchmark cheap to re-run and comparable across labs.
It also means the score is a straight accuracy figure on a fixed set, which is the format most vulnerable to the configuration problem below. There is no place in the number to record how the model arrived at the answer.
What does a high score not tell you?
Whether the model knew any of it.
This is the point of the post, so here is the collision stated plainly.
The benchmark's inclusion rule is that a question "cannot be quickly answered via internet retrieval." The maintainers went further and cleaned the set after the fact: following a bug bounty programme that closed on March 21, 2025, "questions flagged in the bug bounty program and searchable questions have been removed and replaced." Searchability was treated as a defect serious enough to justify replacing questions.
Now read how the scores are produced. Anthropic's Claude Opus 4.6 methodology footnote states that Claude models run "with tools" were run "with web search, web fetch, code execution, programmatic tool calling, context compaction triggered at 50k tokens up to 3M total tokens, max reasoning effort, and adaptive thinking enabled."
Web search. Web fetch. Up to three million tokens of context. On a benchmark whose questions were filtered, and then re-filtered, for not being findable by searching.
That is not cheating and Anthropic is not hiding it. The footnote is right there, the runs are labeled "with tools," and the configuration is a reasonable way to show what the product can do. But the resulting number does not measure the thing the benchmark was constructed to measure. It measures whether a model can run a research process against a hard question: decompose it, search, fetch, execute code, and assemble an answer.
That is a genuinely valuable capability. It is also a different capability, and it deserves a different column.
The practical consequence is that a tools-enabled HLE score and a no-tools HLE score are not comparable, and neither is comparable to the figures in the original paper. If someone quotes an HLE number at you, the first question is whether the model had the internet.
How do the labs configure it?
Maximally, and the footnote is the only place you will find out.
Every element of that configuration lifts the score. Max reasoning effort costs more per query than default, the same cost-as-capability trade ARC-AGI makes explicit. Context compaction at 50k tokens, running up to 3M total, is a large amount of working memory, and what a context window actually retrieves is its own measurement problem. Programmatic tool calling means the model can orchestrate its own retrieval rather than being handed a single search result.
None of this is available to you at a normal price point, and none of it is what happens when a user types a question into a chat window. The gap between benchmark conditions and deployment conditions is wider here than on almost any other row in the table, which is precisely what reading the footnote is for.
Is it contaminated?
Partly, by design, and the maintainers built for it.
The 2,500 questions are public, which creates the standard exposure for any released set. The countermeasure is a held-out private set, one of several defences we weigh up in our post on contamination: "We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting." Comparing public-set and private-set performance is the cleanest contamination check any benchmark in this series offers.
HLE-Rolling adds a second defence by refreshing the target over time, in the same spirit as rolling-cutoff benchmarks elsewhere.
The awkward wrinkle is that the tools configuration partly moots the question. Contamination matters because a model might have memorized an answer. When the model is permitted to search the live web mid-question, memorization stops being the only shortcut, and the boundary between recall and retrieval blurs in a way the original design was specifically trying to prevent.
What should this mean if you are buying AI for your business?
Three things.
Ask whether tools were on. It is the single most important qualifier on any HLE figure, and it is usually absent from the slide. A tools-enabled score tells you about a research agent. A no-tools score tells you about the model.
Prefer the calibration story to the accuracy story. The original finding was low accuracy and low calibration. For business use, a model that is wrong 30% of the time and flags its uncertainty is more useful than one that is wrong 25% of the time with total confidence, because only the first can be supervised cheaply.
Recognise which number is closer to your use case. If you are buying something to answer questions from its own knowledge, the no-tools figure is your evidence, and GDPval is the closer read on whether it can produce work. If you are buying a research assistant, the tools-enabled figure is, and you should be reading it as such rather than as a knowledge score.
That last distinction is not academic to us. The research behind these posts is exactly the tools-enabled task: take a claim, find the primary source, fetch it, and check whether the specific sentence is there. It is also where the failures show up, including ours. Writing our post on vendor-reported scores we reported that a widely circulated benchmark figure had no primary source, and had to correct it: the figure was published, inside a table image a text fetch cannot read. We have also cut a widely circulated statistic from a published post after it failed the same check, and said so in the post. Retrieval capability is what makes that work possible. Retrieval capability is also not knowledge, and a benchmark that scores the two together will flatter both.
Frequently Asked Questions
Why is it called the last exam?
Because it was framed as the final benchmark of its kind, an attempt to build a closed-ended academic test with enough headroom that models could not saturate it quickly. Whether the name holds up is an empirical question, and the launch of a rolling variant within months suggests the authors were not counting on it.
Does a rising HLE score mean models are getting more knowledgeable?
Not necessarily, and this is the trap. If the tools configuration is getting richer between releases, some of the improvement is better research scaffolding rather than better knowledge. Compare like configurations or the trend line is measuring two things at once.
Is the private held-out set available to check a vendor's claim?
No, and that is the point of it. Its value to you is indirect: if a lab reports both public and private figures and they diverge sharply, that divergence is contamination evidence you could not otherwise obtain.
Should a small business care about a PhD-level exam at all?
Only as a ceiling indicator. Nothing in your operations resembles an HLE question. Its usefulness is in showing which models still have headroom on hard reasoning, which is weak evidence about everything else and better than none.
Of all the rows in a current announcement table, this is the one where the gap between what the benchmark was built to measure and what the reported number represents is widest. The fix is one question, asked before you accept the figure: were the tools on?
We write these because we buy this software and build on it. If you want the same scrutiny applied to a tool someone is selling you, see ClickWerxs AI services or get in touch.
Sources
- Phan, Gatti, Han et al., "Humanity's Last Exam" — 2,500 questions; 1,155 authors; "multiple-choice and short-answer questions suitable for automated grading"; questions have "a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval," developed "globally by subject-matter experts"; framed as "the final closed-ended academic benchmark of its kind with broad subject coverage"; motivated by LLMs achieving "over 90% accuracy on popular benchmarks like MMLU"; "State-of-the-art LLMs demonstrate low accuracy and calibration on HLE." arxiv.org/abs/2501.14249 (retrieved 2026-09-08)
- Humanity's Last Exam project site — organized by the Center for AI Safety and Scale AI; nearly 1,000 subject experts from over 500 institutions across 50 countries; 2,500 questions across over a hundred subjects; finalized April 3, 2025; published in Nature January 28, 2026 (Nature 649, 1139–1146); HLE-Rolling launched October 8, 2025; bug bounty closed March 21, 2025, after which "questions flagged in the bug bounty program and searchable questions have been removed and replaced"; "We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting"; "HLE tests structured academic problems rather than open-ended research or creative problem-solving abilities." lastexam.ai (retrieved 2026-09-08)
- Anthropic, "Claude Opus 4.6" — Claude models run "with tools" were run "with web search, web fetch, code execution, programmatic tool calling, context compaction triggered at 50k tokens up to 3M total tokens, max reasoning effort, and adaptive thinking enabled." anthropic.com (retrieved 2026-09-08)
- ClickWerxs editorial record — a widely repeated SWE-bench Pro figure for Claude Fable 5 was reported by us in September 2026 as having no primary source, and corrected on 2026-09-13 once the source was read in-browser: it is the first row of an image table on Anthropic's Fable 5 announcement, in a column covering both Mythos 5 and Fable 5. The 68% local-AI-Overview trigger figure was cut from our published post
ai-seo-local-businessafter failing primary-source verification, with the cut disclosed in the post itself. Operator data, on record.
Benchmark figures cited in this post are snapshots tied to a specific model, date, tool configuration and effort setting, and they change with every release. ClickWerxs builds on commercial AI models and earns revenue from software built with them. This post is operator analysis, not investment or procurement advice.
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut