TL;DR: MRCR hides several near-identical passages across up to a million tokens of conversation and asks the model to return, say, the second one. The task hardens as the copies multiply. That is why an advertised context window and usable memory are different products, and why the document-upload feature disappoints people who expected the number on the box.
The one-sentence answer: MRCR measures whether a model can find and tell apart specific items buried in a huge context, which is the property the marketing term "context window" quietly implies but does not promise.
Context windows are the most-marketed number in AI after benchmark scores themselves. A million tokens, the page says, and the buyer hears: it will remember my whole document set.
MRCR is the row in the announcement-table guide that tests the version of that promise people actually rely on. The gap between window and recall is measurable, and this benchmark is the measurement.
What does MRCR actually test?
The name is multi-round co-reference resolution. The setup, from OpenAI's public dataset card: a long synthetic conversation contains several near-identical "needles," and the model must retrieve a specific instance. The card's own example is the cleanest description: the conversation contains multiple poems about tapirs, and the model is asked to "return the 2nd poem about tapirs."
One needle in a haystack is the classic long-context demo, and models got good at it. MRCR plants two, four, or eight nearly identical needles and asks for one by position. Now the model cannot just find a distinctive passage; it has to track every copy, keep their order straight, and not blend them. The dataset spans context bins from 4,096 tokens up to 1,048,576, with 438 distinct entities and ten writing formats to prevent shortcut heuristics.
Who built it?
Google DeepMind originated the format; OpenAI's dataset card attributes MRCR to DeepMind's long-context evaluation work published in September 2024 (the "Michelangelo" paper, which introduced evaluations for reasoning over long context beyond simple retrieval and found models showing "significant room for improvement in synthesizing long-context information"). OpenAI then expanded the difficulty and released the dataset publicly for reproducibility.
That cross-lab lineage is worth noticing. One lab designed it, a competitor hardened and open-sourced it, and a third quotes it in announcements. Benchmarks with that pedigree are rarer and more trustworthy than single-lab specials.
How is it scored?
Not pass/fail. The response is compared to the correct needle with a sequence-match ratio (Python's difflib SequenceMatcher, per the dataset card), so a partially right answer earns partial credit. One sharp edge: the model must prepend a specified alphanumeric hash to its answer, and a missing hash scores zero regardless of content. Instruction-following is silently part of the retrieval grade.
There is also a footnote-grade detail hiding in the card's changelog: a December 5, 2025 bugfix corrected samples where roughly 10% contained more needles than intended and about 5% had incorrect ground truth. A benchmark is software. It has bugs, and versions of it graded models against wrong answer keys. Scores from before and after that fix are not quite the same measurement, which is one more reason the footnote checklist applies to the dataset as much as the model.
Why is 8-needle at 1M the variant labs quote?
Because it is the ceiling. Eight near-identical items spread across a million tokens is the hardest published cell in the grid, and headroom is what announcement tables need.
Configuration first, per the series rule, then the number: Anthropic's Claude Opus 4.6 announcement reports 76% on what it describes as the 8-needle 1M variant of MRCR v2, against 18.5% for Sonnet 4.5 on the same variant (announcement figures, source below). Read those two numbers together and the lesson is not that one model is good. It is that two models of the same family, both marketed with very large context, sit 57 points apart on the same recall task. The window tells you what fits. It does not tell you what the model can do with what fits.
Where did this lineage start?
Needle-in-a-haystack came first: hide one distinctive sentence in a long text, ask for it back. Useful, and quickly saturated. NVIDIA's RULER benchmark (April 2024) extended it with multi-hop tracing and aggregation, and produced the finding that defined the field, worth labeling a legacy benchmark now but still the cleanest statement of the problem: of 17 models tested, despite all claiming context sizes of 32K tokens or more, only about half maintained satisfactory performance at 32K. Claimed length and effective length diverged in the first generation of long-context models, and MRCR is the current, harder instrument for measuring that same divergence at millions of tokens.
What does a good score not tell you?
Your documents are not synthetic conversations about tapir poems. MRCR's needles are deliberately artificial so that no training data helps; your use case has messier structure, domain vocabulary, and questions that require synthesis rather than retrieval. The DeepMind work that originated MRCR treated retrieval as the floor, not the ceiling, and reported models much weaker at synthesizing across long context than at fetching from it. A model can ace retrieval and still write a mediocre summary of your contract set.
Partial credit also flatters. A sequence-match score of 76% is not "right 76% of the time"; it is an average closeness that can include a lot of near-misses. For a legal or financial lookup, a near-miss is a miss.
Is it contaminated?
Less than most, by construction. The needles are synthetic, generated across 438 entities and ten formats, so no training corpus contains the answers in advance, and generating fresh samples is cheap if a specific release leaks into training data. The honest caveat runs the other direction: the dataset is public, so a lab could tune retrieval behavior against MRCR's specific format, and a model polished for tapir-poem lookups has still never seen your contracts. Format-tuning is the contamination risk here, not answer leakage.
What should this mean if you upload documents?
Three practical rules fall straight out of the benchmark's shape.
First, position and repetition are the enemies, not raw length. If the answer appears once and is distinctive, long context works well. If your corpus contains eight near-identical versions of a clause (every construction contract, every insurance policy), the retrieval task you are giving the model is exactly MRCR's hard case.
Second, test recall at your real length before trusting it. The public dataset exists; the cheaper version is to plant known answers in your own documents and ask for them by position.
Third, design for retrieval instead of stuffing. In our assistant builds we do not hand the model a client's entire document set just because the window technically fits it. The assistant retrieves the relevant slice and reasons over that, because measured recall at long lengths, not the advertised window, is what governs behavior in production. That design choice is this benchmark's finding, applied.
Frequently Asked Questions
So is a bigger context window pointless?
No. A bigger window genuinely raises the ceiling on what can be processed at all, and single-needle retrieval at long lengths works far better than it did in the RULER era. The correction is to treat the window as workspace rather than memory: room to think with materials, not a guarantee of recalling any specific line in them. Buy the window for its ceiling; verify the recall for your workload.
Do all labs report the same MRCR number?
No, and the variant is the variable. Needle count (2, 4, or 8), context bin (4K up to 1M), dataset version (the December 2025 bugfix changed the answer keys on a slice of samples), and the v2 designation all move the number. Two "MRCR" scores from different labs are only comparable if every one of those matches, which is the footnote problem in miniature.
Does retrieval-augmented generation make long context irrelevant?
They solve different halves. Retrieval narrows a large corpus to candidate passages; long context lets the model hold more of those candidates, plus instructions and history, at once. The strongest production pattern uses both, and MRCR is a caution against the design that uses neither well: dumping everything into the window and trusting the model to find the right copy.
Before an AI assistant touches your document set, the question to answer is not how big the window is; it is whether the system can reliably surface the one clause that matters from twenty lookalikes. That is a testable property, and building systems that pass that test is the work we do.
See ClickWerxs AI services or get in touch.
Sources
- OpenAI, MRCR dataset card — task design, needle counts (2/4/8), context bins 4,096 to 1,048,576 tokens, SequenceMatcher scoring and hash rule, DeepMind attribution, December 5, 2025 bugfix note. huggingface.co/datasets/openai/mrcr (retrieved 2026-08-13)
- Vodrahalli et al., "Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries" — the DeepMind long-context evaluation work MRCR originates from; finding of significant room for improvement in synthesizing long-context information. arxiv.org/abs/2409.12640 (September 2024; retrieved 2026-08-13)
- Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?" — multi-hop and aggregation tasks beyond needle-in-a-haystack; roughly half of 17 tested models maintaining satisfactory performance at 32K despite 32K+ claims. Legacy benchmark, April 2024. arxiv.org/abs/2404.06654 (retrieved 2026-08-13)
- Anthropic, "Introducing Claude Opus 4.6" — MRCR v2 8-needle 1M variant description and reported figures (Opus 4.6 76%, Sonnet 4.5 18.5%). anthropic.com (retrieved 2026-08-13)
Kaleb Dickhaut — Founder, ClickWerxs. Kaleb built ClickWerxs from the ground up, from payment processing ISO to the Command Center platform to the AI SEO methodology the blog runs on. He has onboarded hundreds of small businesses onto payment and CRM systems. linkedin.com/in/kaleb-dickhaut