A blind benchmark hands AI only a paper's reference list. Frontier models recover the idea 3 to 15 percent of the time
A preprint posted to arXiv on 17 August introduces Reconstruction, a benchmark that gives a language model only a paper's pre-publication reference list and asks it to recover the paper's core idea. Tested one at a time, seven frontier models managed it 3 to 15 percent of the time. A multi-agent pipeline that ran competing hypotheses through a peer-review-like tournament reached 23 to 42 percent, and still missed most. The work is a preprint, not yet peer reviewed, but its stripped-down design is harder to game than the evaluations behind many AI scientist claims.
What the study did
A preprint posted to arXiv on 17 August 2026 introduces "Reconstruction," a benchmark built around a deliberately narrow question: can a language model recover a paper's core research idea when it is given only that paper's pre-publication reference list, and nothing else? No full text, no author names, no work published after the paper's own cutoff.
The benchmark comes from a team led by Shaolong Chen. It covers 643 papers across six scientific domains, and an independent language model acts as judge, matching each model's proposed hypothesis against the paper's held-out, real idea.
Three anti-leakage rules are the point of the design. A temporal citation cutoff removes any reference that postdates the paper, so a model cannot reason forward toward the answer. Anonymous reference IDs strip author names, closing off the shortcut of recognising a research group and guessing its next move. And each paper's bibliography is frozen, so nothing leaks in at test time.
The results
Tested individually, the seven frontier models scored match rates between roughly 3 and 15 percent. A multi-agent pipeline that pooled several models, had them review each other, and ran surviving hypotheses through a Swiss-tournament comparison, with no web search, raised that to roughly 23 to 42 percent. That is about a 2.4 times lift over the best single model, and it still left most ideas unrecovered.
Why it matters
Analysis. This reads as a measurement-design story more than a scoreboard. The value is in what the benchmark refuses to give the model. Many "AI scientist" evaluations hand the system full text, author identities, or signals from after publication, any of which lets a model retrieve an answer rather than reason toward one. Strip those away and the near-uniform failure across otherwise very different models suggests the gap is structural, not a matter of prompt wording or model size.
The multi-agent result cuts the other way and deserves as much attention as the low solo scores. The lift came from putting hypotheses through an adversarial, tournament-style filter, which is a rough machine analogue of peer review. If that holds up, the useful lesson for anyone building research tools is that the design that helps is a structured, self-critical pipeline, not simply a larger single model.
Caveats
This is a preprint. It has not been peer reviewed, and the authors describe it as a timestamped draft, with per-model and per-domain detail promised in later revisions. The match rates depend on a language model acting as judge, and the computational cost of the multi-agent gains is not fully reported. Treat the numbers as a first, self-declared reading rather than a settled finding.
Source: Shaolong Chen and colleagues, "Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies," arXiv:2608.16645 (submitted 17 August 2026, revised 19 August 2026).