The 82% Problem Isn't That AI Doesn't Understand Science. It's That It Picks the Wrong Method.
A new benchmark from Tsinghua built directly from peer-reviewed Nature-family papers found that the best AI coding agent — Claude Opus 4.7 — beat published research results on 17.8% of real scientific tasks. The failure analysis is more useful than the headline: 45.1% of failures trace to wrong method choice, not ignorance. That's a specific, fixable problem — and a very different story than 'AI can't do science.'
The 82% Problem Isn't That AI Doesn't Understand Science. It's That It Picks the Wrong Method.
When an AI agent fails at a real scientific task, it usually isn't because it doesn't understand the domain. It's because it chose the wrong approach. That's a specific and actionable failure mode — and the most important finding in NatureBench, a new paper from Tsinghua University and FrontisAI that benchmarked ten frontier agent configurations against 90 tasks drawn directly from peer-reviewed Nature-family publications.
The headline number is 17.8%: the share of tasks on which the best configuration (Claude Opus 4.7, running on the Claude Code harness) beat the published results from the original papers. The failure analysis is more instructive. Of unsuccessful attempts, 45.1% were attributed to wrong method choice — agents that understood the task but selected the wrong approach to solve it. Insufficient compute budget accounted for 24.4%. Among the successes, 82.7% came from what the paper calls methodological translation: recognizing a scientific problem as a standard ML pipeline and applying a known method to it. Only 8.3% of successes involved genuine domain-specific reasoning — the kind of scientific thinking that doesn't reduce to a template.
That breakdown is specific and falsifiable in a way that "AI is bad at science" is not. Agents are competitive on tasks that look like supervised learning problems. They fail when the problem requires something other than pattern-matching to a known ML approach.
What NatureBench is and why it's different
Most AI science benchmarks are constructed by researchers who write tasks that look scientific. NatureBench takes a different approach: the research team pulled 90 tasks directly from peer-reviewed Nature-family papers and containerized each one using NatureGym, a standardized pipeline that provides the data, environment, and evaluation criteria from the actual paper — while removing the source method. Agents have to find their own approach. They can't retrieve the original paper; web search is disabled. They get a 4-hour compute budget and iterative scoring through a standardized /evaluate endpoint.
The web-search-disabled protocol and the method removal are the critical design choices. Prior benchmarks that test agents on scientific tasks often allow literature search — which means the agent can find, read, and implement the original method, turning a scientific discovery task into a software engineering task. NatureBench doesn't allow that. It tests whether agents can solve real problems independently, which is the capability that actually matters for the "AI accelerates research" thesis.
The benchmark also provides the first rigorous domain-level breakdown I've seen of where agents succeed versus fail. Relational Reasoning tasks — structured, logic-driven problems that resemble ML pipeline work — show 60% match-SOTA rates. Biomedical Modeling tasks show 17.9%. The average isn't the story. The distribution is.
The immunology prediction problem
A week ago this publication covered the case of immunologist Derya Unutmaz, who found that GPT-5 Pro correctly predicted the results of an unpublished experiment — before he ran it. That story was real. NatureBench doesn't contradict it. But the two are measuring fundamentally different things, and conflating them produces a distorted picture of where AI research capability actually sits.
The Unutmaz case was expert-supervised, single-domain, context-rich, and guided: a world-class immunologist asking a focused question in his own field, providing the relevant experimental context, and getting a prediction back. That's a collaborative workflow. NatureBench is autonomous agents working alone across nine scientific domains, with no domain expert input, no access to the original method, and a compute clock running. These are different regimes.
The comparison matters because the AI research tool market is not selling collaborative workflows where domain experts guide agents through targeted tasks. It is selling autonomous capability — "your AI research assistant," "AI accelerates discovery," "AI-powered drug development." NatureBench measures the autonomous regime. The Unutmaz case is what happens in the best 17.8% conditions: expert supervision, focused domain, curated context.
If you're a pharmaceutical company being pitched an AI research tool, the relevant benchmark is NatureBench. Not Unutmaz.
What 45.1% means for the next generation of agents
The wrong-method-choice failure mode is the most actionable finding in the paper. It means agents are failing not at understanding tasks but at selecting approaches — a metacognitive problem rather than a domain knowledge problem. An agent that understood it was choosing poorly would choose differently.
This is a plausible place for near-term improvement: method-selection modules, tool-use frameworks that explicitly enumerate and evaluate approaches before committing, or structured reasoning steps that force agents to justify their method choice before implementation. The 24.4% of failures attributed to insufficient compute budget are also addressable — agents that ran out of time before finding a solution aren't demonstrating fundamental incapability, they're demonstrating resource constraints that will ease as inference becomes cheaper and faster.
I think NatureBench scores improve materially within 18 months on both of these fronts. The 17.8% figure is not a ceiling — it's a baseline measured with current agent designs under current compute constraints.
The limits worth knowing
90 tasks is a small sample to generalize across nine scientific domains. The g>0.1 criterion — requiring agents to beat published results by at least 10% — is conservative; real scientific progress often comes in smaller increments that still matter. The NatureGym containerization may introduce artifacts that differ from actual lab settings. And the web-search-disabled protocol, while defensible, handicaps agents in a way that real research deployments typically don't — most real workflows allow literature search.
These are legitimate critiques. They don't invalidate the 17.8% finding, but they suggest it's a lower bound on what agents can do in real research settings, not an upper bound. The paper is a quality control mechanism released at exactly the right moment — when "AI solves science" narratives are peaking with Nobel laureate hires, immunology predictions, and frontier lab science investment. It arms skeptics with something concrete and reproducible.
The benchmark design is sound. The sample is small. Both things are true. Watch for a NatureBench v2 with more tasks, or competing benchmarks that contest the methodology. Until then, 17.8% is the most honest number available for autonomous agent performance on real science — and the diagnosis of why the other 82% fail is the most useful thing the paper produced.