Just 31 of 144 attempts succeeded when eight leading AI systems tackled a demanding test of autonomous scientific discovery. Researchers at Northwestern University built a benchmark to evaluate whether general-purpose AI agents can act like real scientists. That meant choosing which experiments to run, learning from the results, and building predictions about what they hadn’t yet tested. The answer, for now, is: sometimes and the failures prove more instructive than the successes.
The project is called Autonomous Science Bench. It covers six genuine scientific challenges drawn from published research, spanning everything from engineering bacterial enzymes to hunting for catalysts that convert carbon dioxide into fuel. Instead of handing AI systems a finished dataset to analyze, the benchmark forces agents to actively acquire their own data, one experimental batch at a time. Each agent works under a fixed budget. Every measurement costs part of that allowance. An AI that wastes its budget on the wrong designs fails, regardless of how capable it appears otherwise.
The best-performing system Anthropic’s Opus 5.5 model running inside a coding environment called Claude Code cleared 67% of its assigned campaigns. That tops the rankings. Yet the same system failed all three of its attempts on the biosensor campaign, a challenge that only OpenAI’s GPT-5.6 configurations managed to crack. Even the strongest AI agent had campaigns it simply could not solve.
At the other end of the rankings, a smaller Anthropic model called Haiku 4.5 passed zero of its 18 attempts. Open-source models on a leaner software scaffold fared little better, each managing just one or two successes from eighteen trials. Those few wins came at a steep price: one model burned through 28 hours of runtime to match results that a commercial system produced in just 6 hours.
The benchmark was designed not only to rank AI systems but to diagnose exactly how they fail. A campaign only passes if the AI clears two separate hurdles at once. It must find enough promising experimental designs within its budget. It must also deliver a model that correctly predicts which untested designs will perform well. These two objectives often pull in opposite directions. The results show that today’s AI agents, even the most capable ones, struggle to satisfy both simultaneously.
What Makes Autonomous Scientific Discovery Different from Standard AI Tests
Most AI benchmarks for science hand a model a complete dataset and ask it to draw conclusions. The agent reads fixed information and produces an answer. Autonomous Science Bench works differently. The agent must decide what to measure before it can see anything useful.
Each campaign provides a pool of possible experiments enzyme variants, catalyst compositions, or reaction conditions alongside a small starting dataset of prior measurements. The agent selects batches of designs, requests measurements from a simulated lab service, and uses those results to shape its next batch. The lab enforces strict rules: no re-testing already-measured designs, no exceeding the batch size, no skipping rounds. Invalid requests waste no budget, but they do consume time.
This structure reflects how science operates at the frontier. A chemist screening drug candidates can’t test every compound. A materials scientist studying magnetic alloys can’t characterize every composition in a phase diagram. Intelligent selection, guided by theory, prior data, and iterative experimental results, separates efficient science from expensive guesswork.
What sets this benchmark apart is its use of general-purpose AI coding agents rather than specialized optimization algorithms. The agents Claude Code, Codex, and mini-SWE-agent are the same tools software developers use for everyday programming tasks. They received no custom scientific prompts and no specialized tools. Each one got only the task instructions and a command-line interface to communicate with the simulated laboratory.
Prior work on AI-driven experimentation largely used purpose-built acquisition functions or simulation environments where responses were computed rather than drawn from real data. Autonomous Science Bench asks something harder: can an AI that writes code and reasons about problems figure out experimental strategy entirely on its own, with no scientific scaffolding beyond the task description?
Six Real Scientific Campaigns That Put AI Agents to the Test
The benchmark contains six campaigns, two each in biology, chemistry, and materials science. Each draws on published laboratory measurements, grounding these tests in actual scientific data rather than synthetic puzzles.
Biology:
The nuclease engineering campaign centers on NucB, a bacterial enzyme that destroys the extracellular DNA holding biofilms together. Biofilms make bacterial infections notoriously hard to treat; a more active NucB could become a therapeutic tool. The pool holds 20,000 multi-substitution protein variants. The agent can measure fewer than 2% of them. To pass, it needs 190 high-activity variants spread across 175 distinct protein families a threshold that stops agents from simply copying the same good design repeatedly with minor variations.
The biosensor campaign targets lead detection in drinking water. The protein naturally responds more strongly to zinc than lead. The AI must find variants that invert that response in just 30 measurements across three rounds. Only 18 of the 49 designable amino acid positions can hold qualifying variants at all, making systematic coverage nearly impossible on such a tight budget.
Chemistry:
The propylene campaign asks agents to identify which dilute copper alloys most efficiently convert carbon dioxide into propylene fuel. Only 14 of the 736 orderable designs qualify as hits a ratio that makes this one of the harder needle-in-a-haystack problems in the benchmark. The original paper had its own candidate shortlist, but the benchmark extended the design space to all 980 possible dilute alloys, blocking that shortcut entirely.
The Suzuki campaign tests a standard pharmaceutical chemistry scenario: find the best combinations of ligands, bases, and solvents for a coupling reaction across twelve molecular starting materials. Yields aren’t comparable across substrates. An excellent condition for one substrate pair may fail completely on another. Both grading tests account for this per-substrate normalization, shutting off any shortcut based on substrate selection.
Materials:
The oxygen-evolution campaign seeks metal oxide compositions for efficient water splitting a step central to green hydrogen production. Its unusual requirement is that the compositions the model gets scored on cannot be ordered at all. Agents measure two- and three-metal compositions, then must deliver a model that accurately ranks four-metal compositions they have never tested.
The coercivity campaign studies magnetic Fe-Co-Ni alloy compositions, seeking those with the highest coercivity resistance to demagnetization. Two separate performance peaks exist in the composition terrain. The benchmark never discloses this fact to the agent. Nine of the 24 attempted trials completed all their rounds without ever locating the second peak.
The Two-Gate System That Defines Success in Autonomous Science
The benchmark’s most distinctive feature is its grading method. Each campaign requires clearing two separate tests. Both must pass for the campaign to count as resolved.
The discovery gate evaluates what the agent measured. Did it find enough genuinely good designs? Each campaign defines “good” scientifically. The nuclease campaign requires 190 high-activity variants across 175 protein families. The biosensor requires 21 selective lead-detecting variants. The propylene campaign requires at least 7 catalysts above the yield threshold. These thresholds were calibrated so that a reference solution using the same data and budget passes with a 10% margin, while simple or static selection strategies fail at least 90% of the time.
The learning gate evaluates the predictive model the agent delivers at the end. Can it correctly rank designs it never measured? The metric is normalized discounted cumulative gain (NDCG) a standard measure that rewards models for placing genuinely good designs near the top of their prediction list. The evaluation set was never available for ordering.
The research team constructed single-gate-passing strategies for four campaigns to demonstrate why neither gate alone suffices.
On biosensors, one approach ranked protein variants purely by how many substitutions they shared with published sequences. It learned nothing from its own experimental results. Yet it found enough hits to pass discovery. It built no genuine model and failed learning entirely.
On the Suzuki campaign, an oracle strategy that selected the actual best 192 wells passed discovery. Yet the model fitted to those wells failed to rank the remaining dataset correctly.
On propylene, the source study’s own Bayesian optimization protocol the algorithm the original researchers developed passed learning but found too few qualifying designs to pass discovery. Even the researchers’ own method fell short.
On coercivity, systematic coverage of the iron-poor region, which the source paper identified as promising, passed learning while missing the second performance peak entirely. Each of these examples illustrates a real failure mode. Neither gate can stand in for the other.
Results: How AI Agents for Autonomous Scientific Discovery Actually Performed
Across all 144 trials, 31 passed both gates. Here is how each configuration ranked:
Opus 5.5 / Claude Code: 12/18 trials resolved (67%) total cost $172.90
GPT-5.6 Terra / Codex: 8/18 trials (44%) total cost $51.57
GPT-5.6 Luna / Codex: 5/18 trials (28%) total cost $6.91
Sonnet 5.5 / Claude Code: 2/18 trials (11%) total cost $7.85
GLM-5.3 Flash / mini-SWE-agent: 2/18 trials (11%) total cost $25.69
Qwen3.8 27B / mini-SWE-agent: 1/18 trials (6%) total cost $21.89
MiniMax M3 / mini-SWE-agent: 1/18 trials (6%) total cost $28.37
Haiku 4.5 / Claude Code: 0/18 trials (0%) total cost $7.62
The cost comparison is striking. GPT-5.6 Luna cost $6.91 total, working out to $1.38 per resolved campaign. GPT-5.6 Terra came to $6.45 per resolution. Opus 5.5 reached $14.41 per success. Moving from Luna to Opus multiplied total spending by 25 while improving the success rate by only 2.4 times steeply diminishing returns as costs climbed.
The open-source configurations offered no efficiency advantage. Qwen3.8 27B cost $21.89 for one resolved campaign. MiniMax M3 cost $28.37 for one. GLM-5.3 Flash cost $25.69 for two, while consuming 28 hours of agent runtime against GPT-5.6 Luna’s 6 hours for similar or worse results.
Not a single trial failed because the AI gave up or submitted a broken model. Every configuration completed all experimental rounds and delivered a valid prediction function. The failures were substantive wrong design choices, insufficient hits accumulated, or models that simply could not generalize to unseen conditions.
The oxygen-evolution campaign was the most tractable, resolved by four different configurations in 8 of 24 total attempts. Coercivity was the hardest: 3 resolutions from 24 attempts, with two going to Opus 5.5 alone.
How Two Gates Revealed Hidden Weaknesses in AI Autonomous Discovery
Of the 113 unresolved trials, 25 failed only the discovery gate, 30 failed only the learning gate, and 58 failed both. Nearly half of failures passed at least one gate a pattern invisible to any single success metric.
The oxygen-evolution campaign saw 12 unresolved trials fail only learning, none fail only discovery. Finding good orderable compositions was achievable. Getting the model to generalize to the unmeasurable four-cation evaluation region was not.
Propylene showed the opposite: one unresolved trial failed only discovery, while 16 failed only learning. Most agents found qualifying hits. Building a model that transferred accurately to unseen designs proved far harder.
Biosensor split differently: 12 unresolved trials failed only discovery, zero only learning. Models were manageable. Finding 21 selective hits in 30 measurements was not.
Coercivity challenged both at once. Eighteen of 21 unresolved trials failed both gates. Nine of the 24 total trials never located the second performance peak.
Comparing configurations exposed contrasting failure signatures. Sonnet 5.5 passed learning in 11 of 18 trials but discovery in only 5. GPT-5.6 Luna passed discovery in 11 but learning in only 6. Both configurations had modest overall success rates, but they were failing for completely different reasons. A team working to improve Sonnet 5.5 would focus on experimental selection. One improving Luna would address modeling. These are genuinely different problems, requiring different technical solutions. A single pass/fail success rate would never reveal which problem to fix.
The Spearman correlation between normalized discovery and learning scores across all 144 trials was 0.43. Within individual campaigns, that figure ranged from 0.30 to 0.71. The two objectives are related but they are far from interchangeable.
Budget Patterns Across Rounds: When and Why Agents Go Wrong
Laboratory ledger records showed how agents distributed their measurements over successive rounds.
By round 2, even failing agents selected qualifying designs 2 to 7 times more often than pure random chance. The agents were learning from their results and adapting strategies. They were not choosing blindly at every step. Above-random performance still didn’t guarantee success 83 of the 113 unresolved trials still failed the discovery gate despite showing above-random hit rates.
Timing proved decisive in several campaigns. On biosensors, resolved trials averaged a 68% hit rate in round 1. Failed trials only reached 71% in the final round, with no budget remaining to capitalize. Starting on the right track immediately was not merely helpful it was essential.
Suzuki showed the same principle in a cleaner form. From round 2 onward, successful trials found qualifying conditions at a 29% rate, against 15% for failed trials. Early momentum predicted final success.
Propylene was different. Both successful and failed trials maintained similar hit rates throughout all rounds. The difference between winning and losing appeared entirely in the final predictive model not in how well agents chose experiments. Two campaigns with nearly identical selection patterns could end up with completely different outcomes depending on how well the agent modeled what it had found.
Planning Ahead: The Adaptive Scheduler for Live Laboratory Use
The research team also proposed a practical framework for when these evaluations connect to actual physical laboratory equipment, where measurement costs are real and resources are finite.
The adaptive budget scheduler watches agent performance during live experiments and adjusts measurement quotas round by round. Agents surpassing a performance threshold receive full allocations. Those falling short receive reduced ones. Remaining capacity flows toward experiments that will most improve the evaluators’ ability to identify the most capable agent.
The system uses Bayesian statistics to maintain running estimates of each agent’s probability of success. It starts from prior results based on recorded experiments and updates continuously as live data arrives. The goal is to identify the best-performing agent while using the fewest possible physical measurements.
This scheduler is not yet deployed version 0.1 relies on recorded measurements and surrogate models rather than live robot experiments. However, the full algorithm is specified in the paper, including smoothing procedures that combine prior rates with live evidence and a greedy approximation for the resource allocation step.
The long-term plan connects the benchmark to actual robotic laboratories. Fresh data from physical experiments would also address a known limitation of the current setup: some AI models may have encountered published results during training. Live experiments would eliminate any possibility of memorized answers influencing the outcome.
What This Means for AI in Real Research Laboratories
For institutions thinking about deploying AI agents in actual laboratories, the numbers carry direct practical significance.
Consider what the nuclease campaign alone implies. The pool holds 20,000 protein variants. Random selection finds roughly 76 qualifying hits. The reference solution finds 278. An AI agent successfully running this campaign would recover nearly four times as many promising variants from the same experimental budget. In a real high-throughput laboratory, that difference could determine whether a screening campaign yields actionable leads or not.
Even the most expensive configuration tested Opus 5.5 at roughly $14 per resolved campaign costs a fraction of what physical experiments demand in any of these scientific domains. High-throughput protein screening, electrochemical catalyst testing, and magnetic materials characterization all involve significant reagent, equipment, and researcher time per data point. AI costs are negligible by that comparison.
The reliability problem, however, is serious. A 33% failure rate on a campaign requiring hundreds of laboratory operations means genuine waste of real resources. For a self-driving laboratory operating around the clock on expensive equipment, one failure in three is a meaningful operational cost that matters beyond the AI invoice.
The dual-gate framework offers a practical diagnostic tool for operators. An AI consistently passing discovery but failing learning needs better statistical modeling of its results. One passing learning but failing discovery needs better experimental selection logic. A single aggregate success metric would never tell an operator which problem to address or where to invest in improvement.
The coercivity result makes the challenge most vivid. Nine AI agents ran complete campaigns four rounds of experiments, strategy updates after each round, final models delivered while entirely missing one of two performance peaks in the alloy system. Nothing in their feedback told them they were missing it. Real science is filled with unknown unknowns that look like certainty from the inside, and the Autonomous Science Bench captures that difficulty with unusual precision.
Conclusion: 31 Successes Is Just the Starting Point for Autonomous Scientific Discovery
The 31 successful trials out of 144 look modest. Context changes the reading, though. The benchmark was specifically calibrated so that a reference solution always passes. This is a hard, realistic challenge not an impossible one. The best AI agent for autonomous scientific discovery cleared it two-thirds of the time.
The benchmark’s code, campaigns, and calibration scripts are publicly available on GitHub. The leaderboard is open to all. Research groups can submit their own AI systems for evaluation or contribute new campaigns from their own laboratory data, gradually expanding the benchmark’s scientific coverage.
Future versions will add more campaigns across more fields. The planned connection to actual robotic laboratories will generate genuinely fresh experimental data measurements no model has ever seen, from experiments no paper has yet described. That is when the hardest tests of autonomous scientific discovery will truly arrive.
For now, the 31 successes stand as an honest early measure of where AI agents in science currently sit: capable, improving, and not yet reliable. The gap between 31 and 144 is the work still ahead.