The Lab AI Test That Rewards Saying: We Do Not Know

OpenAI introduced BioBasicsBench, a 100-task evaluation aimed at that less glamorous but essential part of scientific work: reading messy primary evidence, checking whether controls hold up, spotting procedural mistakes and recognizing when an experiment has failed to answer its question. The benchmark was developed with enzyme-design company Ligo and focuses on routine wet-lab analysis across protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning.

This is a consequential shift in what an AI benchmark asks of a model. A conventional science question might display a polished chart from a paper, ask what it means, and reward a known answer. A working laboratory rarely looks like that. The evidence arrives before the narrative: a faint band on a gel, handwritten notes beside a mislabeled tube, an odd image in a microscopy stack, a spreadsheet export with a missing control. The correct next move is often not an answer. It is a request for a rerun, an additional control, or a closer look at the setup.

The difference between a paper and a bench

BioBasicsBench feeds agents raw material in the formats scientists actually handle. Its tasks can include gel and blot images, microscopy images and z-stacks, colony plates, raw sequencing files, instrument exports, photos of apparatus and reagents, videos of bench work, protocols, notebook pages and reference material such as plasmid maps or ladder charts. One task may require the model to connect several of those artifacts rather than treating an image as an isolated visual puzzle.

That matters because scientific judgment is an exercise in relationships. A bright cell in one channel is not automatically a dead cell if another dye can bleed into the same detection window. A blank negative-control plate is not automatically reassuring if the label on its rim reveals the wrong antibiotic. Cells vanishing in a short video may look like a biological event until the agent recognizes that the microscope is moving through focal depth.

OpenAI’s examples make the point vividly. GPT-6 Astra identified overlapping fluorescence detection windows and recommended controls that could separate the signals, where GPT-5.6 Sol treated brightness as enough evidence to classify cells as dead. In another task, Astra caught the antibiotic mismatch on a negative-control plate. In a third, it interpreted changing focus across frames as a z-stack, not a mysterious disappearance of cells.

The interface implication is easy to imagine. The most useful laboratory agent may not look like an all-knowing chat box. It may behave more like a careful colleague who pins the relevant image region, circles a plate label, links it to the protocol, and tells the scientist exactly which assumption is unsupported. Its greatest contribution could be shortening the distance between “something looks wrong” and “here is why this result cannot guide tomorrow’s work.”

A score is not a successful experiment

The headline result is real progress, but it needs precise interpretation. At the xhigh reasoning setting, GPT-6 Astra received 66.7% mean reward, compared with 43.8% for GPT-5.6 Sol. The reported metric is average credit against expert-created rubrics, not the share of experiments that an agent completed flawlessly.

That distinction is not a footnote. In wet-lab work, an agent can notice an important flaw, make a correct measurement and still choose the wrong next step. A rigid exact-answer benchmark often collapses that partly useful reasoning into a zero. A more forgiving score can instead acknowledge that the model found the right problem but failed to articulate the required qualification, or calculated a number correctly but did not explain why a control limits the conclusion.

Rubrics therefore create a richer picture than simple right-or-wrong scoring. OpenAI says each task breaks performance into criteria that can cover the final conclusion, a numerical calculation, supporting reasoning, or the recognition that evidence is insufficient. The process also included a task author, review by a Ligo scientist who did not write the task, external review by a PhD-level wet-lab scientist, and final internal review.

Raw evidence and questionobservations, measurements, and the task

evidence and question

Agent answerconclusion, calculation, reasoning, uncertainty

conclusion, calculations, reasoning, and uncertainty

Expert rubric with separate criteriaassesses distinct dimensions of performance

Conclusion • Controls • Calculation • Insufficient evidence

Partial-credit scorereflects what the agent got right and where evidence is insufficient

Partial credit is particularly appropriate when the question is whether an agent can be a safe collaborator. Still, it can also conceal a practical problem. Two agents could both score 67%, while one routinely identifies failure modes but cannot recommend recovery, and the other produces good calculations while sometimes makes confident, consequential overclaims. A lab manager deciding whether to trust an assistant needs error categories, not only one blended average.

That is why the remaining failures matter more than the leaderboard gap. Astra could correctly group bands in a cloning-related PCR result yet reverse which pattern represented the intended construct, selecting the wrong colonies for downstream work. It inspected contradictory microscope observations but gave greater weight to evidence matching its preferred interpretation. And it overcounted colonies by roughly 12% in one routine measurement because its marks missed nearby colonies and included empty agar.

Build tests around decisions, not displays

A strong laboratory-agent evaluation should treat every task as a decision point. The model should first inventory the evidence it has received, identify the experimental goal, and verify that the controls can support the intended comparison. Only then should it calculate, conclude, or propose a next action.

The resulting answer should be structured enough to audit. A useful format would separate observations from inferences, name the controls inspected, calculate any requested result, state confidence, and list the smallest next experiment that could resolve ambiguity. It should also force a model to say whether it is recommending action or refusing to recommend action. This makes it harder for a fluent explanation to hide a missing premise.

The test set itself should be designed to punish shortcutting. Hold back tasks whose evidence has never appeared online or in documentation likely to enter training data. Randomize superficial cues such as filenames and image styling. Include believable distractors that reward checking metadata rather than pattern matching. Create matched task pairs in which a single changed label, control, or protocol step should reverse the answer. And use independent experts to grade not merely whether the final sentence sounds plausible, but whether the evidence cited actually warrants it.

BioBasicsBench takes meaningful steps in that direction by using scientists’ own experimental data and multiple layers of task review. But the benchmark’s private task set means outside researchers cannot yet independently reproduce its results or inspect every rubric judgment. That does not invalidate the reported improvement. It does mean the 66.7% figure should be read as a provider-reported signal of progress, not as a universal certificate that an agent is laboratory-ready. That is an inference from the benchmark’s described evaluation design and its unpublished task materials.

The supplied reaction material for this story contained critical commentary only, not a supportive outside assessment. No opposing view should be invented to manufacture balance. The more constructive response is to ask for the evidence that would make the result easier to audit: releases of carefully decontaminated subsets, inter-rater agreement data, breakdowns by failure type, and prospective tests in which an agent’s recommendation is checked against a later human-run experiment.

Capability and safety meet at the same bench

There is a second reason to care about this kind of evaluation. Reliable troubleshooting is useful for discovery, but increasingly capable biological assistance also requires careful safety assessment. OpenAI’s GPT-5.5 System Card describes a separate suite of biological and chemical evaluations, including multimodal virology troubleshooting, open-ended protocol questions, tacit-knowledge testing and a troubleshooting benchmark based on expert-used procedures. It classifies GPT-5.5 as High capability in the biological and chemical domain and says corresponding safeguards are active.

That context clarifies why evidence-grounded tests are valuable. The goal is not simply an AI that knows more biology. It is one whose behavior can be inspected: an agent that identifies what it saw, distinguishes it from what it inferred, notices when controls fail, and declines to turn ambiguous data into an unwarranted instruction.

For scientists, the practical readiness question is not, “Did the model beat its predecessor?” It is, “When this agent tells me what to do next, can I see the chain of evidence, find the uncertainty, and safely override it?” BioBasicsBench begins to turn that question into something measurable. The future laboratory assistant will earn trust not by always sounding certain, but by knowing exactly when the experiment has not earned certainty yet.