A procurement team sees two coding agents separated by two percentage points and assumes it has found a winner. An engineer who has operated either system knows the comparison may be far less settled. One agent may have had faster tools, more memory, fewer queue delays, a more forgiving retry policy, or a harness that kept it from losing the thread halfway through a task. In agentic coding, the model matters profoundly. But the score on the leaderboard is often measuring a larger machine: model, scaffold, sandbox, scheduler, tests, and the clocks that decide when the whole effort is over.
A familiar scene is playing out inside software organizations. A technology leader has been asked to choose a coding agent. Salespeople have benchmark charts. Developers have strong opinions. Finance wants a clean answer about cost per resolved ticket. Security wants to know where code runs and what the agent can access.
Then someone points to a leaderboard.
The temptation is understandable. Software buyers have spent decades comparing databases, cloud instances, compilers, and developer tools using performance numbers. Benchmarks make a complicated decision feel legible. If one model resolves 63 percent of benchmark tasks and another resolves 60 percent, surely the first is better.
That conclusion can be right. It can also be premature.
A coding agent does not simply receive a question and produce an answer. It investigates a repository, searches files, writes code, installs packages, runs tests, reads failures, revises its approach, and eventually submits a patch. At every one of those steps, the surrounding system can help or hinder it. The agent is operating a workshop, not filling out an exam paper.
Anthropic made that point unusually concrete in a February 2026 study of infrastructure noise in coding evaluations. Holding the Claude model, harness, and Terminal-Bench 2.0 task set constant, the company found a six percentage point difference between its most constrained and least constrained resource configurations. That swing exceeded many of the gaps that attract attention on public leaderboards.
The lesson is not that coding benchmarks are useless, nor that models cannot be compared. It is more demanding than that. A serious benchmark must answer a harder question: what exactly did this score measure?
Was it raw model capability? Was it the effectiveness of the agent harness? Was it the quality and generosity of the infrastructure? Or was it a blend of all three?
Those are different things. Buyers, researchers, and model providers should stop treating them as interchangeable.
The agent is not the model
A static benchmark has a relatively clean shape. Give a model a prompt. Inspect the answer. Score it against a known solution or a rubric. There can still be disagreements about task design and grading, but the object under evaluation is close to the object being sold: a model responding to input.
An agentic coding benchmark has a different anatomy.
A task might begin with a bug report from a real software repository. The model needs a way to inspect source files, edit them, execute commands, read error output, and perhaps search documentation. It needs an agent loop that decides when to call a tool, what to do after the result returns, and when to stop. It needs a sandbox that has a particular operating system image, package cache, network policy, CPU allocation, memory ceiling, storage behavior, and timeout. Finally, it needs a grader that determines whether the resulting code actually works.
Anthropic’s own evaluation guidance defines the distinction clearly. The agent harness, sometimes called a scaffold, is the system that processes inputs, orchestrates tool calls, and returns results. The evaluation harness then provides tasks and tools, runs trials, records activity, grades outcomes, and aggregates the result. When someone evaluates an agent, they are evaluating the model and the harness together.
That stack is not incidental plumbing. It is the experiment.
Consider a simple task: fix a failure in a data processing package. A model may form a good hypothesis, then ask to run the test suite. The test suite may take 40 seconds. The command could be delayed because the evaluation machine is under load. The package manager may need to download a dependency. The container may hit a memory spike while compiling a library. The system may kill the process. The agent may spend its remaining time attempting recovery, or the harness may decide the tool call has timed out and terminate the whole run.
In the final results, this can look identical to a model that did not understand the bug. Both receive a zero.
For the user who simply wants a working patch, that outcome may be entirely fair. Reliability is part of product quality. But it is not fair to describe that result as a pure measurement of model reasoning. The model may have had the right plan and lost because the workshop lost power.
This distinction matters because companies are increasingly using coding benchmarks as procurement evidence. A score presented as a verdict on model intelligence can conceal a multitude of operational decisions. The buyer is then left comparing not just models, but two companies’ tool implementations, inference capacity, sandbox policies, and test orchestration.
That is not automatically a problem. It simply needs to be named.
Three kinds of advantage hiding inside one number
The most useful way to read an agentic coding score is to separate three forms of advantage.
Capability advantage is what most readers mean by model quality. It includes understanding the task, navigating unfamiliar code, generating a correct patch, recognizing when a test failure disproves a hypothesis, and choosing a better approach after a setback. Better reasoning, broader software knowledge, and more reliable tool use belong here.
Harness efficiency is the advantage created by the system that turns a model into an agent. It includes the initial instructions, context management, tool descriptions, planning steps, retry behavior, compression of long histories, checks before risky commands, and the policy that decides when the agent should continue or stop. A strong harness can preserve useful context, prevent repetitive mistakes, and convert a capable but inconsistent model into a more dependable worker.
Infrastructure advantage is what the runtime makes possible. It includes inference latency, concurrency limits, CPU, memory, disk performance, network bandwidth, package caching, container startup time, service reliability, and enforcement of time and resource limits.
All three affect the outcome. Only the first is close to a statement about a model in isolation.
Think of a restaurant. Capability is the skill of the chef. Harness efficiency is the kitchen’s workflow, order tickets, prep list, and quality control. Infrastructure advantage is the stove, ingredients, refrigeration, and how quickly supplies arrive. A diner cares about the meal, not which layer deserves the credit. But a restaurant owner who wants to improve performance must know where the bottleneck lies.
The same is true for a coding organization. If the model cannot understand a complicated codebase, adding RAM will not solve the problem. If the model understands the codebase but constantly loses context after several tool calls, a better harness may provide more value than a larger model. If successful plans die during dependency installation or test execution, the apparent model problem may actually be an infrastructure problem.
A single pass rate turns all of that into one opaque number.
Why the variance compounds
The issue gets more serious as tasks become longer.
A one-shot benchmark asks whether a model can generate a useful answer once. An agentic task asks it to make a series of decisions under uncertainty. Each step carries a chance of success, delay, error, or misinterpretation. Some failures are recoverable. Others redirect the entire trajectory.
Suppose an agent must complete eight consequential stages: inspect the repository, identify the relevant code, reproduce the bug, formulate a hypothesis, modify files, run tests, interpret the failure, and finalize a patch. It does not need perfection at every stage, because a good agent can recover. Yet the system still compounds risks.
A slow tool result consumes wall-clock budget. A retry consumes tokens. A partial test failure may create more context to read. A container restart erases a useful temporary artifact. A package installation failure pushes the agent into a less familiar path. A long transcript fills the context window and causes the harness to summarize away a detail that mattered.
The model is making decisions inside a feedback loop whose state is always changing.
This is why a slower endpoint can lose to a weaker model.
Imagine two agents that have comparable reasoning quality. Agent A uses a slightly stronger model but waits six seconds longer for each inference response and has a stricter 20 minute wall-clock limit. Agent B uses a somewhat weaker model but returns quickly, has faster command execution, and is allowed a well-designed retry when a tool call fails transiently.
If a task requires 25 model turns and 15 shell commands, seemingly modest latency differences accumulate. Agent A may face the final test with too little time left to inspect the error and revise. Agent B gets another complete cycle. The benchmark reports that B solved the task and A did not. In a real engineering setting where human developers can wait longer, the ranking could reverse.
This does not mean fast systems are cheating. Speed is a real product characteristic. A developer values an agent that completes work before a meeting ends. But it does mean that the resulting score should be labeled as an end-to-end product result, not a clean model ranking.
The same problem arises with token budgets. One system may permit a longer internal reasoning process or preserve more of the repository and terminal history. Another may use tighter limits to reduce cost. The first agent might solve more tasks because it has more room to investigate. That may be a legitimate business choice. It may also be a different test.
The key question is not whether a configuration affects performance. Of course it does. The question is whether the configuration makes the test more reliable without meaningfully changing the capability being measured.
When more resources repair the experiment, and when they make it easier
Anthropic’s Terminal-Bench experiments provide a useful dividing line.
The company tested resource configurations from strict enforcement of task specifications to uncapped execution. Under the strict configuration, containers were guaranteed a certain resource allocation and killed if they exceeded it. This creates a brittle situation. A brief memory spike can end a run even if the agent was otherwise on track.
As the researchers added headroom, infrastructure errors fell from 5.8 percent under strict enforcement to 2.1 percent at three times the specified resources, then to 0.5 percent when uncapped. The overall score did not move significantly from one times to three times the task specification. Beyond roughly three times, however, success rates began to climb more quickly than the reduction in infrastructure errors. From three times to uncapped, the score rose by nearly four percentage points, and the total improvement from strict limits to uncapped was six points.
That pattern is more informative than the headline number.
Up to a point, additional resources were mainly fixing the measurement apparatus. A transient memory spike should not decide whether a model understood a repository. If a task specification says a container needs a certain amount of memory, treating that number as a hard kill threshold may create failures that are artifacts of enforcement rather than meaningful tests of software work.
Past that point, extra resources began expanding what the agent could do. It could install a large library stack, start expensive subprocesses, or run a memory-intensive suite that would not fit in a constrained setting. Anthropic gave the example of a Bayesian network task where some agents tried to install common data science packages, while other agents pursued a leaner implementation using the standard library. The resource policy decides which style of engineering wins.
Neither style is inherently illegitimate.
A constrained benchmark can reward efficiency, careful dependency choices, and low-resource engineering. A generous benchmark can reward broad tool use, experimentation, and practical reliance on modern packages. These are both real capabilities. The error is presenting them as the same capability under a shared, insufficiently documented score.
This is the central test for benchmark designers: does a configuration change remove accidental noise, or does it alter the strategy space?
If it removes accidental noise, it improves the experiment. If it alters the strategy space, it may still be valuable, but it has changed the meaning of the result.
The harness is not a dirty word
There is a tendency in AI evaluation to treat the harness as an embarrassment. If a score improves after prompt design, better tool descriptions, task decomposition, or a separate evaluator agent, critics sometimes call it score inflation. That reaction misunderstands what an agent is.
A model is a general purpose prediction engine. An agent is a system that assigns that engine a role, grants it tools, gives it memory, and structures its work. A company deploying an agent will not deploy a bare model in a vacuum. It will deploy the full system.
Anthropic’s work on long-running application development illustrates the point. Its researchers describe a harness with planner, generator, and evaluator roles. The planner turns a brief request into a product specification. The generator works through features in chunks. The evaluator tests the resulting application, reports faults, and enforces thresholds before the work progresses. The system also uses structured handoffs to manage long tasks.
That is not merely cosmetic prompting. It changes the working process.
A single agent trying to build a complete application can become overconfident, forget why it made an earlier choice, or declare success after testing only the happy path. A separate evaluator creates productive friction. A planning step reduces the risk that implementation begins with an underspecified goal. A disciplined handoff makes a long task less dependent on the fragile contents of one context window.
In a production setting, those are engineering improvements. Users should want them.
The right response is not to ban harness innovation. It is to report it honestly.
A leaderboard should distinguish at least two columns:
Standardized model comparison, where every competitor receives the same task set, tool interface, runtime policy, timeout, token budget, and resource configuration.
End-to-end agent product comparison, where each provider may use its best documented harness and infrastructure within common safety and cost boundaries.
The first tells buyers more about the underlying model. The second tells buyers more about the product they may actually use. Both are worth having. Combining them without labels produces confusion.
The difference between legitimate tuning and score-shopping
Every evaluation involves choices. A benchmark cannot avoid them. The practical question is how to make those choices credible.
A benchmark change is legitimate when it improves reliability, aligns the test with a clearly stated deployment setting, applies symmetrically, and is disclosed before results are compared.
For example, raising a memory ceiling may be legitimate if the prior ceiling caused random container deaths unrelated to the agent’s work. The evidence should show that infrastructure failures fall while task success stays within expected statistical noise. That was the logic behind Anthropic’s recommendation to calibrate a range between a resource floor and ceiling, rather than treating the minimum requirement as an automatic kill limit.
By contrast, a change begins to resemble score-shopping when it is selected after inspecting outcomes, benefits one provider’s known strategy, or quietly changes the task from “solve this problem efficiently” to “solve this problem with nearly unlimited compute.”
Other warning signs are simpler:
- The harness changes between model runs, but the comparison presents a single ranking.
- A provider uses private caches, privileged network access, or dedicated inference capacity that outside evaluators cannot reproduce.
- Timeouts are revised only after a favored model narrowly fails.
- Failed infrastructure runs are silently excluded rather than reported.
- The benchmark relies on one attempt for a stochastic system.
- The grader is altered after observing model behavior, without a preregistered correction process.
- The reported result omits cost, time, number of attempts, and the rate of operational errors.
Transparency does not solve every methodological disagreement. It does make the disagreement visible.
A useful analogy comes from sports. Better shoes, coaching, nutrition, and track surfaces all help athletes run faster. It would be absurd to claim that only leg strength matters. But it would also be misleading to compare two race times when one athlete ran downhill with a tailwind and the other ran indoors, then call the difference a measure of innate speed.
Agentic coding is entering its version of that moment. The field needs rules that tell readers which race they are looking at.
How to design an evaluation someone else can reproduce
A reproducible agent evaluation begins with a discipline that is less glamorous than a leaderboard launch: write down the contract before running the models.
Start by defining the target of measurement. Is the goal to estimate model capability under a common scaffold? Is it to compare commercially available coding products? Is it to test whether an internal agent can resolve the kinds of tickets your organization actually receives? One suite can serve several purposes, but the reporting should not pretend they are identical.
Next, freeze the task environment.
Record the repository revision, operating system image, package versions, network policy, CPU allocation, memory floor and ceiling, disk quota, hardware type, concurrency level, and timeout rules. Record whether dependencies are cold-installed or available from a cache. Record whether the model is served through a shared endpoint or reserved capacity.
Then freeze the agent configuration.
Publish the system prompt, tool definitions, tool permissions, token and reasoning budgets, maximum turns, retry policy, summarization method, stop conditions, and any multi-agent roles. If an agent has a planner, a coding worker, and a verifier, say so. If it receives feedback from tests or a secondary model, say so.
Run multiple trials. Models vary across runs, and so do infrastructures. A pass rate from one attempt is often a story about luck. At a minimum, report the number of trials per task, the randomization policy, and uncertainty intervals. For public comparisons, repeat evaluation across different times or days when shared services could experience variable load.
Finally, preserve the traces. A final patch is not enough. The transcript should show tool calls, command outputs, retry events, timeout causes, container restarts, tokens, elapsed time, and final grader outputs. It is the equivalent of a flight recorder. When a result looks surprising, the trace distinguishes a brilliant recovery from a broken sandbox.
The scorecard should be richer than a single pass rate. A useful version includes:
- Task success rate, with confidence intervals and number of trials.
- Median and tail completion time, not just average time.
- Cost per successful task.
- Token use per task.
- Tool calls and retries per task.
- Infrastructure failure rate, separated from task failure rate.
- Timeout rate, with the timeout type specified.
- Resource configuration, including CPU, memory, and disk rules.
- Harness version and configuration hash.
- Grader version and repository revision.
- A failure taxonomy: incorrect patch, incomplete work, model abandonment, tool failure, sandbox failure, grader issue.
This makes the score less tidy. It makes it more useful.
An executive may still prefer a simple recommendation. That is fine. The evaluation team can ultimately say, “Product A achieved the best completion rate at our latency and cost target.” But the team should be able to answer the next question: “Did it win because it reasoned better, because its harness used the environment more effectively, or because its environment was more forgiving?”
If the answer is “all three,” that may be a perfectly good reason to buy it. It is not a reason to call the result a pure model benchmark.
What buyers should ask before trusting a close leaderboard
The most important practical consequence is a modest one: be skeptical of small differences.
Anthropic argues that public leaderboard differences below three percentage points deserve skepticism until the evaluation configurations are documented and matched. In its analysis, ordinary sampling uncertainty could already span roughly one to two points, before infrastructure confounders were added.
That does not mean a two point difference is meaningless. It means it is a hypothesis, not a verdict.
A buyer evaluating coding agents should ask five questions.
First, what is fixed across participants? If model A and model B run under different scaffolds or cloud environments, the comparison is product versus product, not model versus model.
Second, what does a failure mean? A failed test after a carefully attempted patch says something different from a container that never started.
Third, what resource policy is being tested? Tight constraints may reward efficient engineering. Generous constraints may reward broader exploration. Match the choice to the deployment reality.
Fourth, what is the operational budget? A model that solves slightly more tasks but takes three times as long and costs five times as much may be the right choice for critical migrations and the wrong choice for everyday bug fixes.
Fifth, can the result be replayed? If the provider cannot describe the harness, task image, trials, and grader, the score should be treated as marketing evidence, not scientific evidence.
The larger point is not cynical. It is constructive.
Coding agents are becoming systems that developers rely on for real work. The most useful evaluation will therefore be one that mirrors the actual conditions of work, while remaining honest about what it measures. The model’s intelligence is part of that story. The harness that keeps it organized is part of it too. So is the infrastructure that gives it time, tools, and a stable place to operate.
A leaderboard can still help us see progress. But it should be read less like an IQ test and more like a systems performance report.
The winning number is not merely an answer to the question, “Which model is smartest?”
It is an answer to a more practical question: “Which complete machine was most able to finish the job under these exact conditions?”