Coding Harnesses Can Triple Costs Without Making AI Agents Better

That matters because developers rarely buy a model in isolation. They encounter complete products such as coding assistants, command-line agents and editor extensions. Each product combines a model with instructions, tools, file handling, retry rules and decisions about when to stop. The result can make it difficult to know whether better performance comes from the model itself or from the system directing it.

Yangze Liu and Zhongyi Han’s paper, “What Does a Harness Buy? Tokens, Mostly,” examines that question directly. The researchers held five coding models constant and tested them across three agent harnesses: Claude Code, mini-SWE-agent and OpenCode. They evaluated the combinations on SWE-bench Verified, a benchmark built around real software engineering tasks.

The central finding is uncomfortable for anyone assuming that a preferred harness provides an automatic capability advantage. Harness changes generally shifted accuracy by about as much as rerunning the same setup, according to the paper. At the same time, the cost of completing tasks could change by as much as three times.

In practical terms, an organization might spend substantially more on one orchestration layer without receiving a reliable improvement in the number of tasks solved. The extra spending can instead go toward longer conversations with the model, repeated attempts, additional tool calls and the tokens consumed while the agent searches for a solution.

The hidden product around the model

A coding model does not normally receive a programming task and return a finished patch in one step. An agent harness decides how the model interacts with a repository. It can give the model access to a terminal, ask it to inspect files, run tests, revise its work and try again after an error.

Those decisions resemble the operating procedures around a skilled employee. Two employees with similar abilities may produce different results if one has better access to information or a more disciplined process. But the study suggests that process improvements are not automatically valuable. Some may simply encourage the model to do more work without making the final answer more likely to succeed.

That distinction is important for buyers. Token usage is not merely a technical statistic. It is a direct operating expense for many AI systems, especially when models are paid for by the amount of text they process. A harness that triggers more exploration or failed retries can increase the bill even if its success rate remains broadly unchanged.

The paper’s comparison also challenges the way companies and users discuss coding-agent performance. A statement that one product is “better” can conceal several different variables. The product may use a different model, a more forceful system prompt, a larger context window, a distinct tool strategy or a more aggressive retry policy. Comparing products as single objects can therefore confuse model quality with workflow design.

Accuracy is only part of the bill

The benchmark result does not make cost the only relevant measure. A harness that uses more tokens could still be valuable if it produces patches that require less human correction, handles difficult repositories more reliably or reduces the time engineers spend supervising it.

But those benefits would need to be measured rather than assumed. A system that solves a task after several failed attempts may look successful in a final benchmark score while creating additional review work in a production team. Engineers may have to inspect a larger trail of changes, understand why tests failed or determine whether the agent’s final patch fixed the underlying problem.

This is where benchmark scores can become incomplete decision tools. A benchmark records whether a task was solved under specified conditions. It may not capture the full cost of reaching that result, the amount of human oversight required or the risks created by unnecessary edits.

The study therefore points toward a broader accounting system for coding agents. Companies may need to track cost per successful task, average tokens per attempt, retry frequency, time to completion and the number of patches that require substantial human revision. Those figures could reveal that a slightly less accurate system is preferable if it is cheaper, more predictable and easier for engineers to supervise.

A result with limits

The findings should not be treated as proof that all harnesses are interchangeable. The experiments covered five models, three harnesses and one benchmark. A different task mix could reward a harness with stronger planning, more effective tool use or better recovery from errors.

Weights & Wire argued that the result is useful but may not be universal, noting that earlier studies may have observed genuine harness effects under different tasks, benchmarks or evaluation setups. No opposing response to the study was identified in the supplied material, so that limitation is the main caution available alongside the paper’s conclusion.

The researchers also released their code, configurations, data and analysis materials in the project repository maintained by Liu and Han. That makes the comparison more useful than a product review based only on a single set of reported scores. Other researchers can test whether the pattern holds as models, tools and benchmarks change.

For the industry, the lesson is less dramatic than the promise of a breakthrough, but more practical. Better orchestration can matter. It can also become an expensive layer of activity that mostly produces more tokens. Until companies measure both outcomes and effort, a coding agent that appears more capable may simply be working harder, retrying more often and charging the difference to its users.