# Benchmark finds coding-agent harness costs shift sharply between models
A large coding-agent benchmark has found that the software harness surrounding an AI model can materially change both cost and task success, with no single harness delivering the best value across every model tested.
Runta published FrontierHarness Eval v1.1 on November 17 after evaluating 22 harness configurations with five models on 62 terminal-based tasks. Each combination was attempted four times, producing 27,280 runs and consuming 83 billion tokens, according to the company. Infrastructure provider Baseten served the models used in the experiment.
The tested models were GLM-5.3, GLM-5.3 Flash, DeepSeek-V4.1-Flash-1017, Muse Spark and Kimi-K3. Runta kept each harness at the same version and settings across the models and used its maximum reasoning-effort option. Tasks had time limits ranging from 15 minutes to eight hours, and every trial ran in a fresh environment that was deleted afterward.
Results showed that the preferred harness varied by model. Runta selected Codex CLI for GLM-5.3, Pi for GLM-5.3 Flash, ZCode for DeepSeek, Oh My Pi for Muse and Alma for Kimi-K3 under its value rule. That rule generally minimized cost per successful task, while allowing a more accurate harness to win when it cost no more than 30 percent extra per pass and improved pass rate by at least five percentage points.
Across all models, OpenCode v2 posted the lowest average cost per task at $0.561, while ZCode reached the highest reported pass rate at 61.9 percent. Pi and Codex CLI also appeared on the benchmark’s efficiency frontier, meaning no other measured harness beat them simultaneously on cost and pass rate.
The ranking changes were substantial. Codex CLI ranked first for task cost with GLM-5.3 but fell to 16th with Kimi-K3. The median harness shifted 11 places in the 22-entry cost ranking when models changed. With a model held constant, the most expensive harness cost between 1.8 and five times as much per task as the cheapest.
Runta tested whether ordinary variation between repeated runs could explain those gaps. It shuffled trials among harnesses for each model 300 times and found the observed high-to-low cost ratios exceeded every shuffled result, reporting a probability value of 0.003. Split-sample rankings also agreed more strongly when the model stayed fixed than when different models were compared.
The evaluation applied a zero score to 216 trials flagged for reward hacking, including 163 involving online lookups. Runta said the controls were intended to prevent agents from gaining credit through behavior outside the task rules.
The findings suggest that organizations evaluating coding agents should test the complete model-and-harness pairing rather than treating the harness as a neutral interface. A tool that is economical with one model can become relatively expensive with another, while higher spending does not reliably produce a better pass rate. Runta’s benchmark is a vendor-produced evaluation and its conclusions remain specific to the selected tasks, configurations, prices and models, but the scale of the experiment provides a detailed comparison of interaction effects that smaller leaderboards may miss.

![500px provided description: Anohter Laptop With Cool Free Software Stickers At Guadec [#stickers ,#2015 ,#Tech ,#GNOME ,#FOSS ,#Open Source ,#Free Software ,#GUADEC ,#Confrance]](/media/articles/commons-e039f8ff3448.avif)

