> ## Content Index
> Fetch the complete content index at: https://claudefox.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# FrontierHarness Eval v1.1: with open models, Claude Code was among the costliest harnesses per solved task
- URL: https://claudefox.com/frontierharness-eval-v1-1-best-harness-per-model/
- Published: 2026-11-17T21:26:54.000Z
- Updated: 2026-11-17T21:30:13.000Z
- Description: Runta and Baseten ran 22 configurations of 18 coding-agent harnesses with five open models over 62 terminal tasks. The best-value harness changed with the model, and Claude Code was the second most expensive per solved task. Closed models were not tested.
- Author: hupicake
- Tags: Research

Runta, which sells an agent execution layer, published FrontierHarness Eval v1.1 on November 17 with Baseten: 18 harnesses in 22 configurations (four DeepSeek Harness profiles and two OpenCode versions count separately), from Codex CLI and Claude Code to OpenCode, Pi and GitHub Copilot CLI. Each ran with five open models (GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open and Kimi-K3) on 62 terminal tasks, four attempts each: 27,280 runs and 83 billion tokens, at maximum reasoning effort and default harness settings. All dollar figures are Baseten list prices.

The headline: the best-value harness, which Runta defines as the lowest cost per pass unless another costs at most 30% more and passes at least 5 points more often, depends on the model. The per-model picks were Codex CLI (GLM-5.3), Pi (GLM Flash), ZCode (DeepSeek), Oh My Pi (Muse) and Alma (Kimi-K3). Only Codex CLI with GLM-5.3 was a clear win, first in 99.6% of resampled draws; the other four were 25% to 53% likely, so treat them as leaders among close harnesses. Codex CLI ranked 10th per pass with Kimi-K3 and 16th per task.

Across all five models Codex CLI cost least per pass at $0.937 and Antigravity CLI most at $1.63\. Claude Code was second-most expensive at $1.55 per pass; with Muse it cost $3.75 per pass for a 52.8% pass rate, against Oh My Pi's $1.21 and 58.9%. On cost per task, OpenCode v2 was cheapest ($0.561) and ZCode had the highest pass rate (61.9%).

Two findings are useful when tuning any agent setup. The model explained 92.8% of the variance in cost, so changing the harness matters less than changing the model, though with the model fixed the most expensive harness still cost 1.9 to 4.9 times the cheapest per pass. And the number of steps tracked cost (rank correlation +0.51), while prompt size, tool count and cache hit rate did not, because every step resends the whole context. Costlier harnesses didn't pass more often (rank correlation between cost and pass rate ranged from −0.39 to +0.22 by model), and 64.5% of all spending went to trials that did not pass: 52.7% to failures and 11.8% to timeouts.

Limits, per Runta: only open models, served and priced by one provider, so closed models such as Claude Opus 5.5 or GPT-6 Astra may rank harnesses differently; Runta says those come next. Each pass rate has a 95% interval of about ±10 points, so small gaps are ties. Runta's own advice: when you change models, test the candidate harnesses with your model on a private set of your own tasks, large enough to see past the noise.

## Related on claudefox

- [Product Engineering: 45 free agent skills for shipping a product with Claude Code and Codex](https://claudefox.com/product-engineering-45-agent-skills-claude-code-codex/)
- [Researcher says released weights show six Chinese models were started on Nvidia Blackwell GPUs](https://claudefox.com/checkpoint-forensics-chinese-models-blackwell-fingerprint/)
- [Agent Autoloop: run Claude Code and Codex in the background and get a card when they need you](https://claudefox.com/agent-autoloop/)
- [huashu-motion lets Claude Code or Codex make a product launch film with music that you edit by clicking a frame](https://claudefox.com/huashu-motion-agent-launch-film-skill/)

Source: [Runta: Introducing FrontierHarness Eval v1.1](https://runta.com/blog/introducing-frontierharness-eval-v1-1/?ref=claudefox.com), [FrontierHarness dashboard](https://frontierharness.org/?ref=claudefox.com)

This story first appeared in the [claudefox daily briefing for November 18, 2026](https://claudefox.com/daily-2026-11-18/).