Terminal-Bench 3.0 launched with 74 tasks across seven domains and immediately did what a new benchmark is supposed to do: it broke the leaderboard. GPT-5.6 Sol running in Codex tops it at 34.4 percent, with Fable 5 in Claude Code at 33.8 percent. The same Fable 5 configuration scores 83.8 percent on the previous version, Terminal-Bench 2.1. The benchmark, formerly announced as Frontier-Bench, is built by the team behind earlier Terminal-Bench releases and the Harbor evaluation framework.
Key facts
- 74 tasks across 7 domains in the first release. Best score is about 34 percent.
- Leaderboard as published: GPT-5.6 Sol (Codex) 34.4%, Fable 5 (Claude Code) 33.8%, Opus 4.8 (Claude Code) 21.1%, GPT-5.6 Terra (Codex) 20.8%, Grok 4.5 (Cursor CLI) 17.8%, Sonnet 5 (Claude Code) 14.6%, GPT-5.6 Luna (Codex) 14.3%, GLM 5.2 (Claude Code) 5.1%.
- Fable 5 and Opus 4.8 are separated by 4.9 points on Terminal-Bench 2.1 and 12.7 points on 3.0.
- Primary source: Terminal-Bench 3.0 announcement, with results also tracked at tbench.ai.
The team's stated reason for rebuilding is bluntly diagnostic. "Many Terminal-Bench tasks have become saturated and leaderboard entries are condensed into a narrow range that may not fully represent capability gaps across models," the announcement says. That is the failure mode every benchmark eventually reaches: once the top six models sit within five points of each other, the number stops carrying information, and the differences you can still see are mostly noise.
The fix was to widen the definition of work rather than simply lengthen it. Terminal-Bench began from the premise that the command line is a good proxy for what an agent can do. Version 3.0 keeps the terminal and stretches what sits behind it into multi-container networks, GPUs, and long-running jobs. The list of artifact types the grader accepts is the clearest description of the scope: database snapshots and Kafka archives, machine learning checkpoints, register-transfer-level hardware designs, formal proofs in Lean and Coq, virtual machine images, compiled binaries, engineering CAD files, music scores in MusicXML, reverse-engineered specifications, and business-process decisions. That is not a coding benchmark with extra steps. It is an attempt to sample the range of jobs an agent might plausibly be given.
The anti-cheating design deserves attention because it addresses a real and growing problem. The agent works in one container; at the end of a trial its artifacts are downloaded, logged, and uploaded into a separate verifier container that does the grading. That severs the path an agent takes when it decides the easiest way to pass a test is to edit the test -- a behavior general enough to have its own name, reward hacking, and one that has quietly inflated scores on older agent evaluations. It also means a broken verifier can be fixed and every past trial re-graded, which is why the project describes itself as continuously versioned with CI/CD and semantic versioning rather than as a fixed snapshot. Each task passes through proposal, static checks, an LLM judge, agent trials, an agent judge, and both reviewer and senior reviewer sign-off, with frontier, oracle, and deliberately cheating agent runs used to check that a task is well specified.
The result the team highlights is discrimination rather than difficulty. Fable 5 and Opus 4.8 differ by 4.9 points on 2.1 and 12.7 points on 3.0. Spreading the field out is the entire point; a benchmark's job is to separate, and a compressed leaderboard cannot.
What is more interesting than the ranking is the cost data underneath it. GPT-5.6 Sol and Fable 5 land within a point of each other on pass rate, but Sol is roughly 40 percent cheaper and uses about 50 percent fewer tokens. The behavioral traces differ too: Fable 5 and Opus 4.8 spend more tokens across fewer actions, while the GPT-5.6 models take more actions with fewer tokens -- one style thinks longer before acting, the other acts more and thinks in smaller increments. All three GPT-5.6 models sit on the cost-versus-pass-rate frontier for the runs published.
The caveat is the one that follows every agent benchmark and is impossible to design away. Each entry names a model and a product: Codex, Claude Code, Cursor CLI. The number measures the pair. A model's score reflects how well its vendor's harness manages context, retries, and tools as much as it reflects the weights, and we have watched harness changes alone move a model by twenty tasks. GLM 5.2 scoring 5.1 percent inside Claude Code is a fact about that combination, not a clean verdict on the model. The benchmark is also an open-internet evaluation, with agents merely instructed not to search for task-specific solutions -- an instruction the team reports is "surprisingly effective," which is a candid way of saying it is not enforced. And with the best agent below 35 percent, the interesting question for the next six months is not who leads but whether the remaining two-thirds falls quickly, as it did on the refactoring benchmark that stopped the best agent at 41 percent, or holds.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)