We stopped choosing models and started choosing what to measure

The default way to use an LLM is to pick one and send it everything. That is what the integration looks like, it is what a model marketplace is shaped around, and it is what most agents I read about do. Pick the good one. Send it the work.

We run a routing and verification gateway, so we have had to measure a lot of models against the same tasks with money attached. I want to put down what our own data keeps saying, including the parts where it makes us look bad, because the conclusion is not "we found the better model."

We ran the experiment everyone runs to pick a model. We ran it twice. We got a different winner.

Three models, 164 HumanEval+ problems, temperature 0, one completion each. Then the identical thing again an hour later. Nothing changed between the two runs except time.

Each model is scored over the problems it actually answered. Two of them answered all 164 in both runs. qwen3.8-max timed out on 4 and then on 9, so its denominator is 160 and then 155. The partition table further down uses the smaller set where all three answered, so its percentages are not directly comparable to these.

First and last swapped. The model we would have picked on Tuesday is the one we would have rejected on Wednesday, and the new frontier model that lost to our cheap tier by 1.3 points beat it by 1.4 the second time.

Temperature 0 is not determinism, and 164 problems is not enough to separate models sitting two points apart. We can put a number on how much of this is flicker: run the same model over the same problem twice and it changes its answer 3.7 to 5.2 percent of the time. Any ranking gap smaller than that is not a result. Ours were all smaller than that.

But something did survive both runs, and it is the more useful thing.

Stop asking how many each model got and ask which ones.

Roughly 15 to 18 percent of the time, the model you picked decided whether you got a right answer.
Note what the same table says about the other 82 percent: all three models agree, 130 of them right
and 2 of them wrong together, and the choice is irrelevant. This is a claim about a fifth of the work, not about all of it.

The reason to believe the fifth is real rather than flicker is not the size of the number, it is that it lands on the same problems both times. Twenty four problems were contested in the first run and twenty three in the second, and seventeen of them are the same problems. If the disagreement were noise you would expect about four to coincide. Ten of them are contested in both runs without a single model ever changing its own answer.

The ceiling is the part I would think hardest about. The best single model scores about 92 percent. If you could always take whichever of the three happened to be right, you would score 98.8 percent, and that number was stable across both runs too. Picking one model, even correctly, leaves about six points on the table.

A leaderboard cannot show you any of this. A leaderboard is column totals, and all of it lives in the rows.

Capability is shaped like the task, not like a ranking

Our verification witness is llama-3.3-70b, pinned to one provider. On 400 standard tool-calling tasks it scores 382, which is 95.5 percent, and matches our primary path. On an adversarial set built to tempt it into calling a tool when it should decline, it scores 30 of 40.

I have to be careful with that 40, and I only learned why today. It is four distinct trap shapes, each parameterised ten ways. The replicates swap a city or a number; they do not add a fifth kind of trap. So the honest denominator is four, not forty, and I am not going to put a confidence interval on it. What the instrument legitimately supports is: across four adversarial tool-selection shapes, parameterised ten ways each, it failed ten times.

The interesting part is not the 75 percent, which I now think is close to meaningless. It is that all ten misses are the same verdict. Every one is a false call: it invoked a tool where the correct behaviour was to abstain. So the model matches the primary at doing the thing and fails it at knowing when not to. Those are different abilities, they are not correlated the way a single score implies, and only one of them was on the benchmark.

If you are building on a model because it scored well at a task, check whether the thing you actually need is the task or the restraint.

The price on the page is not the price

We screened a cheaper candidate for our cheap tier. Advertised at 0.05 and 0.20 per million against our shipped tier at 0.07 and 0.27, roughly 30 percent cheaper on both sides.

On a real tool call it emitted 132 completion tokens, of which 123 were reasoning tokens that we paid for and never saw. Its effective output price came out around fifteen times its sticker, making it roughly ten times more expensive than the tier it appeared to undercut. The tool call itself was correct. The capability was real. Only the economics were inverted.

This is one call, so treat it as a mechanism rather than a rate. The mechanism is enough: compare cost per successful result, never cost per million tokens, and read the reasoning token count before you believe any price comparison.

You may not be choosing a model at all

Same model, same 400 prompts, same scorer. Pinned to a single serving provider it scored 95.5 percent. Unpinned, letting the marketplace route it, 78.2 percent. The model was held constant and the score moved 17.3 points.

For scale, adjacent models in our own sweep differ by 0.5 to 6 points, so the routing spread was larger than the entire model spread we were trying to measure.

And then we checked whether that was a law, because it would be a convenient one for us. We ran the
same pinned-versus-unpinned experiment on gpt-oss-120b. It moved 0.6 points, 65.2 percent pinned
against 65.8 unpinned. So this is not "marketplaces degrade models." It is a property of the specific
model and provider pair, and the actionable version is that you have to check yours rather than assume
either way.

The newest one is not automatically the answer either

Qwen 3.8 came out yesterday. We benchmarked it the same night against the two models it might replace.

And this is where I have to be careful, because the first version of this section said "it did not beat our cheap tier" and the second run says it did. On accuracy it is indistinguishable, in both directions, and I am not going to pretend otherwise in either.

What did not move between runs is the part that actually decides it. It cost 64 to 76 times more per solved problem, depending on the run, priced at the marketplace list rate rather than at what the vendor's own API would have charged us, which we did not read. It was slower at the median in both. And its timeouts more than doubled, 4 then 9 out of 164, with a worst case over 200 seconds. The cost multiple moves because it is a ratio of two noisy things, so treat it as a band and not a constant, which is the same rule this piece applies to everything else.

So the reason not to adopt it is not that it is worse at the task. It is that it is indistinguishable at the task, dramatically more expensive, and less reliable, and those three are all stable across runs while the score is not. The score was the only thing I originally judged it on, and the score was the only thing that turned out to be noise.

So what do you do instead

If models differ by task rather than rank, then "which model is best" is optimising the wrong thing. The unit we route on is the task shape plus a check: send the request to whatever suits its shape, serve the cheap answer when something independent agrees it is right, and escalate only when the check disagrees. The model becomes a swappable component, and being wrong about one becomes survivable.

And the uncomfortable part, because leaving it out would make this an advertisement. Our own escalation target never cleared the cheap tier it exists to rescue on code, in either run. On the first run it scored below it, 90.0 against 92.5. On the second the two tied at 92.9. By this piece's own rule I cannot report either of those as a ranking, so the honest version is the weaker and worse one: we have no evidence that the model we escalate to is better at code than the model it is rescuing. That is not what an escalation target is for. A gate can only be as good as the thing it routes to, so the thing we are fixing is not the gate, it is the ladder, and last night's benchmark was us checking whether the new frontier model was the fix. It was not.

That is scoped to code, and I should say so rather than let it sound general. This benchmark did not test the tool path at all, and our own data has the ordering of these tiers inverting by task shape. Which is the whole argument again, made against us this time: the rung that looks wrong on one task shape is not automatically the wrong rung.

Caveats, in full

The 98.8 percent ceiling assumes an oracle that always picks the model that happened to be right. No router has that, ours included, and we have not measured what share of those six points our own routing actually captures. Read it as the size of the prize, not as a result.

Single run per arm on the 160 problem comparison, temperature 0, one completion each. Temperature 0 was not deterministic for us: two runs of the same 8 problems gave different results, so treat small gaps as noise, including ours. The 95.5 versus 78.2 pair is 400 per arm, single run, no confidence intervals. The trap set is four shapes replicated ten times, not forty independent cases, so treat it as four observations and ignore any interval anyone computes on n=40, including one of ours that is now struck. The pricing finding is one call. Our benchmarks are code and tool calling and they are not your workload.

None of this says frontier models are overrated. We escalate to one and would not ship without it. It says that after measuring a lot of them against the same tasks, the most useful thing we learned was not which one wins. It was that they fail in different places, and the difference between them is bigger than the gap in their scores.

---