Microsoft measured our thesis, and we still cannot quote ours

Microsoft's .NET team published the benchmark behind their polyglot unit testing agent. We have been arguing its conclusion for months. We can't cite our own version of it, because we withdrew our number on 2026-08-06 and haven't earned it back.

Both halves of that are worth writing down.

What they measured

The setup is clean. One tool, one model, 152 tasks from real repositories. The only thing that varies is a plugin that makes the agent research the repository, plan, implement, then verify before it claims to be done. A task passed only if the repository built, all tests passed, the agent added at least one test, and it removed none.

63% fewer failures, from the same model and the same tool.

The breakdown is where it gets useful.

On detailed prompts the two arms tie exactly. The whole gap comes from vague prompts, where the agent has to work out for itself what to test, which framework is in use, and where the tests belong. So the scaffold isn't making the model smarter. It's taking away the guessing.

Then the per-model results, on 45 .NET tasks:

Their own summary of that table is one sentence: "The workflow helped every model. With Opus, it added eight wins with no losses."

There's a reading they don't offer, and I want to be clear that it's mine rather than theirs. GPT-5.5 inside the workflow finished 91.1%. Opus 4.8 outside it finished 77.8%, on the same 45 tasks. Read across the rows instead of down them and the cheaper model in a system came out ahead of the stronger model on its own. Microsoft compares each model only against itself and never makes that claim. I'm making it, from their published table, so weigh it as my inference and not their finding.

Two caveats we'd want applied to us

Their coverage barely moved: 72.4% against 72.2% line, 49.8% against 49.1% branch. The agent also wrote slightly fewer tests than stock, 6,963 against 7,129. So the workflow didn't produce better tests. It produced tests that exist, build and pass, more often. They say so themselves, and it's a completion result rather than a quality one.

The gains also concentrate where the request is underspecified. On detailed prompts the workflow buys nothing at all. That's a real boundary on the claim, and it belongs in the headline rather than a footnote.

We'd hold our own numbers to exactly that. So we will.

Why we can't quote ours

We have a version of this result. On our own harness, switching the verification layer off dropped correctness from 100% to 75%, and switching the guards off as well took it to 50%. Small n, our harness, not a public benchmark, and we say that every time we cite it.

The bigger claim was a lift figure: a cheap model alone, then the same model inside our cascade. We published it. Then we audited our own truth layer and found something ugly. Our two baselines for the same quantity, the same cheap model alone on the same benchmark, disagreed by 3.1 points. One page computed the lift from one baseline. Another page computed it from the other. Each traced back to a real measurement, which is exactly why nobody caught it.

Both lift figures came off every public surface that day, and they stay off until one baseline is re-run.

So here's the honest position. Microsoft has a clean, large, well-controlled measurement of something we believe. We have an unresolved 3.1 point disagreement with ourselves about ours. Their result doesn't repair ours. Someone else agreeing with you isn't a re-measurement of your own work, and being right isn't the same as having measured it.

The part we did measure, and it points the same way

One result of ours survives a second independent run and an adversarial pass, so it can sit next to theirs.

We ran three models over the same 160 HumanEval+ problems, with the same prompts and the same scorer. They finished 92.5%, 91.2% and 90.0%. A 2.5 point spread that reads as interchangeable.

Partition by task instead of by column total and it looks different. 83% of the set is uncontested, where all three models get it right, and the models disagree on roughly 15 to 18 percent of the work across two runs. What makes that citable isn't how big the disagreement is, it's how stable: the same problems come up contested each time. 17 tasks are contested in both runs against 3.6 expected by chance, and ten of them are contested in both runs without any model ever changing its answer.

Which problems they split on is a property of the models. Which model scores highest is a property of the run. That's HumanEval+ Python on our harness, not customer traffic.

Put it beside Microsoft's table and you get one argument instead of two. They measured that the workflow around the model carries most of the gain you can reach. We measured that the ranking you'd use to pick a model is mostly noise. Both point at the same place, and it isn't the model.

What we took from it

One technique, straight away.

Before their agent declares a task complete it runs a set of checks, and one of them reads: "It considers small code changes that should make the tests fail. This is a lightweight form of mutation testing." No full mutation framework. Just a cheap step that catches an assertion which can't fail.

We knew that rule. We wrote it down after a scorer of ours passed its positive control perfectly while being structurally incapable of returning a negative. We wrote it down again when a guard printed OK at every boot and then fired on zero of seven deliberately injected defects. We were applying it from memory, and memory isn't a source. Microsoft made it a step in the loop.

So we built it. Every guard in our tree can now declare, in its own header, an injected defect that must make it exit non-zero. A runner executes each one in a fresh directory and reports whether the guard can still say no. Three grades: PROVEN, BROKEN, and UNPROVEN for guards that haven't declared a control yet. UNPROVEN counts as a warning rather than a failure, because a gate that's red on day one gets overridden by reflex, and a reflex override has stopped being a signal.

We watched it fail before trusting it. A stub guard that always exits zero grades BROKEN.

Then we pointed it at the guard from the story above, the one printing OK at every boot. We injected a wrong percentage where a real measured one belongs. Exit 0. We injected a cost multiple we'd already struck from our own public copy. Exit 0. A third run, with the correct value in place, confirmed the guard really was reading two live surfaces against the real ledger, so a trivial pass couldn't be mistaken for a catch. It graded BROKEN by its own declaration, and it's since been fixed.

As I write this the runner reports 7 proven, 0 broken, 33 unproven, out of 40 guards. That last number moved while I was drafting, because writing this article involved building another checker and the gate made me declare a control for it before it would let me commit. The unproven count is the honest one, and it's the one we expect to be asked about. It's also the point of the exercise. Before we built this, the answer to "which of your guards can still fail?" was "we assume all of them."

A postscript, because it happened while writing this

Fact-checking this article, I flagged four of its own numbers as fabrications. The branch coverage figures. The Haiku row. The three-model percentages. Our own ablation result.

All four were real. I'd checked them against a news summary of Microsoft's post rather than the post, and against the wrong one of our own benchmark artifacts. The summary leaves out branch coverage entirely and never mentions Haiku, so an absence looked like an invention.

We have a rule for this and I broke it anyway: never relay one party's account of another's work without reading the original. I was one hop further out than I thought, which is the only place that error lives.

Two defects did survive the pass, and both were in the draft's account of our own work. A comparison of mine that read as Microsoft's, and a stale guard count.

And I have to finish that sentence honestly, because the first version of this article went out before I caught either one. It was live for about three hours, saying that a guard "is now marked BROKEN" when we had fixed it that morning, and reporting 2 proven of 37 when the runner said 7 of 40. This text replaced it in place, which is why the two paragraphs above are written the way they are. An article about withdrawing a number published a wrong one first. That is the whole argument for having a correction path that edits at the claim instead of appending a note at the bottom, and it is why every figure here now carries a line in a verification block saying where it was re-derived from.

Sources

• Microsoft .NET blog, the polyglot unit testing agent: https://devblogs.microsoft.com/dotnet/polyglot-unit-testing-agent/
• The plugin, MIT licensed: https://github.com/dotnet/skills