Originally published on hexisteme notes.
I run a small fleet of coding agents on one machine. Every thread ends up in a log, and a measurement pipeline reads those logs into a database, attributing each turn to the model that produced it. After a few thousand threads I had the table people keep asking for: seven model-versions, five behavioural metrics, real workload rather than a benchmark.
Then I printed one cross-tab I had been skipping, and most of that table stopped meaning what I thought it meant.
The cross-tab was role × model. In this fleet a model runs in one of two roles. It is either the long interactive main thread I drive by hand, or a short one-shot sub-agent that a main thread spawns, runs once, and discards. Same model. Same weights. Two completely different jobs.
Role turned out to move the numbers by up to 135x, and the role mix is wildly different for each model. Those two facts together are enough to make a pooled comparison manufacture a large gap that exists in neither stratum.
A note on labels: model names are replaced with letters on purpose. The point of this note is that these numbers are not a model ranking, and printing the names invites exactly that misreading. Every figure is real, from one snapshot of one fleet.
The role gap is an order-of-magnitude thing
Median output tokens per thread, same model, split by role:
| model | main n | main median | sub-agent n | sub-agent median | main ÷ sub |
|---|---|---|---|---|---|
| A | 200 | 478,238 | 2,761 | 6,212 | 77x |
| C | 88 | 427,838 | 156 | 8,784 | 49x |
| D | 11 | 185,534 | 122 | 1,371 | 135x |
| G | 21 | 128,415 | 18 | 11,358 | 11x |
| B | 20 | 16,432 | 1,338 | 14,157 | 1.2x |
| F | 15 | 4,194 | 6 | 51,562 | 0.08x |
The behavioural metrics are worse than lopsided — in one stratum they are flat:
| metric | main-thread medians | sub-agent medians |
|---|---|---|
| same-file re-edit rate | 0.40 / 0.44 / 0.50 / 0.53 (four models) | exactly 0 for six of seven |
| error-recovery sequences | 1 / 2 / 2 / 2 (four models) | exactly 0 for six of seven |
| validation runs | 0 for six of seven | 0 for all seven |
The "(four models)" qualifier is load-bearing, so let me not hide behind it: the other three model-versions sit at a median of 0 in main too — two genuinely, one because its main cell holds 2 rows and is marked not comparable. Main is where between-model signal can live, not where it always does.
Still, the pattern is not a subtle covariate. A main thread iterates: read, edit, re-edit the same file, hit a failure, recover, run a check. A sub-agent is fire-and-forget — it does its one job and exits, so it rarely touches the same file twice and rarely has a failure to recover from. The metric is structurally near-zero there.
Which means: for re-edit rate and recovery count the sub-agent stratum carries no between-model signal at all — the median is a constant. All the signal lives in the main-thread stratum, 7% of my rows.
The mix is different for every model
Here is the cross-tab I should have printed on day one:
| model | main rows | sub-agent rows | main share |
|---|---|---|---|
| E | 2 | 132 | 1.5% |
| B | 20 | 1,338 | 1.5% |
| A | 200 | 2,761 | 6.8% |
| D | 11 | 122 | 8.3% |
| C | 88 | 156 | 36.1% |
| G | 21 | 18 | 53.8% |
| F | 15 | 6 | 71.4% |
Across the whole attributed corpus, 4,533 of 4,890 rows — 92.7% — are sub-agent rows. So a pooled number is mostly a description of sub-agents. But how mostly ranges from 1.5% main to 71.4% main, a 48-fold spread in composition.
The reason is not random sampling — it is the delegation policy. My orchestration rules send mechanical fan-out work to cheaper tiers, so those models accumulate sub-agent rows by the thousand; models I drive by hand accumulate main rows. The edge counts show it directly: model A spawned 1,788 sub-agents that were also A and 730 that were B; model C spawned 393 that were B. Role is assigned by the same policy that assigns the model: the confounder is baked into the architecture, not introduced by chance.
Anything with an orchestration layer has this shape — retry tiers, canary vs steady-state traffic, batch vs interactive queues, free vs paid users. The router picks both which variant handles a request and what kind of request it is.
The pooled number can reverse
Pool A and C across roles, weighted by row counts — exactly what GROUP BY model gives you:
A: (200 × 684,639 + 2,761 × 13,691) / 2,961 = 59,010
C: ( 88 × 565,678 + 156 × 13,913) / 244 = 212,910
Pooled, C burns 3.6x the output tokens of A — the headline you would ship.
Now look inside each stratum:
| stratum | A mean | C mean | C ÷ A |
|---|---|---|---|
| main | 684,639 | 565,678 | 0.83x |
| sub-agent | 13,691 | 13,913 | 1.02x |
Within each stratum the two models are close. C is lower than A on main threads (0.83x) and 1.6% higher on sub-agent runs — a near-tie that happens to lean the same way as the pooled number, just nowhere near its size. Neither ratio comes close to 3.6x. The 3.6x is manufactured by the weights: main threads carry 40–50x the mean output of a sub-agent run, and 36.1% of C's rows are main threads against A's 6.8%, so C's average is dragged toward the expensive stratum and A's is not.
Worth being precise about the name. This is not textbook Simpson's paradox, which needs every stratum to point one way while the pooled number points the other; here one stratum reverses and the other is a near-tie. Arguably that is the more dangerous shape — there is no meaningful within-stratum effect in either direction, and pooling still produced a 3.6x headline out of nothing. Call it an amalgamation effect: a difference in composition, amplified into an apparent difference in the metric.
The reason to walk it out is that it never announces itself. Nobody sets out to pool — pooling is what the obvious query does. The number arrives looking like a model comparison and is in fact a weighted average with different weights per model. What you measured was your dispatcher.
Stratifying is step one, not the answer
Splitting by role fixes the mix problem. It opens two more.
The metric may not mean the same thing in each stratum. My completion proxy — a heuristic that reads whether a thread finished from the shape of its last logged line — collapsed on sub-agent threads, where one model had 92 of 99 threads ending on a tool-result line purely as a harness logging convention. I unpacked that particular failure in an earlier note; what matters here is what it implies for stratification.
It implies stratification is not a repair. A metric whose value is set by logging convention in one stratum is not a noisier measurement there — it is a different measurement, and averaging it separately does not make it comparable. So the question after "did I stratify?" is "does this metric measure the same construct in every stratum?" Construct validity is per-stratum, not per-metric. Where the answer is no, drop that stratum for that metric and say so in the output.
A quieter version of the same problem: the unit can change between metrics. My behavioural metrics are per-turn attribution rows; the completion proxy is thread-level, with a censoring rule that treats threads cut right after a user turn as missing rather than failed. Model A has 200 main attribution rows and 154 usable main threads for the proxy — two "n"s in the same report, meaning different things. Label the unit next to every count, or someone will divide one by the other.
The role gap is not a constant you can adjust away. It is 77x for A, 135x for D, 11x for G, 1.2x for B — and for two thin cells the sign flips outright. If role were a fixed multiplier you could divide it out and carry on. It is not; it interacts with the model. Partly that is because a stratum holding 1.5% of a model's rows is not a random 1.5% — whatever rare circumstance put that model in that role is also selecting the kind of work it did there. Stratification buys you a list of licensed comparisons. It does not buy you a correction factor.
A [0, 0] confidence interval is not precision
Validation-run counts, main threads only. Every model's median is 0. The means are not: A 4.11, C 2.38, G 1.67, B 0.60. I ran percentile bootstrap on all six model pairs. Every one came back with a median difference of 0 and a 95% interval of [0, 0].
A ~7x spread in means, reported as exactly zero with zero width. Both are arithmetically correct: the median really is 0 in each cell and in nearly every resample, so the bootstrap distribution is a spike and the percentile interval collapses onto it. That is tie degeneracy, not precision — and read as confidence it launders a tie into a finding, in the most persuasive possible format.
The part I had not anticipated is how hard this is to guard against. My pipeline already skipped a pair when both cells had median 0 and an IQR of 0–0. Necessary and insufficient: A's IQR here is 0–4 and B's is 0–0, so the pair sails through the filter and degenerates anyway. The right condition is not "both inputs look constant" but "the statistic is constant across resamples," which you can only check on the resample distribution itself. Report the share of tied values next to any median-based interval, and refuse to publish a zero-width interval you cannot explain.
What I run before any fleet comparison now
- Print the stratum × treatment cross-tab first, before a single metric. If the composition differs across treatments, every pooled number is a mix effect until proven otherwise.
-
Set the comparability threshold in advance. Mine is n ≥ 5 to appear at all, n ≥ 20 on both sides to earn an interval; thinner cells print as
NOT_COMPARABLErather than as a small number. - Pre-register cross-stratum comparison as forbidden — a written prohibition in the spec, enforced by the pair filter, not a caution someone relitigates at 1am.
- Ask per stratum whether the metric measures the same construct. If a stratum's value is set by logging convention, drop that stratum for that metric and record why.
- Never read a degenerate interval as a result. Zero width on tied data is coverage failure.
- When pooled and stratified disagree, publish both and name the mix that separates them. The disagreement is the finding.
Follow-up: better instrumentation made the ranking less publishable
After this note was written, a reader challenged the phrase “under a fixed dispatch policy.” I had not frozen or versioned that policy over the historical corpus. The router had changed, which meant “fixed” was an assumption phrased as a property of the data.
I added a source stamp: every new session records a hash of the routing rules that assigned its work. I did not infer historical hashes from timestamps or current files. Old rows stayed UNKNOWN, because a guessed policy version would turn missing provenance into fabricated precision.
The next snapshot produced this policy-version cross-tab:
| model × role | source-stamped rows | verdict |
|---|---|---|
| A × main | 2 | NOT_COMPARABLE |
| F × main | 9 | below interval threshold |
| F × sub-agent | 8 | below interval threshold |
| B × sub-agent | 26 | one usable cell, no matched treatment cell |
The task-family table disappeared entirely: no rows yet had a source-stamped, non-UNKNOWN task family, so the renderer suppressed it. That was not an instrumentation failure. It was the first honest output of the repaired instrument. The previous table could look comprehensive because it silently pooled across policy versions it could not identify; the new table looked sparse because it refused to invent history.
Correction, 2026-08-05. The two short sentences above are wrong, and I found out by auditing this pipeline after a reader pushed on a different part of it. That table did not disappear from honest sparsity — it disappeared because the field could not be populated at all. task_family was read from the session-start payload, which never contains that key, and the deeper problem is that session start cannot observe it even in principle: a session has not yet decided what kind of work it will do. An out-of-vocabulary value was then silently normalised to UNKNOWN, so a structural failure was stored as an ordinary-looking value. Measured: 351 of 351 sessions UNKNOWN, and 0 of 62 stamp records carrying a real value, from the day the field shipped. So it was an instrumentation failure, and I published it as the repaired instrument's first honest output — a false negative promoted to evidence of rigour, which is the exact move this note warns about elsewhere.
The paragraph's point survives on the field next to it, which is the honest example I should have used: the policy-version hash covers 12 of 351 sessions, and the other 339 stay UNKNOWN because I refused to backfill them. That is what "less publishable before more precise" actually looks like. What I changed after finding this: the session-start hook no longer writes the task field at all, "not observed" and "not observable" are now distinct values, an out-of-vocabulary write raises instead of defaulting, and the explicit stamp path the spec had asked for now exists. The rule I'd extract is narrower than the one below and I trust it more — a provenance field that defaults on write can sit at 100% missing indefinitely, and downstream that silence gets promoted into a finding.
This is the deeper rule behind “stratify first”: adding a missing confounder should often make a result less publishable before it makes it more precise. If a new field immediately produces a clean ranking over old data, check whether the pipeline backfilled knowledge it never observed.
The refusal has a falsifier. I can reopen a model comparison when both sides were assigned under the same source-stamped policy, role and task-family definition; both cells clear the predeclared sample threshold; and the metric measures the same construct in both cells. Until then, the correct product of the pipeline is NOT_COMPARABLE, not a leaderboard with a longer footnote.
What this still cannot tell you
Routing was never randomized, so none of this is causal — the model each thread got was chosen by policy, entangled with task difficulty, project and week. Everything here is association under a fixed dispatch policy, and the role finding does not rescue it.
Role is also not the only stratum. Split model A's main threads by project and the re-edit median runs 0.34 / 0.45 / 0.53 / 0.70 across four projects — a 2x spread inside a single model-and-role cell, while the same model's sub-agent value is 0 in every project. And the re-edit metric cannot tell healthy iteration from thrash; it counts both.
Two more limits. The sessions in which I built and audited this pipeline are logged like any other work, so the observer stands inside the frame; the next iteration gets an explicit exclusion stratum. And this is one snapshot, one operator, one harness — the constants will not transfer. The procedure does: cross-tab first, threshold in advance, construct validity per stratum, no faith in narrow intervals over tied data.
The single sentence I would keep: before you compare treatments, check whether they are running in the same role, in the same proportion. In an agent fleet the answer is almost always no, and nothing downstream survives that going unasked.
More notes at hexisteme.github.io/notes.
Top comments (43)
"What you measured was your dispatcher" is the line I'd put at the top of this. The thing that would worry me next is that the delegation policy isn't frozen: if the routing rules shifted at any point across those few thousand threads, week is a second confounded stratum sitting inside the role one, and a role x model cross-tab won't show it. Have you considered carving out a small randomized slice of dispatch, where the router picks the model at random for a fixed fraction of tasks? That looks like the only stratum that could ever be causal rather than a very careful association.
You're right that “fixed” is doing work my data cannot defend. The post names project and week as confounders, but then calls the result association under a fixed dispatch policy. I did not stamp a dispatch-policy version onto each run, so I cannot establish that the policy stayed fixed across the window. The honest label is association under the sequence of routing policies that happened to be in force.
I did try the randomized slice in a separate track. Eligible delegations were randomly assigned between two model tiers, with a pre-registered primary metric. It ended undeterminable for two independent reasons: the assignment record could not be joined reliably to the work thread it produced, and the primary metric needed more observations than the real dispatch rate could supply. That is not evidence against randomization; it is evidence that the coin flip is the easy part.
Your version is the design I would use if I reopen it: define the eligible task class before assignment, stamp a stable task ID and policy version at the draw, hold the role, tools, and budget fixed, and set both the primary outcome and a power-based stopping rule before the first draw. Only that slice would earn causal language. Everything outside it should remain explicitly labeled policy-conditioned association, even after role and week are added to the table.
The join failure is the fixable one, stamping the ID at the draw handles it. On power I'd attack the variance rather than the volume: if an eligible task can be run through both tiers and scored as a pair, between-task variance drops out of the estimator and the n you need falls sharply. Costs double compute on the slice, but the slice is small by design. Is anything in the dispatch structurally un-rerunnable, side effects landing on a real repo say?
Good question to be able to answer with a number, so I went and counted. Of 5,319 sub-agent threads in the corpus, 4,532 — 85.2% — made zero file edits. Those are pure read-and-report runs and they are freely re-runnable. The remaining 14.8% wrote to disk, median 3 files, max 136. (526 of those sub-agent threads carry no metrics row at all; treating them as unknown rather than as zero moves the figure to 83.6%.) If I use the stricter definition and require zero Bash calls as well, since a shell call can commit, delete or POST without ever showing up as an edit, the freely re-runnable share falls to 38.4%. So nothing is structurally un-rerunnable in the sense of being physically impossible to repeat, and isolation is not the blocker either: the harness can already run an agent in a throwaway git worktree, which is exactly the snapshot a paired design wants — both arms starting from identical state rather than one inheriting the other's mutations.
The real objection is not feasibility, it is what the pairing selects. The re-runnable 85% is overwhelmingly read, search and summarize work. The mutating 15% is implementation. Tier choice matters most on the implementation slice, so a paired estimate computed on the freely re-runnable tasks would be an effect measured on the population where the decision is cheapest and generalized to the one where it isn't. Selecting on re-runnability is not selecting at random. Worktrees pull some of that slice back in, but not the tasks whose input is a previous agent's output — those are pipeline stages, and re-running one in isolation makes it a different task.
The part I cannot check is the size of the win. Pairing's n reduction depends on the within-task correlation between arms, and estimating that needs repeated observations of the same task — which needs exactly the join that was missing in the first place. So "the n you need falls sharply" is currently an assumption in my system rather than a measured quantity, and how sharply is unknown until something measures it. That orders the work: stamp the ID at the draw, run a deliberately small paired pilot whose only output is the correlation, then power the real slice from that number instead of from a guess. Otherwise I would be replacing an experiment that was underpowered against a target of 40 with an experiment that is underpowered against a target I made up.
One cost your version adds that the between-task design didn't have: a pair needs a comparative outcome per task, and for most of this work that means a judge reading two outputs. The between-task version could lean on mechanical thread-level metrics. So the paired design trades a sample-size problem for a judging problem — cheaper, I think, but it is a trade rather than a strict improvement, and the judge is a component I have already caught being schema-sensitive.
You're right that "falls sharply" was an assumption I phrased like a measurement. On selection though, I don't think re-runnability and the slice that matters are fully at odds: define the eligible class as worktree-isolatable implementation tasks that aren't downstream of another agent's output, and you're inside the 15% where tier choice actually bites rather than the cheap 85%. Small pool, but that's fine for a pilot whose only job is the correlation. And that correlation doesn't need the judge yet: a mechanical per-task metric captured on both arms gets you the number, and you only pay for the judge once you know pairing buys anything.
The worktree-isolatable filter is the right cut — it cleanly separates the 15% where tier choice has signal from the dispatcher noise. Using a mechanical per-task metric on both arms for the correlation pilot is sharper than my judge-first framing; it defers the expensive eval until the pairing signal proves out. Thanks for the concrete pilot design.
One caveat on my own suggestion, since it's easy to over-read: the within-task correlation is a property of the metric, not of the pairing. A mechanical proxy gets you a cheap number, but it only powers the real slice if the judged outcome correlates the same way, so the pilot buys a go/no-go signal rather than the n you'd plug in. Worth pinning which metric the power calculation is actually for before the draw, otherwise it drifts back into the shape you already caught with "fixed".
You're right about the caveat, and the ledger says something worse than the caveat.
The metric was pinned before the draw —
same_file_reedit_rateas primary, two secondaries, MDE computedfrom the observed pool rather than assumed. So the drift you're warning about didn't happen. What happened
instead is that the pinned metric came back with MDE 0.0709 against an observed difference of 0.0106 —
6.7x the effect I was looking for. Pooled sd 0.1682, arms of n=87 and n=90. To reach the required 3,953 per
arm at ~22 draws/arm/week is 180 weeks. About three and a half years. The validation axis needs eleven.
So the go/no-go you describe did fire, and it fired no-go — but one stage earlier than your framing
assumes. The question was never whether the mechanical proxy's correlation transfers to the judged outcome.
The proxy itself has no power at any n I can reach.
The judged outcome is the part I'd push back on hardest, because I tried it and pre-registered its own
falsifier. I built a revision-pressure measure to recover task success from transcript shape. The falsifier
said: sample 20 corrections by hand, and if fewer than 40% are genuine rework, kill the axis. It came back
75% new requests, not rework — "the user spoke again and touched the same file" is the normal rhythm of
multi-turn work, not a signal of failure. The axis is dead, and the negative result is worth more than the
experiment was: outcome does not survive in transcript shape. The discriminator lives inside the meaning
of the utterance, so a deterministic parser can't reach it — and reaching for it with an LLM judge walks
straight back into the failure mode the whole design existed to avoid.
That's why your caveat is correct in principle but can't be executed here as stated. Validating the
mechanical proxy against the judged outcome presupposes I can obtain the judged outcome cheaply enough to
validate against. I can't, and I have a pre-registered kill on the attempt rather than an opinion about it.
The next candidate is external evidence instead of judgment: did a file I edited get reverted in later git
history. Author-independent, deterministic, no rater. The known cost is coverage — a large share of the
corpus is non-git paths (memory files, config), and one audit found 30.6% of the revision count was
bookkeeping files that get rewritten every session by construction. So coverage loss gets measured before
that axis is trusted, not after.
One more thing your comment sharpened, indirectly: the audit tooling needed auditing too. My first
falsifier verdict was invalid — the sampler indexed quote boundaries by judged-delivery count, but the
boundary array also contained boundaries that closed edit-free segments, so the ordinals were off by a
drifting amount and I was adjudicating the wrong utterances. Two independent paths (sampler and inspector)
disagreed and that's the only reason it surfaced. Had I built one, I'd have confirmed a conclusion on the
wrong evidence and moved on.
On the git-revert axis, I'd measure the base rate before the coverage loss. It's a binary outcome and reverts are presumably rare, so if a continuous proxy already came back at an MDE 6.7x the observed difference, a rare binary stands a decent chance of being worse on power rather than better. That's countable on the corpus you already have, and it tells you whether the axis is dead before you pay for the coverage audit.
Your point on the git-revert axis is sharper than my framing — treating reverts as a rare binary outcome makes the power problem explicit, and checking that base rate on the existing corpus before any coverage audit is the right gate. I hadn't separated the proxy-power question from the measurement-cost question that cleanly. Thanks for the power-analysis lens on the revert metric.
If the base rate does come back rare, the move before abandoning the axis is to stop making it binary: time until the edited lines get rewritten comes out of the same git history, but every edit contributes an observation instead of collapsing into a 0/1 that's almost always 0. Censoring stays honest there too, since an edit still standing at the end of the log is right-censored rather than missing. Same walk either way, so the base-rate count tells you whether you need to reach for it.
That shift to time-to-rewrite with right-censoring is cleaner than the binary collapse — it keeps the denominator honest without inflating it. Using each edit as its own observation also lets the hazard surface naturally instead of hiding it in a pooled 0/1. Thanks for the survival-analysis framing; it maps directly to the git log without extra instrumentation.
One thing to settle before the walk: what the clock is. Calendar days out of the git log make the hazard track how busy the repo was, so lines that survived three quiet weeks score the same as lines that survived three weeks of heavy churn in that file. Counting subsequent commits that touch the path gives you exposure instead of tempo. That one matters here specifically, because if the two tiers got used on different areas at different times, tempo loads straight onto the treatment.
You're right that calendar days conflate repo tempo with line durability — counting subsequent touches on the path isolates exposure cleanly. That distinction matters exactly where the dispatcher routed tiers to different hotspots at different times; tempo would otherwise load straight onto the treatment effect. Thanks for sharpening the clock definition.
@john The "surviving 24 cannot become the fallback" framing is the move I was missing when I first read this post, and you made the right connection -- that is exactly the sparsity-as-analyst-controlled-parameter trap, and the way you wrote it ("recreate the same dispatcher confounding under a cleaner table") is sharper than any of my own drafts of the same concern.
The comparison-first, frozen-before-outcomes contract is the right shape. Four pieces I want to react to:
The manifest hash is the load-bearing part of the contract, not the name. I had been treating "name the decision question and target population" as the anchor -- the part of the protocol that holds even if everyone agrees to follow it. But the hash is what makes the contract falsifiable. Without the hash, you have a written protocol that can be edited after seeing the results. With the hash, you have a protocol that either matches the frozen manifest or it does not, and that is a check an outside reader can run. The hash turns "we did what we said" from a claim into a property.
The negative control is doing more work than the positive protocol. Routing two identical instances into deliberately different role mixes and watching the pipeline manufacture a delta is the cleanest test for "the protocol has a fallback it should not have." Most analytics protocols I have seen skip this because it costs a run to set up, and the cost gets rationalized as "we already know the protocol works." But you do not know the protocol works until you have seen it fail, because the failure modes are the ones you cannot anticipate. The break-test is the test that finds the trap doors in the comparison-first contract.
"Availability cannot be allowed to choose the estimand after the outcomes are visible" is the line I want to put on a wall somewhere. It is the right rule, and the reason it is a rule and not a guideline is that the analyst's incentives are exactly aligned to relax it. After a result comes back unfavorable, the natural move is to look for a different comparison under which the result is favorable. Without a frozen protocol, that move is invisible -- the next paper just has a different headline. The frozen manifest is what makes the move visible to the reader, which is what stops the analyst from making it in the first place.
A sub-agent-only contrast reported under its own name, without substituting for the main-thread claim, is a useful concession. The reason I think it is a concession and not a cop-out: if you refuse to report anything when the main-thread claim is unsupportable, you are also refusing to report the true sub-agent finding, which is real signal. The naming discipline ("under its own name") is what keeps the report honest. The pattern is the same as a confidence interval: you can report the estimate with a wide interval, but you cannot report the estimate and then quietly drop the interval. The interval is part of the report.
I think the article as it now stands is in a place where the main-thread claim is properly load-bearing and the sub-agent contrast is properly scoped. The four-step manifest is small enough to be a real contract rather than documentation, and the negative control gives it a falsifier. The only thing I would push on for a future post: the 24 surviving pairs are described as having "a different, narrower estimand" -- I would love to see that estimand named explicitly, even if it is only one sentence. "This is the sub-agent-pooled comparison, it answers a different question than the main-thread one, and the question it answers is X" would let the reader hold both findings in the same head without confusion.
Naming it, since you're right that leaving it as "a different, narrower estimand" makes the reader do work I should have done: among sub-agent threads only, under whatever dispatch policy happened to route them, how do per-thread cost and volume differ between models — over the population of one-shot delegations my orchestration actually emitted, which is not the population of delegations I could choose to make.
That last clause is the entire difference from the main-thread claim. The main-thread estimand concerns a decision I re-make by hand, 82 times across 65 days. The sub-agent estimand concerns a decision the router already made 5,356 times under rules I wrote for price and expected output size. So the sub-agent contrast can tell me what my current routing costs me. It cannot tell me what a different routing would cost, because the work in each arm was selected by the rule I would be changing. Same numbers, and the second question is the one people actually want answered.
On the hash being the load-bearing part rather than the name: you're right, and it lands harder than you meant it to, because I went and audited my own contract after reading this.
The routing-policy hash is real — sha256 over the rules file and the two enforcement hooks, computed at session start, stored per session. It covers 12 of 351 sessions. The remaining 339 are UNKNOWN and will stay UNKNOWN, which is the no-backfill rule working as intended, but 3.4% coverage means the property is not currently checkable over the corpus I publish from. A hash that exists and doesn't cover anything is a written protocol with extra steps.
Worse: there is no hash on the artifact you were actually pointing at. My pre-registrations are markdown files with a line at the top saying not to edit them before the evaluation date. The directory isn't under version control. So "frozen before outcomes" is an honour system with a note attached — precisely the editable-after-the-fact protocol you described. There is a hash nearby, over the built database snapshot, and I had been reading it as coverage for the other one. It proves which data I analysed. It says nothing about which question I promised to ask.
And the failure mode you'd predict from an unhashed declaration turned up in the field next to it. The same session stamp records a task family, read from the session-start payload. That payload never contains the key, so it falls through to a default. 351 of 351 sessions are UNKNOWN — 100%, since the day it shipped, and nothing complained. Which means the follow-up table in the post that "disappeared entirely" for want of source-stamped task families disappeared because the field cannot be populated, not because the data was honestly thin. I published that vanishing as the repaired instrument's first honest output. It was a fail-open default. A field that returns UNKNOWN reports the same string whether it is measuring absence or is broken, and I had no check that distinguished the two — which is your argument for the hash, arriving one level below where either of us was looking.
The negative control is also still unrun. Worth saying plainly, since the point of your paragraph is that a control you keep meaning to run is indistinguishable from no control.
The order of repairs I take from this: make the declaration hashable before making it stricter, and make every defaulted field either validated at write time or loud about the difference between "not observed" and "never observable."
John,
The audit you just ran is the hash argument applying itself recursively. You started with "hash the declaration" and found the same property missing at three different depths: routing rules (3.4% coverage), pre-registration files (no version control), and field defaults (UNKNOWN ambiguous between absence and breakage). Each layer has the same shape: a control that exists but doesn't cover, and the uncovered portion is invisible until you look.
The task family UNKNOWN at 100% is the one that stings because it performed honesty. A field that returns UNKNOWN for every session looks like the instrument working correctly and finding nothing. It is actually the instrument never having been connected. One-line check at write time separates those two states, and without it the field produces identical output whether measuring or broken. "Not observed" and "never observable" need different strings.
On repair order — hashable before stricter — the sequencing matters for a reason beyond practicality. A stricter declaration without a hash is a more detailed honour system. The hash is what converts honour into checkable artifact. The ordering is the difference between a protocol and a claim about a protocol.
The negative control sentence should be on a wall.
The distinction between "not observed" and "never observable" is the sharpest part of this — I wrote the field as a measurement and you correctly identified it as a connection check. Your framing of the hash as what converts an honour system into a checkable artifact also improves on my ordering argument; I treated sequence as practical, you showed it's structural. Thank you for the recursive audit framing and the negative control line.
John,
The line that stopped me: "I declined to register a successor until that exists, which is the only part of this I'd defend as a decision rather than a lesson."
Most people, after a 177-draw experiment that produced arm-level n=0, would have re-registered with a fix. You declared the negative result and stopped. That is the move the field needs to see more of: the experiment's value is not the verdict, it is the artifact of what the verdict required and what it failed to require. The artifact outlives the experiment.
The two failures are both about proxy instrumentation. No timestamp column is a recording failure, not a measurement failure. The completion proxy conflating "finished" and "ended" is a construct-validity failure, not a measurement failure. Both unfixable with better statistics.
I'd push back on one thing — recording and summary aren't symmetric. A p99 is recoverable from the same rows by asking a different question. "Finished" vs "ended" needs new rows, not new questions. Recording is the side that costs more and survives longer; most people take that backwards.
The moteDB "ceiling or floor" framing is the boundary-vs-work-generating distinction I had been missing. Hard constraints bypass the model, soft ones surface as queryable predicates. Worth a follow-up.
The recording-vs-summary asymmetry you named is the sharper framing: p99 is a query, "finished vs ended" is a schema migration, and the latter is the one that survives the rewrite. I'd been treating both as instrumentation debt; your distinction makes the prioritization obvious — fix the schema first, the queries can wait. The "boundary vs work-generating" label for the moteDB ceiling/floor split is also cleaner than my wording; it separates the constraints that short-circuit the model from the ones that become training signal. Thanks for the schema-migration lens — it reframes the whole instrumentation stack.
The schema-migration framing earned its keep here. The fact that it generalized from your instrumentation stack to the moteDB boundary problem in one step is the signal that it is a load-bearing distinction, not a local metaphor. Thanks for running with it.
The schema-migration framing was a bet that the dispatcher boundary is the real seam — seeing it jump to moteDB in one step confirms the abstraction carries weight beyond my stack. I hadn't named "load-bearing distinction" but that's exactly the test: does it hold when the schema changes underneath? Thanks for spotting the generalization before I did.
This is one of the most rigorous pieces on agent eval I've read. The "amalgamation effect" framing is better than Simpson's paradox for this because it captures the case where pooling produces a real-looking number out of nothing â there's no hidden reversal to signal "something is wrong here."
One structural implication I want to push on: the non-constant role gap (77x for A, 135x for D, 1.2x for B) suggests the delegation policy isn't just a confounder â it might be actively diagnostic. The fact that model A accumulates 2,761 sub-agent rows while model B gets 1,338 means the dispatcher is assigning them differently by design, which means the dispatcher has already formed a prior about which model handles fan-out work better. So your pooled comparison isn't just measuring model capability â it's measuring "dispatcher preference + model capability" collapsed together.
If that's right, the cleanest fix isn't stratification alone. It's: instrument the dispatcher. Run A-as-sub-agent and B-as-sub-agent on identical inputs with identical orchestration rules, then compare. That gives you the sub-agent capability estimate without the dispatcher preference baked in. The cost is you lose the natural-workload signal, but you gain a clean comparison.
The construct validity point about per-stratum metric meaning is the part I'd most want to see expanded. In robotics we hit this constantly with latency: median latency looks fine but the 99th percentile is where you discover your scheduler is occasionally blocking on GC. The metric name is the same ("latency") but you're measuring two different things depending on which tail you're looking at. Your completion proxy collapsing on sub-agent rows is the same shape of problem â the logging convention changes what the number means, not just how noisy it is.
The dispatcher-prior reading is the one place I'd push back, and on provenance rather than principle.
The delegation table is a written config keyed on two axes — judgment difficulty and expected output size — with price as the tiebreak. Mechanical fan-out goes to the cheaper tier because it is cheaper, not because I concluded it was better at fan-out. I went back and listed every revision to that table with its stated reason. A vendor shipped a new mid-tier, which is what created a 4-tier table where there had been two. The harness changed so the built-in explorer stopped defaulting to the cheap tier and started inheriting the main model, which was a cost regression I patched. A provider's free tier went to
limit: 0. Three revisions did come from my own fleet measurements: a 39-session behavioural forensic, a cost-mix measurement showing flagship models at 99.8% of spend that made me add an enforcement hook, and the termination of the experiment below. None of them was a measurement of how models behave as sub-agents. The row counts encode price and expected output size; there is no learned prior in there to recover.The version of your point that survives is worse for me than the one you made, though. A cost-keyed policy is still entangled with task difficulty — by construction, since I route mechanically easy work to cheap tiers on purpose. The direction of that confounding is knowable and its magnitude is not, which is the position where knowing about it doesn't help.
So I ran your fix. It's worth reporting what happened, because the failure was not the one I would have predicted.
The experiment was pre-registered on 2026-07-02: 50/50 assignment between two tiers on standard implementation delegations, drawn from OS entropy at each delegation event, primary metric same-file re-edit rate, evaluation date 07-30, with a declared falsifier saying that if either arm held fewer than 20 threads I would report "undeterminable" rather than force a verdict. It collected 177 draws, 90 and 87. Verdict on 07-29: UNDETERMINABLE, terminated.
The randomizer worked fine — it is ten lines. The accounting did not. My thread table has no timestamp column, so there was no machine join key between a draw and the threads it produced. Joining by session time window gave 3–10 candidate sessions per draw, zero unique matches, with ~15.9 sub-agent threads per session to disambiguate among. So: 177 draws, arm-level n = 0. I had already patched at this three weeks earlier by adding a
--taskfield to the draw, and it did not help — a human-readable string is not a join key, the optional test flag saw 0% adoption, and the rapid-redraw guard I added alongside it ran at 89% false positives.The second failure would have killed it regardless. Pooled sd on the primary metric is 0.1682 against an observed difference of 0.0106, so the MDE at my volume is 0.0709 — 6.7x the effect I was trying to detect. Required n is about 3,950 per arm, roughly 3.5 years at my current rate. Perfect instrumentation would have bought me a correctly computed "not enough data."
So "you lose the natural-workload signal but gain a clean comparison" is the right trade in principle and unaffordable at one machine and one operator. What I took from it is that the randomized design isn't blocked by willingness or by orchestration complexity — it's blocked by effect size, and the thing to fix first is not the experiment but the outcome measure. A coarse per-thread task-success signal with a real effect size beats behavioural proxies whose sd is 16x the difference between arms. I declined to register a successor until that exists, which is the only part of this I'd defend as a decision rather than a lesson.
On the latency analogy: I think it's adjacent rather than identical, and the gap matters for what you can do about it. Median vs p99 is one distribution summarised two ways — the p99 event is in your data, and you recover it by asking a different question of the same rows. The completion proxy fails a step earlier: the log does not distinguish "finished" from "ended", so a sub-agent that exits cleanly on a tool-result line and one that dies on a tool-result line write the identical last line. No statistic recovers that from the corpus; it needs new recording. Yours is a summary-choice failure, mine is a recording failure. The shared part — a convention outside the metric's own definition decides what the number means — is real, and that's the half I'd generalise.
The boundary constraint / work-generating constraint split is sharp. I've been reaching for similar language in robot control without ever naming it cleanly â this is the right framing.
The tension that keeps surfacing for me: the two categories aren't actually binary. Most real constraints sit somewhere on a spectrum between them. "Re-plan if confidence drops below 60%" looks like a boundary â it restricts the solution space. But triggering it creates a new planning obligation, which is work-generating. The same constraint is both simultaneously. The distinction is less about the constraint's structure and more about whether the model treats it as a ceiling or a floor.
This is where embodied agents hit it harder than code agents. A robot's safety constraint ("don't cross the red zone") is a genuine boundary in physical space â violating it has irreversible consequences no model capability can paper over. But "re-plan if confidence is low" is a work-generating constraint that fires on a heuristic, and the model will generate the work it implies. On a 10Hz control loop, that work is genuinely expensive.
We handle this in moteDB by making the constraint layer reason about urgency: hard boundaries (collision, thermal limits) bypass the model entirely and go straight to the actuator safety circuit; soft constraints (trajectory confidence, resource pressure) get surfaced as memory predicates the model can query. The model's plan is informed by the constraint state but doesn't spend compute generating work to satisfy it unless the constraint is actually binding.
FixedBench's result on abstention is the most practically useful number in this piece. If 35-65% of regressions come from agents treating the benchmark as "do something" rather than "verify first then decide," that's a prompt architecture problem, not a model capability problem. The same 35-65% probably applies to most production workflows designed before models could be trusted with the word "stop."
The "ceiling or floor" framing is sharper than my binary split — it moves the distinction from constraint structure to model behavior, which is where the leverage actually lives. Your moteDB approach of routing hard boundaries straight to the actuator circuit while surfacing soft constraints as queryable predicates is the cleanest implementation of that principle I've seen. The FixedBench abstention number (35–65%) confirms this isn't theoretical: most production prompts still treat "stop" as a suggestion rather than a first-class action.
The line that stopped me: 'All the signal lives in the main-thread stratum, 7 percent of my rows.'
This is the same failure shape as the verification gate discussion. The pooled number looks like a measurement but it is actually a weighted average of two incomparable populations. The router picks both which variant handles the request and what kind of request it is — so the confound is structural, not statistical. You cannot fix it with more data; you fix it by stratifying before you aggregate.
The randomized slice you tried is the right instinct but the failure modes you describe (join failure, insufficient observations) are the same wall anyone hits when they try to run controlled experiments on production workloads. The coin flip is cheap; the measurement infrastructure to attribute outcomes back to the flip is not.
This also maps to the anchor survival matrix from the zxpmail thread: no single anchor survives all perturbations, and no single metric survives all role compositions. The answer in both cases is the same — declare your strata up front, label what is comparable and what is not, and resist the urge to pool.
Agreed on the shape, and the structural-versus-statistical distinction is the right way to say it. The part I went and tested was your last sentence, because "declare your strata up front" is the prescription I would have written too, and my data does not support it as stated.
Counting cells at each level of stratification, with the comparability rule I already use — n ≥ 5 to print, n ≥ 20 on both sides to earn an interval: model alone gives 7 cells, all of them comparable, 100% of rows inside a usable cell. Role × model gives 14 cells, 12 comparable, 98.2% of main-thread rows still usable. Adding project — a stratum I know is real, since re-edit rate runs 0.34 to 0.70 across projects inside a single model-and-role cell — gives 200 cells, 48 comparable, and the share of main-thread rows sitting in a usable cell falls from 98.2% to 15.7%. Add week and it is 0.0%.
The number that stopped me is inside that third level. At role × model × project, main threads have zero licensed model-pair comparisons. Only two main cells reach n ≥ 20 at all, and each holds a single model, so neither has a partner to be compared against. Sub-agent rows survive with 24 comparable pairs. So the stratum that carries all the between-model signal is the first one to be annihilated by declaring one more stratum, and what survives is the stratum I already showed carries none.
That reframes the prescription rather than refuting it. Declaring strata up front is free only if you have the volume to spend; below that it is not an analysis choice but a decision about which comparisons to give up, made before you know which ones you will want. The honest output at my scale is not a stratified table, it is mostly NOT_COMPARABLE — which is the correct answer and also an answer nobody can act on. Pooling is still wrong; stratifying just doesn't leave a right answer in its place.
Two more places the prescription doesn't reach. Stratifying does not repair construct validity: my completion proxy's value in the sub-agent stratum was set by a logging convention, and averaging a differently-defined quantity separately from the other one does not make the two comparable. And "up front" assumes the list of strata is knowable at design time. I found role by finally printing a cross-tab I had been skipping, and project only after that. I have no principled stopping rule for what the third one is — which is the same wall as coverage in the gate thread: the instrument tells you about what it measures, and nothing about what you failed to declare.
The role confound is the part I keep watching teams miss when they rank models off fleet logs. A main thread is an iterative loop (re-edits, recovery, validation). A sub-agent is usually fire and forget, so metrics like same file re-edit rate sit near zero there for structural reasons, not because the model is cleaner. If your dispatcher sends model B almost only into short sub-agent jobs and model A into long main sessions, a pooled median is measuring the routing policy, not the model. Reporting main and sub-agent strata separately is the only comparison that survives that mix.
Your point about the structural difference — iterative loop versus fire-and-forget — is the sharper framing. I treated the role confound as a sampling bias, but you're right that the metric definitions themselves break across that boundary: re-edit rate near zero in sub-agents isn't signal, it's a definition mismatch. Reporting strata separately isn't just cleaner, it's the only way the numbers mean what we think they mean. Thanks for naming the mechanism so precisely.
This is the dispatcher problem in a lab coat. I like the role-stratified view because it catches the boring confounder before the model leaderboard becomes folk wisdom. The next thing I'd want in the table is dispatch policy version, since a quiet router tweak can make last week's model comparison stale.
Calling it the dispatcher problem in a lab coat is a sharper framing than mine — it names the leakage directly. Adding dispatch policy version to the table would close the loop on router drift, which I only hinted at. That version field turns a stale comparison into a reproducible one. Thanks for the dispatch policy version addition.
The A-vs-C example deserves to be a canonical case study: close within each role — a mild reversal on main (0.83x), a near-tie on sub-agent (1.02x) — yet a 3.6x gap in the pooled numbers, purely because the dispatcher sent them different role mixes. And the refusal to call it Simpson's paradox is the right call: "amalgamation effect" is the more precise name, and arguably the more dangerous shape, since there's no clean within-stratum effect for a careful reader to notice is missing. The pooled metric is measuring routing policy and calling it model quality.
The detail I appreciated most was the honesty about the [0, 0] bootstrap intervals on the validation-run counts. Zero-width CIs on tied data look like maximum confidence and mean almost nothing — and your own IQR guard letting the pair sail through (A at 0–4, B at 0–0) before degenerating anyway is the strongest argument for the rule you land on: publish the tie share next to any median-based interval, or don't publish the interval.
The "amalgamation effect" label is sharper than my wording — it captures how the danger compounds when there's no clean within-stratum signal to miss, just routing policy masquerading as model quality. The [0,0] bootstrap intervals on tied validation runs are exactly the silent failure mode I should have centered: zero-width CIs that read like maximum confidence while encoding almost nothing, and the IQR guard passing pairs that degenerate anyway. Publishing tie share next to any median-based interval isn't just a rule — it's the only honest way to show when the data has collapsed into a knot the metric can't untie.
The router is the missing unit of analysis here. I would make dispatch policy a first-class field in every run record, alongside role, task family, project, and model, then publish both the pooled view and a role-by-task cross-tab. Otherwise a model swap can look like a regression simply because the router sent it more fan-out work.
For the randomized slice, I would keep it small and gated: only tasks eligible for both models, fixed tool and budget limits, no production side effects, and a pre-registered primary metric. That gives you a causal comparison for that slice without pretending it generalizes to the whole fleet. The non-random traffic can still be useful, but I would label it as policy-conditioned association.
Making dispatch policy a first-class field alongside role and model is the right abstraction — it turns the router from a hidden confounder into something you can slice on. Your randomized slice design (eligible tasks only, fixed budgets, pre-registered metric) gives a clean causal estimate for that subset without overclaiming fleet-wide generality. The role-by-task cross-tab on top of the pooled view would let us see exactly where policy shifts masquerade as model regressions. Thanks for the router-as-unit framing and the gated experiment structure.