Point a model at your repositories, ask why something broke, and you get an answer. Coherent, names a mechanism, shows you how it got there. The working is the problem rather than the reassurance. It looks the same whether it ends at a value in a record or at what usually happens, and only one of those is evidence.
I ran the same question at four levels of evidence, on two systems, across three models. The question was never find me a bug. It was always this already happened, work out how. The pass that worries me is the one just short of enough evidence. There is enough there to build a real hypothesis, and not enough to tell a real one from a plausible one. Two of its guesses sent me to look at things I had not checked. One was a dead end. The other was a real problem I had not known about, and nothing in either answer told me which was which.
A good guess costs the same to chase as a real one. You find out which it was at the end.
The four rungs, and what each one removes
I stopped thinking about this as adding context. Each rung takes away something the model would otherwise have to guess at, and that is the more useful way to look at it.
One. Repository access, broad. Point it at everything and ask. It removes nothing. The answer was structurally sensible, named components that were not involved, and arrived with no more hesitation than the correct one did three rungs later. If your team tried this once, got something confident and wrong, and decided the tooling is not there yet, this is probably where you stopped.
Two. Three bounded repositories, plus a written map. Which service talks to which, over what protocol, with what delivery and ordering guarantees. This removes rediscovery. What made it work was not the map. It was telling the model to treat the map as true and not go and check. That bought focus and gave up verification. Maps go stale quietly, and I have removed the step where it might have noticed.
Three. Plus traces and logs covering real executions. This removes guesswork about sequence and about what crossed a boundary. It is the closest it got without evidence of state, and it is the rung the top of this piece is about.
Four. Plus a small set of records showing the behaviour, customer fields removed and identifiers replaced, structure intact. This is the one that tells you which path the code actually took, and that turns out to be the whole game.
That last pass ran on Sonnet, which had been the weakest of the three lower down: vaguer, and closer to optimistic than the evidence supported. Opus and Fable pointed nearer the right area on thinner evidence. None of them got there. Then the one that had been worst at guessing was enough, once there was nothing left to guess.
I am not arguing that model choice is irrelevant. It changed how good the guessing was. It did not change whether there was guessing.
Why the records worked
The first defect I had already diagnosed, so I may have picked, at every rung, exactly the evidence that leads there. The second was a live bug on a project of my own, unsolved when I started. Old code, the kind that acquires a condition per edge case over years, so one entry point can take a large number of paths through it.
At rung three it reasoned forward. Here is what the code can do, here is what probably happened. At rung four it reasoned backward. Given these values in this record, this branch ran and these did not. Then it wrote a test for that path, ran it, confirmed the behaviour, and handed back a reproduction.
The reason is mundane. A branch you did not log is a branch you cannot rule out. Logging every branch in a service that has been accumulating conditions for years is not a decision anyone makes, because it costs storage, latency and review attention on every change, and the value is hypothetical until the day it is not. A record from an affected run settles it afterwards, because the values in it determine which conditions evaluated true. Not always. Two branches can converge on the same stored state, a later write can erase the evidence, and the deciding value may never have been persisted at all. Where none of those apply, the record tells you which branch ran.
Which is the job a log line would have done, if I had written one.
That is also the real difference between my two runs, and it is not technical. On the project I own, getting records was a query. Anywhere else it is a legal question somebody answered years ago without knowing they were answering it. That is a separate piece and I owe you one.
Where I could be wrong
Two runs, three models from one vendor. Not a benchmark.
Someone will say rung four hands it the diagnosis. I do not think so. Those are the same records a person opens, and opening them is not the same as knowing what they mean. The argument stands or falls on that: whether the records give you the answer, or give you what you need to find it.
What would falsify it: a defect where the branch taken is knowable from code and traces alone, with no conditional ambiguity. Rung four should add nothing.
What I keep ready now
- A brief per service and a communication map, written for a model, regenerated on merge rather than maintained by hand. Delivery and ordering guarantees included, since no codebase states them.
- A last-verified date on that map, because I am telling a model to trust it.
- One field-aware extraction script that strips sensitive values and leaves field names, timestamps and joins intact. Written while nothing is on fire.
- The habit of asking which branch it thinks ran, and what in the data says so. That question is the difference between rung three and rung four, and you can ask it from any rung.
Have you chased down a plausible side finding from a model and had it turn out to be real? Or turn out to be nothing, which costs exactly the same.
And if you are pointing a model at documentation you have told it to trust, how do you know the documentation is still true?
I write about architecture and AI engineering for companies that aren’t Google. The newsletter lives at mayankkaul.com
Top comments (7)
Rung four moves the rung-three ambiguity rather than deleting it. At rung three the answer announces itself as a hypothesis. At rung four it arrives as a derivation, and if the record falls into one of your exception cases (two branches converging on the same stored state, or the deciding value never getting persisted at all), that derivation has the same confident shape as a correct one. Backward reasoning fails looking like proof.
The trade is still worth making, because the replacement question is smaller: does this record actually determine the branch? That one is a property of schema plus code, so it is answerable in advance, while nothing is on fire. Walk each conditional and ask whether every value it reads is recoverable from persisted state once the run completes. Where it isn't, that branch sits in a permanent blind spot and no amount of rung-four evidence will settle it. It also gives you the affordable version of the branch logging you rule out: log only the deciding inputs that can't be recovered afterwards, a set you read off the code rather than off incident history.
That puts a constraint on item 3 too. An extractor that strips values while preserving field names, timestamps and joins keeps the structure, but if the deciding input is itself the sensitive field (customer tier, country code, consent flag, account age), the redacted record is rung three wearing rung four's clothes, and you now believe you're at rung four. The determinacy pass can hand that extractor a must-survive set, and those fields go out as stable equivalence-class tokens instead of blanks, so branch attribution still works while the plaintext stays home.
What makes this fundable is the shape of the ask. Branch logging is an unbounded request against every conditional in a service, so it competes with feature work and loses, every time, and correctly. A list of inputs that do not survive the run is finite, it is derived rather than argued, and someone can cost it in an afternoon. Same capability, and the two requests get very different answers in a planning conversation.
What I do not have is a home for it. It is not a debugging technique, it is a property of the design that stops being true whenever the design moves, so it belongs to whoever owns schema review rather than to whoever is on call. That is a real ask of a team, and I have watched better artefacts than this one die because nobody owned the step that kept them true.
Tokens rather than blanks is the smaller change. Identifiers already go out as replacements, so it moves fields between existing categories, and data classification usually has an owner already.
Where did you put it, and did it hold? I am more interested in what happened to it after the person who introduced it stopped watching than in what the first pass found.
The home problem is what decides whether any of this gets built, and I think the artefact's shape is what creates it. A checked-in list of deciding inputs that don't survive the run is a document, and documents need exactly the step you say nobody owns. So don't check the list in. Emit it. A pass over schema and code regenerates the set on every build and diffs it against the committed copy, so a change that moves the design fails on its own diff, at the moment it moves, attached to the change that moved it.
That turns it into something a schema-review owner can plausibly accept. They see a diff on the changes that produce one and nothing at all on the changes that don't. Nobody has to remember to keep it true.
Whether it holds turns on coverage. Staleness is the visible failure and the cheap one. The quiet one is the pass going blind: predicates fed by config, dispatch resolved at runtime, values arriving from a call the analysis can't follow, anything sitting behind a flag. Those get dropped rather than reported, and the set gets shorter on its own. A list that shrank because the pass stopped seeing looks exactly like a list that shrank because the design got simpler. So the artefact has to publish its own coverage next to the set, how many conditionals it could resolve out of how many it walked, plus names for the ones it couldn't. That count is the thing worth watching once nobody is watching.
Which makes me curious what your schema review receives today. A document to read, or a diff that can fail. If it's already the second one, a good part of the ownership problem was never yours.
A document to read. Which is why I framed ownership as the hard part, and it is only hard for artefacts that need someone to remember them.
Emitting it rather than checking it in is the right correction. The coverage count is the load-bearing half, though. A set that shrinks because the pass went blind and a set that shrinks because the design got simpler both produce a green build, and the second is what everyone assumes when the number drops. So the count is what gets watched, not the list.
The failure I would expect is not staleness. It is a diff firing on a config-driven predicate someone added on purpose, and the fix being to widen what the pass ignores. Same erosion, slower, with a build number on it.
Did the coverage count actually get watched, or did it become another green thing on a dashboard?
A count that is only reported turns into wallpaper. Nothing that carries no obligation survives contact with a busy quarter. Counts do not get watched. Gates do. The only version worth building is one where something refuses when the number moves the wrong way, because otherwise the lesson people learn is that green means someone else looked.
Your erosion case is the one I would worry about too, and it says the escape hatch matters more than the metric. So the pass should not have a category called ignore. When it cannot follow a predicate, it resolves conservatively: every input that predicate reads is assumed non-recoverable and lands on the must-log list. Blindness then fails toward work. Take your config-driven predicate. Under that rule the person who added it does not get to widen an exclusion and move on. They either give the pass something it can resolve, or they accept the logging their predicate just bought. Widening stops being the cheap fix, which was the incentive producing the slow version of the erosion.
Some exclusions are still legitimate. Those should be per-site waivers, one conditional named per waiver, committed next to the change that needed it. The cost of silencing then scales with how much gets silenced, and review sees a pile of line items instead of one quiet edit in a settings file.
For the count itself, a ratchet beats a threshold. Fail the build when coverage drops below the last accepted value. Lowering the floor becomes a rare and ugly edit with a name attached to it.
Where this is weak: ratchets get lowered under release pressure, and one lowering can flatten the thing for a year. There is also a cheaper check to run first. Who computes the denominator? If "conditionals walked" comes out of the same pass that may have gone blind, a pass that stops seeing a construct can stop counting it too, and coverage stays high while the truth underneath it drops. The denominator has to come from code that did not produce the numerator, a compiler-emitted control flow graph or a separate cheap parse. Otherwise the number is the analysis grading its own homework.
Rung four is where it finally stopped guessing. We ran into the same thing in extraction work: logs showed no errors on 95% of inputs, but the actual records caught that merged-cell tables were being silently skipped before the parser was ever called. We'd been chasing a parser config issue for half a day based on the logs. Turns out the parser wasn't being reached for those inputs at all.
95% clean would have kept me on the parser too. The logs were accurate, they were just accurate about a component that was never reached, and that is harder to catch than a log that is plainly wrong.
Half a day in, something sent you to the records. What was it?