I have been building with coding agents daily for a while now. They are good. The
problem I keep running into is not that they write bad code — it is that they
change things faster than I can produce evidence of what changed relative to what
I had already accepted.
This is a different problem from correctness, and I think it is being
under-discussed.
Tests answer a different question
When an agent hands back a change, the reflex is to run the test suite. If it
passes, ship it.
Tests answer: does the code do what the tests assert?
They do not answer: what moved, relative to the state I reviewed and accepted?
Those come apart in ordinary situations. An agent refactors three files while
fixing one. A generated migration rewrites a fixture you were treating as a
frozen reference. A tool run mutates a config file nobody was watching. Every
test still passes, because no test was ever written about the thing that moved.
You find out later, or you do not find out.
The gap widens as agents get more capable. A weak agent touches one function. A
strong agent touches whatever it decides is in scope — and "whatever it decides"
is precisely the part you cannot pin down in advance, because that
non-determinism is where the value comes from.
The verification layer should be orthogonal to the agent
The tempting fix is to make the agent verify itself: ask it to summarize its own
changes, or run a second agent as reviewer.
I do not think this works as an acceptance boundary, for a structural reason:
both the change and the verification then come from the same probabilistic
process. When they disagree you learn something. When they agree you have learned
almost nothing, because agreement is exactly what a shared failure mode produces.
What I want is a verification layer with the opposite properties:
- Deterministic. Same bounded input, same finding, every time.
- Model-neutral. Swapping the agent should not change what acceptance means.
- Offline. No network call in the verification path — a verification step that can fail for network reasons is not a boundary.
- Machine-consumable. A stable exit code and a structured finding, not prose a human has to interpret.
Note that none of this makes the agent less useful. It stays as flexible and
probabilistic as you like. The boundary is drawn at acceptance, not at
generation.
Freeze, change, verify
The concrete mechanism I settled on is boring, which I take as a good sign.
Freeze a bounded set of sources into a hash-addressed manifest. Let the agent do
whatever it does. Verify against the manifest.
$ assurance corpus freeze ./src --manifest baseline.jsonl
result=PASS
exit_code=0
manifest=baseline.jsonl
write_disposition=CREATED
source_record_count=1
# ... agent runs, edits a file ...
$ assurance corpus verify baseline.jsonl
result=HOLD
exit_code=4
counts={"changed": 1, "match": 0, "missing": 0, "self_ingested": 0, "type_changed": 0}
FINDING {"code":"CI03_SOURCE_CHANGED","severity":"ERROR","message":"filesystem
source bytes changed","path":".../src/greeting.py","location":"source_record", ...}
(Real output, with a few header lines — module id, rule-set version, profile —
elided for width, and the absolute path shortened.)
The two things that matter here are the ones that look least interesting.
The exit code is part of the contract. 4 means an integrity finding
specifically, not "something went wrong." A CI job can branch on it without
parsing anything. Once exit codes are contractual they have to be versioned and
tested like any other public interface, which is a constraint worth accepting
early rather than discovering later.
"Bounded" is doing real work. The manifest covers explicitly supplied roots.
Not the whole machine, not an implicit working directory. An unbounded integrity
check is one that eventually gets disabled because it is too noisy, and a
disabled check is worse than no check because you still believe it is running.
A fair objection at this point: for a clean Git repository, git diff covers a
substantial part of the ordinary file-change case. It genuinely does. The cases
where it differs are when the bounded evidence set is not identical to the
repository — several explicit roots at once, deliberately untracked files,
members inside ZIP archives (read and hashed individually), symlink identity
(the link target path itself is recorded and hashed, which is a different fact
from the target file's contents), and the machine-consumable exit semantics
above. If your evidence set is exactly "tracked files in one repo," use
git diff. It is right there and it is excellent.
Authorization must be out-of-band — a lesson I learned by getting it wrong
This is the part I would most like to pass on, because I got it wrong in public
and the failure mode generalizes well beyond my own project.
Suppose your verification layer checks that a mutation was authorized: the record
claims a decision authorized it, and the tool confirms the decision exists,
covers the same object, and was made by the right authority.
The question is: where does "the right authority" come from?
My first public implementation had the authority identity hard-coded as a literal
string in the validation logic. It worked perfectly — for exactly one person.
Every other user constructing a fully well-formed record got a HOLD, because
their authority identity was not the one baked into the source. Three of the
tool's modules were structurally unusable by anyone but me.
I did not notice, and my test suite could not have told me, because all my
fixtures used the same identity as the code. It surfaced during an independent
adversarial review of the repository, and it surfaced only because the reviewer
built two byte-identical inputs differing in exactly one field and observed that
one passed and one did not. Reading the source had not found it; a green suite
had not found it. A single-tenant constant hiding in validation logic is
invisible from inside your own tests, because your fixtures share the constant.
The obvious repair is to let the input document declare its own authority. This
is worse. If the record under verification names the authority that will be
accepted, then a record can authorize itself:
{
"authority_identity": "WHOEVER_I_SAY",
"decisions": [{ "decider": "WHOEVER_I_SAY", "state": "AUTHORIZED" }]
}
The validator dutifully confirms the two agree, and the check has become
decorative. This is the same shape as a certificate that vouches for its own
issuer.
The repair that actually holds is to take the expected authority out of band —
supplied by the caller, at the call site, never readable from the artifact being
checked:
$ assurance check pack.json --authority-id PROJECT_AUTHORITY → PASS, exit 0
$ assurance check pack.json --authority-id SOMEONE_ELSE → HOLD, exit 3
$ assurance check pack.json → HOLD, exit 3
(fail-closed)
and, importantly:
# pack declares its own authority_identity, no --authority-id given
$ assurance check self-declaring-pack.json → HOLD, exit 3
The last two lines are the ones worth arguing about. Missing expected authority
is a HOLD, not a pass-through — an authorization check with no expected authority
has nothing to check against, and defaulting to permissive is how these things
quietly stop working. And a self-declared authority never overrides the
out-of-band value, even when it happens to agree with it.
The transferable lesson is narrow and worth stating plainly: trust anchors do
not belong inside the artifact being verified. The second-order version is the
one that nearly caught me — the naive fix for a coupling problem introduced a
self-authorization hole, and it looked like a clean generalization while doing
it.
The layer should refuse to make the decision
The last design constraint is the one people push back on most, so I will state
it plainly: this kind of tool should not decide whether to accept a change.
Concretely, in mine, risk classification returns a tier and an explicit field
saying the classification is not an authorization. Handoff validation reports
structural observations and explicitly reports that receiver readiness was not
machine-determined. Those fields are not decoration; they exist so that no
downstream automation can quietly read a PASS as a go-ahead.
The reason is not modesty about what software can do. It is that the moment a
deterministic checker is treated as an approval authority, people start shaping
inputs to satisfy it, and you have rebuilt the thing you were trying to avoid —
a probabilistic process optimizing against a proxy. Keeping the tool
descriptive, and keeping acceptance with a person, is what preserves the
boundary's meaning.
Where this leaves things
I do not think "assurance for AI-assisted engineering" is a solved problem, or
that a manifest checker is the whole answer. What I am fairly confident about is
the shape:
probabilistic generation → deterministic verification → human acceptance
with each stage refusing to do the next one's job. Agents stay flexible.
Verification stays reproducible and inspectable. Acceptance stays with someone
accountable.
It is explicitly not a replacement for Git, for tests, for CI, or for human
review. It sits beside all four.
I built FABLE5 as one
implementation of this shape — a local CLI, Python 3.11+ standard library only,
no network calls, no daemon, no model invocation, Apache-2.0. It is early: a
0.3.0 prerelease with 276 tests and CI across Python 3.11–3.14, maintained by
one person. There is a self-contained runnable example that walks the whole
freeze → change → detect → re-freeze cycle in a disposable temp directory in
about two seconds.
I would rather have the architecture argued with than the tool adopted. If you
think the acceptance boundary belongs somewhere else, or that this is a problem
existing CI already handles, I would genuinely like to hear it.
Top comments (8)
Exit codes as a versioned public interface is the line I would put in front of people, because it is the part teams skip and then rediscover the first time a CI job branches on "nonzero" and swallows a real finding. Two things I would push on. First, bounded cuts both ways. A manifest over explicitly supplied roots is honest, and it has the same blind spot as a test suite: it covers what you remembered. I would want the manifest to record what it did not cover, so the gap lives in the artifact rather than in somebody's memory. Second, I think you concede too much to git diff. Your mechanism is a claim about a state you reviewed and accepted, not about a commit, and those two come apart precisely when the agent is the thing making commits.
Had the same issue. I'd deployed an agent that was quietly rewriting Neo4j edges during retrieval passes, and tests passed because the fixtures were pre-baked snapshots rather than live reads. It's exactly the correctness-vs-drift gap you're naming. The bounded scope part matters a lot; in my case it crept from "just the src/ dir" to "everything except node_modules" within two sprints, at which point the manifest was catching basically nothing.
You're right that this is the part that matters, and I got it wrong on the first pass — I tested the adjacent direction and called it a reproduction. Worth correcting in public.
What I'd checked was widening the exclusion list and watching coverage shrink silently. Real gap, wrong failure. Yours is the opposite: the covered set grew. So I ran that instead.
A small synthetic reproduction:
A root = src/ 2 files
B root = project/, exclude node_modules 32 files
one real edit under src/, plus a normal rebuild:
A verify → 1 finding (the one you'd want)
B verify → 31 findings (30 of them build/cache churn)
So "catching basically nothing" is right operationally, but the mechanism isn't that detection stopped. It detects harder than ever. B is a wall of legitimate findings with the one that matters buried at a signal-to-noise ratio of 1:30. Both runs fail; only one is readable. A check that fires on routine churn quickly becomes background noise. The failure mode is different from silence, but the operational result can converge on the same thing: nobody reads it.
That's why I think widening scope has to be treated as a change to the contract rather than just an edit to a config line. Bounded scope isn't a performance knob; it's part of what gives a finding meaning. Widen it and you haven't merely loosened the check — you've asked it a different question.
For what it's worth, on what actually holds that line, in decreasing order of how much I trust it:
The mechanical version is to make widening produce an artifact rather than be an edit. Freeze here is no-clobber, so a wider scope can't overwrite the baseline in place — you end up with two manifests whose headers each state their own roots and exclusions. That's half a ratchet by accident; the half that's missing is anything that consumes the pair. On your side that doesn't require changing the verifier: keep the previous manifest and require the header diff on the PR that widens scope. The point isn't the diff, it's that the widening now has to be addressed by someone instead of succeeding silently.
The version I'd trust least is the same rule written down as a principle. I've watched that fail on my own work — four rules written for one project, three of them violated within days, and the only one that held was the one I happened to still be watching. That experience changed how I classify rules: if nothing mechanically observes the invariant, I treat the rule as documentation rather than as a control. A paragraph in a README can explain a boundary; I wouldn't count it as enforcing one.
In between, and cheap: don't ask the config whether the check still works, ask the check. Inject a known meaningful mutation and confirm it still emerges clearly above the routine churn. That doesn't prove the checker is complete — it proves something narrower and operationally useful, that a failure you care about is still visible under the scope you have actually configured.
If you want a number rather than a judgement call, the cheapest one I've found is the noise floor: run verify after routine churn with no meaningful change, and count what comes back. Same three-step creep, measured:
just src/ 2 files noise floor 0
Each step is locally reasonable, and the floor moves on the first run after it rather than two sprints later. A non-zero floor means every real finding now has to be distinguished from routine churn before the result is actionable.
Which is the other thing your comment turned up. I had assumed scope worked like the authority identity: supplied independently at the call site and not taken from the artifact being checked. That's true at freeze time. At verify time there is no caller-supplied expected-scope input: the source records come from the manifest, exclusions come from its header, and --detect-new also uses the header's roots to define what it scans. So the baseline carries its own verification scope. That's weaker than the boundary I thought I had.
Related and smaller: there's a finding named CI07_UNBOUNDED_ROOT that sounds as if it guards breadth, and it doesn't. It fires for an empty root list or for a root that isn't an existing non-symlink directory. CI07 itself accepts an existing directory regardless of how broad it is.
On the Neo4j half, PM-04 has a more basic limitation: its source types are filesystem files, symlinks and ZIP members. There's no graph-aware source, so it has no direct way to observe a live edge mutation as graph state. At most it could notice byte drift if that mutation surfaced in files that had deliberately been included in the corpus.
Was the widening one decision in your case, or accretion — a run of "just add this one directory" changes? Would a scope-diff review gate have helped, or did each incremental widening look locally reasonable enough that it would have passed review anyway?
Strong separation of generation, verification, and acceptance. I’d add one more boundary: byte-level integrity and semantic admissibility should produce different findings. In a research pipeline, a source can be byte-identical yet stale for the current as-of date, or legitimately changed because an official filing was amended. I would freeze evidence metadata alongside the corpus—source URI, retrieval timestamp, effective date, parser version, and schema—and return separate integrity, freshness, and lineage statuses. That lets the human reviewer distinguish unauthorized mutation from expected evidence evolution. Have you considered versioning a policy manifest alongside the corpus manifest so permitted roots, freshness windows, and parser versions are reviewed rather than embedded in the verifier?
This is the same structural lesson as the authority bug, one level up.
Freshness windows and parser versions are policy, and policy hard-coded in a
verifier is invisible from inside your own tests for exactly the reason a
hard-coded authority identity is. It also lands on a boundary the repo declares
rather than hides — docs/LIMITATIONS_AND_FUTURE_SEAMS.md opens with "Hashes
prove byte identity, not semantic truth or provenance."
Concretely, the corpus module (PM-04) answers both of your cases correctly and
helps the reviewer in neither, in opposite directions. A source that is
byte-identical but stale verifies clean — correct about the bytes, silent about
admissibility. A legitimately amended filing raises
CI03_SOURCE_CHANGED—also correct about the bytes, and equally silent about whether that change was
expected. The finding vocabulary has one axis, so nothing in the output
distinguishes unauthorized mutation from expected evidence evolution.
Partial correction, since some of what you're asking for is already there:
permitted roots and exclusions are frozen into the manifest header alongside
the rule-set version, so bounded scope is reviewable as data, not embedded in
the verifier. What's entirely absent is anything time-shaped — source records
carry root, relative path, size, and sha256, with no retrieval timestamp, no
effective date, no notion of parser or source schema. (
schema_versionexists,but it versions the manifest format itself.) So the policy manifest you
describe is skeletal for scope and absent for freshness and lineage.
The boundary I'd want kept sharp is that byte integrity is intrinsic to the
frozen evidence, while freshness is contextual — "stale for the current as-of
date" is a question about something outside the evidence set. That doesn't make
it non-deterministic; it makes the context an input. The check stays
reproducible only if the as-of date is an explicit call-site parameter, never
read from the artifact — same rule as
--authority-id. And note the recursionyour proposal implies: the policy manifest then needs its own trust anchor,
also supplied out of band, or the self-authorizing artifact has just been
rebuilt one level up.
Given that input, the three-way split looks right, and part of it already fits
the existing model: freeze is no-clobber by construction — writing different
bytes to an existing manifest path is a collision finding, not an overwrite —
so an amended filing has to become a new manifest artifact rather than mutate
the old one. What's missing is that the tool reports the drift as an integrity
event with no vocabulary for calling it a lineage one.
That's a larger change than adding fields, so I'd rather think it through than
bolt a timestamp onto a source record. But integrity / freshness / lineage as
separate finding families rather than one validity bit — yes.
Curious how you handle the as-of date today, if you have a pipeline doing this:
pinned per run, or derived from the filings themselves? That choice seems to
determine most of the rest.
Pinned per run. I treat evidence time and execution time as separate fields, with an explicit as_of timestamp supplied by the run context. Filings contribute their own filing, effective, and amendment dates; they never silently set run time. The acceptance layer evaluates freshness relative to the pinned as_of, so the same evidence snapshot, policy version, and as_of reproduces the same statuses. For amendments, I preserve both artifacts and link predecessor/successor lineage: the newer filing may supersede applicability without erasing the older byte-integrity record. The trust-anchor recursion remains, so the policy manifest or signature must come from a separate approval boundary.
We tried this exact pattern first: second LLM pass as reviewer on our entity linker decisions, metrics looked promising on the test set. Didn't catch the real problem for six weeks. The merge errors that were systematic just agreed with each other - the romanization blindspots that tripped up pass one weren't visible to pass two at all. We had to take the LLM out of the accept/reject gate entirely and replace it with a hardcoded structural check.
Six weeks is the part I'd want to sit with. The agreement wasn't just failing to catch the error — on that failure class, it was also depriving you of the disagreements that might have made the pattern visible sooner.
The way I'd put your result is that a second pass is useful as an independent boundary only to the extent that its failure modes differ from the first on the cases that matter. "Different model" is at best a proxy for that. Independence can come from different evidence, but also from a different representation, rule set, or decision procedure. In your case the two passes shared the romanization blind spot, so their agreement carried very little independent information exactly where you needed it most.
That's the part I think is underrated. On a shared systematic failure class, a correlated reviewer doesn't fail neutrally: it produces fewer disagreements, which means fewer cases get surfaced for inspection, which means it can take longer to accumulate enough examples to name the failure. Random errors tend to create visible disagreement. Systematic shared ones can come back looking like consensus.
That also changes how I'd evaluate a reviewer. A 99% agreement rate by itself tells me almost nothing about whether the second pass is useful. What I care about is its marginal error detection: which failures does it surface that the first pass would otherwise miss, and are those disagreements cheap enough to inspect? Aggregate accuracy still matters, but agreement is not independence.
The clearest case I've had of both halves was in the toolkit itself. It shipped with an authorization check tied to a maintainer-specific identity, so the public contract did not generalize to an external maintainer. Reading the code, I then got the trigger condition wrong — confidently wrong, because the authorization check is only reached for an action that is both executed and mutating.
What settled it was a synthetic differential: governance packs held constant except for the deciding identity, exercised against explicit expected-authority inputs. The useful part wasn't another interpretation of the code; it was constructing cases where the expected distinction was known in advance, then asking the implementation to make it.
The fix exposed the other half of the problem. My first instinct was to let the input document declare its own authority. That would have replaced a hardcoded identity with a self-authorizing artifact. The boundary only became meaningful when the expected authority moved out of band, as a caller-supplied parameter. What made the review useful wasn't agreement with my diagnosis; it was disagreement with my proposed fix, for a reason that could be checked.
On the structural check you ended up with, I'd keep the claim narrow. It isn't a correctness oracle. It can only encode an invariant someone already knows how to state. Your romanization invariant was apparently expressible after the failure class had become visible; the deterministic gate didn't discover that knowledge. What it gives you after that point is reproducibility — the same bounded inputs exercise the same explicit rule, instead of asking another model to rediscover the judgment.
So the practical version, for me: don't ask whether a second pass is "independent" in the abstract. Test whether it adds signal on the failure classes you actually care about. Aggregate metrics on a held-out set can still hide a shared blind spot if that class is underrepresented, unsliced, or averaged away. Seeding known instances of a specific failure class gives you something narrower but cleaner — ground truth by construction, for that class.
The obvious limit is that you can only seed failures you've already named, which is exactly the loop your six-week story exposes. So for unknown failure classes the useful property may be less "the reviewer is usually right" than "the disagreements it creates are cheap enough to inspect that the rare useful ones survive triage."
What finally made the romanization failure expressible after six weeks — had enough examples accumulated that the invariant became obvious, or did one person notice a pattern the review process had never surfaced? Those are pretty different failure modes.