The Bug Was in the Brief, Upstream of Both Reviews

Originally published on hexisteme notes.

I write these essays with a lot of delegation. I draft a brief describing what happened and what the essay should argue, hand it to a writer instance, and then hand the draft to an independent reviewer instance whose only job is to compare the draft against the source material I pointed it at. That pipeline has worked well enough, often enough, that I mostly stopped worrying about it. Then I wrote a brief with four confident factual claims in it, and all four turned out to be wrong — and the reviewer caught exactly two.

The essay in question was about a scheduled background job that silently failed while its own status output kept reporting green (a different story, told elsewhere — I won't retell it here). What matters for this piece isn't the job or the bug. It's the four sentences I put in the brief describing how bad the failure had been, and what happened when I went back and actually measured them.

Four claims, one brief, one blind spot

Here's what I told the writer, and what the primary data said once I checked it properly.

The writer copied all four claims faithfully into the draft. That's the writer doing its job correctly — a writer's task is to turn a brief into prose, not to audit the brief.

The reviewer's job was different: compare the draft against the source document I'd pointed it at, and flag anything the draft says that the source doesn't support. It reported four blockers. Two of them were the claims in the table above marked "yes" — the ones where the source document actually described events differently than the brief claimed. The other two blockers were things the reviewer found on its own, unrelated to my brief: a missing field in a code snippet, and a count of failing tests that was off by two. All four of those catches were correct, and I fixed all four.

But notice what's missing. The reviewer's four blockers and the brief's four wrong claims overlap in exactly two places. The two claims about frequency and age — "every night," "weeks old" — sailed through untouched. Not because the reviewer was careless. Because the source document the reviewer was checking against never said how many times the job had failed, or how old that database row was. It was silent on both counts. And a draft that repeats a claim the source is silent about does not contradict the source. It matches it — vacuously, but it matches.

Why a compare-to-source reviewer can't catch this

A review built as "does the draft say anything the source doesn't support" is a refutation engine. It's very good at catching claims the source actively disputes — dates that don't line up, a sequence of events told out of order, a fact the source states differently. It has no leverage at all over claims the source simply never addresses, because there's nothing there to disagree with. Silence isn't evidence against a claim; it's the absence of evidence either way. A reviewer running a pure compare-to-source check treats absence of contradiction as a pass, which is the same thing as treating silence as agreement.

The part that makes this worth writing about is where the wrong claims came from. Not the writer, who copied faithfully. Not the reviewer, who compared faithfully. Both of them received the same four claims from the same single document — the brief — and had every reason to trust it, because that's what a brief is for. The brief is the one point in the pipeline that supplies both the writer and the reviewer with the same factual premises, before either of them does any independent checking of its own. If a wrong number gets into the brief, it doesn't hit one gate and get caught. It hits both gates, because both gates were built to trust the same upstream source.

What eventually caught the two missed claims wasn't a smarter reviewer. It was going back to the raw material directly — reading the complete log file rather than trusting a remembered impression of "seems like it fails a lot," and pulling every row from the relevant database table in order rather than assuming an age. Once I had those numbers in hand, the two false claims collapsed immediately, and I had to correct the essay's title, its URL slug, its summary, and both of its structured-data blocks, because the wrong numbers had already spread into every one of those surfaces.

Verification isn't uniformly blind — this is a narrow gap

I want to be careful not to overclaim here, because I've watched the same review method work in other cases, and I don't want this piece to read as "compare-to-source review is useless." In one case, the same kind of reviewer caught a place where a draft described an event as something unrelated when the source had actually described it happening as part of the same ongoing task — that's a contradiction, not a silence, and the reviewer caught it as a blocker. In another case, a reviewer applying a strict rule — every descriptive sentence must trace back to something the source actually says, no exceptions — deleted three sentences describing a person's reputation that the source had simply never mentioned. Not contradicted. Just absent. The reviewer caught it anyway, because it was enforcing "must be supported" rather than "must not be contradicted," and those are different rules that happen to look identical on the claims that get explicitly disputed.

That distinction is the whole story. The blind spot doesn't open because compare-to-source review is weak in general. It opens specifically on quantitative and existence claims — counts, durations, "this already exists," "this already passes" — and specifically when the review is checking for contradiction rather than requiring support. A related case from the same pipeline backs this up from a different angle: a brief once instructed the writer to violate an established formatting convention, and the automated gate that checks drafts before publishing simply had no rule covering that convention at all — the gate's checklist was silent, not the source, but the shape of the failure is the same. A brief-originated error passes any check whose scope doesn't happen to cover it.

What I changed

Three things, none of them exotic.

First, before a number, a count, or a duration goes into a brief, I confirm it against the primary material directly — the full log file with an actual count, not a remembered impression; a full, ordered dump of the relevant table rows, not a guess at recency; a direct listing of what test files actually exist, not an assumption that surely something covers it. Memory and a plausible-sounding estimate are exactly what put the wrong claims in the brief in the first place.

Second, every quantitative claim in a brief now gets a citation — the exact command or file location that backs it up. A claim with no citation either doesn't go in, or goes in explicitly marked unconfirmed. Counting how many claims lack a citation is itself useful information; a brief where every number traces to a command is a very different artifact from one where half of them are "I'm pretty sure."

Third, and this is the part that's specific to using an independent reviewer at all: I stopped assuming that "compare it to the source" was a complete instruction. Now the reviewer gets an explicit list of which claims the source doesn't confirm or deny, with instructions to either re-verify them directly against primary material and show its work — the actual command and its actual output, not a sentence claiming to have checked — or mark them "source silent, unverified" and report those separately from genuine blockers. A reviewer's report of "N blockers, everything else is fine" now gets read as a checklist to reconcile against the brief's claim list, not as a certificate that the facts have been settled. In this case the two sets — the reviewer's four blockers and the brief's four wrong claims — only overlapped by half. That gap is exactly what "N blockers, rest is clean" hides if you don't go looking for it.

The pattern outside writing pipelines

None of this needs an AI writer or an AI reviewer to happen. Any workflow where one document seeds two downstream checks that are supposed to be independent has the same shape. A bug ticket that says "reproduces every time" feeds both the engineer who writes the fix and the regression test that's supposed to confirm the fix later — if the frequency claim was wrong, the test has no basis to disagree, because nobody built the test to question the ticket's premise. A design doc that states "this endpoint has three callers" feeds both the migration plan and the reviewer checking the migration plan against that same doc — if the real number is five, the reviewer isn't checking a fact, it's checking consistency with a document that was wrong from the start. The fix in both cases is the one I ended up with: pull the count from the system directly before it becomes a written premise, and make sure whoever reviews the downstream work has a way to say "the spec doesn't actually establish this" instead of just "the spec and the draft agree."

More notes at hexisteme.github.io/notes.