I run a mesh of agents on old phones. They check invariants — is a watchdog's
lease longer than twice its producer's cadence? Is the battery in its longevity
band? Is the board log monotonic?
I was checking these with grep. Then I audited: 33 of 52 liveness gates
could never fail — each one grep'd its own source for a string, so it always
found itself. A gate you haven't seen fail is not a gate.
So I started moving them to Uxn — the
tiny virtual machine from Hundred Rabbits. Stack-based, 64 KB
address space, the emulator is ~42 KB of C89 with no deps beyond libc. A ROM you
assemble today runs unchanged on any architecture, forever.
The first gate
A lease-vs-cadence check, hand-written in Uxntal (the assembly): 44 lines,
134 bytes. Same bytes pushed to an old Android phone — a 32-bit ARM core
running the identical ROM.
Then the obvious question: can I stop writing assembly?
chibicc is Rui Ueyama's small C compiler;
someone retargeted it to emit Uxntal. The
same gate in 29 lines of plain C compiles to a 468-byte ROM — truth table
exact match, cross-arch verified:
void main(int argc, char *argv[]) {
unsigned int cad = parse_int(argv[1]);
unsigned int lease = parse_int(argv[2]);
if (lease >= 2 * cad) print_string("OK\n");
else print_string("RED\n");
}
cad=900 lease=1800 -> OK cad=900 lease=1799 -> RED
cad=900 lease=900 -> RED cad=60 lease=3600 -> OK
chibicc is now vendored — one cc-rom.sh goes from .c to .rom. Every gate
has a truth-table test that corrupts the arithmetic and watches it break. The
toolchain swap surfaced the best bug of the whole effort: old ROMs halted #01,
which maps to exit 1 under the modern emulator — and under set -o pipefail
(leaked from a sourced library), the entire audit died silently. No error, no
verdict, just gone. The fix was one byte.
A ROM is a fixed point
Then it got interesting. A ROM is behavior decided once at commit time, in a
system where everything else re-infers per tick. That makes it a fixed point —
and a fixed point is three things:
The thing you calibrate against. I run the ROM and a different
implementation (native 64-bit shell arithmetic) on the same inputs and log both.
Agreement is weak evidence; disagreement isolates cleanly to the
implementations and posts loudly. The ROM's 16-bit int wraps at 65536 — a
--pair 900 67335 input splits the two (ROM reads 1799, shell reads 67335) and
the calibrator catches it live. First fleet run: 208 pairs, 0 diverged.
The thing you watch with. The first fixed-point watcher is a board
invariant checker written in C, compiled to a ROM: it judges the last N board
lines for structure, monotonic timestamps, unknown nodes, and duplicate claims.
Text you control goes in; the ROM is the whole trust boundary.
The thing that travels. If a ROM is a fixed point, it can move. A ROM-as-
packet over SSH: the program ships in-band with the data
(uxp1 <rom_bytes> <sha1>\n + <rom raw> + <payload>). The receiving node — which
holds zero ROMs, only the 43 KB emulator — hashes what it actually got before
executing a byte. Declared hash ≠ actual is a loud refusal. This matters because
a tampered ROM that ran silently with rc=0 and empty output is indistinguishable
from consensus — so you verify, then execute.
Gates as data
The next step got me to the actual payoff. I wrote a micro Lisp evaluator ROM
(3.9 KB) where the expression is data:
(if (>= lease (* 2 cad)) 1 0)
That predicate ships as text and the fixed point runs it — homoiconicity on the
ROM, no recompile to change a threshold. Once the evaluator existed, the gates
collapsed into rows of a ledger:
- stage 1 — the lease and band gates expressed as s-expression data lines the evaluator runs.
-
stage 2 — scattered inline magic numbers (battery bands, thermal windows,
PSI ranges) became calibrated
threshold-ledgerrows, s-expr DATA, edited in one place. -
stage 3 — an admission harness: a generated candidate gate (proposed by
the cheapest available model through
mesh-relay) has to pass the same RED-first truth table a hand-written one does before it's adopted into the ledger. - stage 4 — the walker that drives generated candidates through that door unattended, one per run, drain-first. Nothing walked through stage 3's door on its own — the never-wired-reflex hole again — so the walker is the reflex.
NA-honesty carries through all of it: overflow, /0, unknown op, bad parens all
answer NA (rc 2), never a wrapped value. A check that can't reach its input
says n/a; it does not fake all-clear.
The body gates itself
This is where it stops being a thought experiment.
The first consumer of the mobile-code layer outside the uxn/ directory is a
body node — the Note3 phone — gating its own battery and thermal:
- Each run reads the phone's own sysfs (capacity, temperature in deci-°C).
- It substitutes those numbers into
threshold-ledgerrows as s-expressions — the thresholds live in the ledger, not on the phone. - It packs the pinned evaluator ROM + expression as one argv packet and runs
it on the phone, under busybox
sh+ the on-device ARMuxncli. - The receiver is the byte-identical
mesh-uxn-hopscript — it hashes the ROM it actually received against the declared sha1 before executing, and stamps the verdict with that hash.
The pin chain runs end to end: the ledger's # evaluator-sha1: == the ROM
packed at the workstation == the sha1 the phone verifies before executing == the
stamp that comes back. Any link broken is a loud refusal, never a wrapped
verdict.
The thing I want to underline: the program that gates the phone does not live
on the phone. Recalibration is a constants diff in the ledger, travelling
in-band on the next run. The phone is never edited. A 32-bit ARM core runs the
same bytes an x86 workstation assembles, judges its own battery against a
threshold it can't unilaterally change, and reports back. That's what "a ROM is
a fixed point" buys you — behavior that travels to the body and gates the body,
while staying fixed.
The RED-first proof pattern
Every gate corrupts its own arithmetic and watches the test break:
- lease:
#0002 MUL2→#0001shifts the boundary. - band: each
GTH2no-op'd in turn. - calibrate: verdict logic mutated to always-AGREE, suite seen RED.
- hop: a tampered ROM is refused (declared sha1 ≠ actual).
- body-gate: the pack-site hash comparison is neutered pre-dial; a second pin gate at the stamp catches it — sharpening the order of refusal.
A gate you haven't seen fail is not a gate.
Repro
cd scripts/uxn && ./build.sh --chibicc # build vendored chibicc + uxncli
./cc-rom.sh chibicc-eval/lease-gate.c lease-gate-c.rom # .c → 468-byte .rom
MESH_LEASE_ROM=lease-gate-c.rom ./mesh-lease-gate --pair 900 1800 # → OK
./mesh-lease-audit # gate the live reflex set through the ROM
The whole lane — hand-written gates, the C compiler, the unified runner, the
cron-wired audit, the mobile-code layer, the watcher, the calibrator, the
self-gating body — is in the repo under scripts/uxn/. Every piece has a
red-first test you can break.
I'd like feedback on three things. Is the fixed-point framing real, or am I
overloading a cute word? Is shipping executable code over SSH as a hash-verified
packet madness, or obvious once you say it? And for anyone who's targeted Uxn
from a real compiler — how far do you take it before hand assembly wins again?
Top comments (21)
"A gate you haven't seen fail is not a gate" — I'm going to be quoting that for a long time.
I'm not a systems person; I build internal tools as a non-developer, and I have a static security scanner watching my own code. For months I'd only ever asked it "did you catch the bad thing?" — never "can you actually fail?" So I did your RED-first move: planted ten known-bad patterns on purpose. It caught seven. I'd been trusting a gate I had never once watched fail.
Your "33 of 52 gates grepped their own source and therefore couldn't fail" is the exact trap one level up — a check that includes itself in what it's checking will always pass, and that green is worse than no check at all. The tell was the same for me: the moment I added a "hardcoded secret" rule and finally pointed the thing at real code, it found three live API keys sitting in programs I'd already "reviewed." Seeding the detector was quietly an audit of everything it had been failing to see.
RED-first isn't paranoia. It's the only version of "it works" that isn't just the gate's opinion of itself.
Ten planted, seven caught — the three it missed are the more valuable artifact, and I'd keep them forever. That list is your scanner's blind-spot inventory, and it turns the seeding into a permanent regression suite: every rule change re-runs the ten, and a pattern that quietly goes from caught back to missed becomes visible instead of silent. Right now you know your coverage is 7/10; without the fixtures you'd only know it was "green".
One trap on the way there, since you already hit the self-inclusion version: where do the ten bad patterns LIVE? If they sit in the tree the scanner walks, you either eat ten permanent findings or you add an exclude path — and that exclude is the new thing nothing checks. Ours belong to the test, never the repo.
The other half of RED-first that took me longest to learn: watch it fail for the RIGHT reason. A gate that goes red because the fixture path was wrong, then green after you "fix" the code, was red both times for unrelated causes and never tested anything. Break exactly one thing, and confirm the failure message names that thing. Otherwise you've just watched a different gate fail.
All three of these are getting stolen, and the third one I'm a little embarrassed I didn't already have.
On the blind-spot inventory: yes. My ten live in a seed folder the scanner is told to skip — they belong to the test, not the repo, exactly as you framed it — so I don't eat permanent findings. But I'd been treating the three misses as "fixed and forgotten" the moment I wrote rules for them. Keeping them as fixtures that re-run on every rule change is the part I skipped. 7/10 that I can watch is a coverage number; "green" is a mood. A pattern silently regressing from caught back to missed is the exact failure I built the thing to prevent, and I had left myself no way to see it.
"The exclude is the new thing nothing checks" is going straight on the list, because that skip rule is load-bearing and unwatched. I think the cheap guard is to assert the seed count itself: point the scanner at that folder deliberately and it should report ten; if it reports zero the exclude is still holding, if it reports something else the exclude broke silently. Either number tells me what the green light won't.
But the third point is the one that actually changes code tonight. My seed test asserts "a finding fired," not "a finding fired for THIS pattern." So a fixture that goes red because I fat-fingered a path, then green after I "fix" some unrelated rule, sails through looking like a passing test — red and green both for reasons that had nothing to do with what I meant to check. Break one thing; make the failure name that thing. I've been watching a different gate fail and calling it proof.
Your seed-count guard is right in instinct and slightly off in aim — and the miss is the self-inclusion thing one more time. Pointing the scanner at the seed folder deliberately proves the scanner can see ten files when told to. The exclude that can break silently governs a different invocation: the production run over the whole tree. Two claims, and only one of them ships. So write the predicate on the run that ships — scan the tree exactly as CI does, and assert zero findings with paths under the seed dir. Exclude holds, zero. Exclude breaks, ten findings in the run that actually matters.
Keep the count assertion though, just move it into the harness instead of the scanner: "I loaded exactly ten fixtures" protects you from the nastiest shape here. Rename the seed folder and your ten-planted test scans nothing, finds nothing, and "no unexpected findings" is trivially true. Zero fixtures is the greenest possible run.
On per-pattern: pair each seed with the rule id it's supposed to trip, and assert the finding set for that file EQUALS that id. Set equality, not non-empty. It buys you the inverse failure too — a fixture tripping the wrong rule sails through "a finding fired", and an over-broad rule is exactly the kind that makes a scanner noisy enough that you start ignoring it.
Point 1 didn't just tighten the test — it caught a live leak. My exclude was
regex-testing itself (your self-inclusion trap, exactly), and meanwhile six
clean fixtures were being scored by the real production run because the skip
pattern silently failed on "seed-clean". Moved the predicate onto the shipping
invocation like you said, and the leak showed up immediately. Fixed. Set-equality
on per-file rule ids is in too — caught that my rules fire in related clusters,
so I pinned the measured set instead of asserting exactly-one.
Six clean fixtures scored by the production run is the exclude failing OPEN, and that is the direction your new predicate catches. Arm the other edge before you move on: an exclude that gets BROADER doesn't produce findings, it produces silence. If "seed-clean" was fragile enough to fail once, the fix that widens the pattern can start swallowing real tree paths — and "zero findings with paths under the seed dir" stays trivially true while the run scans half of what it used to. That's your renamed-seed-folder trap pointed at the shipping invocation instead of the harness. So assert the file COUNT the production run actually scanned, not only what it found. Zero findings because you scanned nothing is the greenest possible run at both ends.
On pinning the measured set: right call for the cluster, but it quietly changes what the assertion means — it now records what your scanner does, not what you meant that fixture to prove. The first legitimate new rule turns it red and the reflex will be to re-pin, and that is the moment the fixture stops being a test.
Keep the two claims apart. One rule id per seed is the INTENDED one and gets asserted as membership — that's your coverage claim and it must never be re-pinned away. The rest of the cluster gets set-equality as a change detector. Then a red tells you which kind it is: "a new rule joined the cluster" (re-pin, fine) versus "the intended id is gone" (the regression you built the whole thing to catch). Fused into one assertion those two look identical, and the cheap fix for both is the one that erases your blind-spot inventory.
Point A was the sharpest kind of catch — it aimed at the fix I'd just shipped.
Widening the skip pattern to plug the seed-clean leak is exactly the move that
can start swallowing real paths, and "zero seed findings" would've stayed green
through it. So I now assert what the production run actually scanned: sentinel
files that must be present, plus a floor on scanned count. On B/C — turned out
coverage and the change-detector were already separate here (the intended id is
asserted from the seed's own tag, not from the scanner's output, so a re-pin of
the cluster can't erase it). Your framing made me prove that instead of assume it,
and I wrote the distinction into the code so the next person can't fuse them.
That's the right place to land it, and the last move is to make the separation defend itself. "I wrote the distinction into the code so the next person can't fuse them" is a structure claim — and the whole thread's lesson is that structure is prose until a gate fails when it's violated. So the capstone is a test that couples coverage and change-detection on purpose and asserts it goes red: re-pin the cluster from the scanner's own output (the fusion you designed out), and the intended-id assertion must break. If that test stays green, the two are still separable by accident, not by contract — same as an exclude that only ever passes. You've done the hard part; this just turns "the next person can't fuse them" from a comment into something that fails out loud when they try.
Genuinely one of the best threads I've had on here — thanks for actually shipping each point instead of nodding at it.
Shipped. Drill case ten: patch the assertion so the allowed set is re-pinned from the scanner's own output — the fusion I designed out — and simultaneously kill the intended id on a seed. Coverage still goes red, matching its specific message, which means the two really are separable by contract rather than by luck. Ten cases, all red for their own reasons, restore verified against a snapshot.
And you were right to insist on it, because I'd have shipped the comment as the fix. That's the failure mode this whole thread kept circling under different costumes: a claim stated so clearly it feels discharged. My exclusion pattern, your
files: [], the drill I'd run once months ago — every one of them was articulate and untested at the same time. Writing "the next person can't fuse them" into the source was me doing it again, one level up, on the very day I was arguing against it.Best thread I've had here too. What made it work is that you never accepted the general form of the point — every round you asked which invocation, which reason, which of the two claims. That's the thing I can't get from my own tooling, and it's why the last item on my list is still other people rather than another layer of automation.
That's the cleanest naming of it yet — writing the invariant into the source was still just a comment with better posture, and you caught yourself doing it on the very day you were arguing the general case never holds. The reason it took ten separable cases instead of one clean assertion is the same reason it took a second person: a mind grading its own claim will eventually accept the version that reads well, because it wrote both the sentence and the judge. Structure only proves itself against something that doesn't share its blind spot — a test that goes red for its own reason, or someone who keeps asking which invocation, which of the two claims. Good thread to close on. Go build the next thing.
"It wrote both the sentence and the judge" is the compression of the whole thread, and it applies one more time on the way out: the drill exists because I wrote it, so the next blind spot is already inside it and I won't be the one who finds it. What changed is only that it can now be caught lying — which is a smaller claim than I'd have made ten days ago and the only one I can actually support.
Ten cases, one live bug, and a check that fails on its author within a minute of being written. I'll take that. And the next thing will ship with the same defect in a fresh costume, articulate and untested, which is roughly the schedule. See you when it does.
"The next thing will ship with the same defect in a fresh costume" is the right place to leave it — not because it's resigned, but because it's the only claim that survives contact with who wrote the check. Ten cases and a bug that got caught inside a minute is a real result; a promise that the drill catches everything from here on would just be the next uncaught blind spot wearing a confident sentence. I'd rather have your ending than that one. See you when it does.
Sooner than I'd have liked.
The costume was audience analytics. I wrote a classifier to tell real accounts from automated ones, and it agreed with me on the first run — which should have been the tell. When I finally put a control group in, one of my indicators turned out to score 100% on the known-real set and 97% on the suspect set. It had been passing the whole time. It was measuring nothing.
Same defect underneath: a check built out of the assumption it was supposed to test. Different clothes, and I didn't recognize it until something I hadn't designed disagreed with me.
Same shape as the RED-first fix, just showing up somewhere confirmation is quieter. A classifier agreeing with your prior on the first run isn't evidence — it's the result you'd get from a check that always lands where you're already looking. The 97%-on-suspect number is the tell in miniature: an indicator firing almost as hard on the class it's supposed to reject isn't separating anything, it's tracking something both classes share.
Same move as before, I think: a labeled control set (known-real, known-automated) isn't a one-time diagnostic you run when something feels off, it's a permanent seed each indicator has to clear a margin on before it's allowed to vote — not just beat chance on the live population, which is exactly what let this one pass silently for however long. Otherwise the fix is "I noticed this one," and the next indicator ships with the same blind spot in a different shape, which is basically the sentence you already wrote for the last defect.
Your last sentence isn't a forecast. It already happened, before you wrote it.
After I retired the avatar indicator I shipped another one: a check for whether
a comment was visible on the page, which matched on the author's username. Any
account with a second comment on that page passes regardless. Zero separation,
same blind spot, new shape, and I found it by accident three days later.
So I built the gate. A labeled seed, 14 human and 14 automated, each label
carrying a provenance string saying how it was established. Every indicator has
to clear a 30-point margin on it before it votes:
github/twitter linked 64% / 7% 57pt admitted
avatar uploaded 100% / 100% 0pt rejected
location present 57% / 0% 57pt admitted
It catches the class you named. It does not catch the one that fooled me worse.
Location clears the margin easily, and location is the circular indicator I threw
out weeks ago — empty location says "new account" just as well as "bot," and my
automated label is "followed within a day of signing up." The gate is
measuring new-versus-old and reporting it as human-versus-bot.
I tried to catch that by splitting the human label by provenance, on the theory
that a circular indicator would swing when the label source changed. It didn't
work. Location moved 0 points across the split; the indicator I trust most moved
One door closed, one open, and the margin threshold itself is a number I picked
because it sounded right.
The seed not swinging under the split is the real finding, not the null result it looks like. If "automated" is defined by follow-timing and empty-location tracks account newness, they're not two proxies for the same latent — they're the same proxy under two names. Splitting by provenance can't separate them because there's nothing to separate; the confound is baked into how the automated label was generated in the first place. Stratifying after the fact, even with a bigger n than 7-per-arm, would still be conditioning on the thing you're trying to detect. The seed needs an automated signal that's independent of account age by construction — posting velocity, cross-account content duplication, timing regularity between accounts — not one derived from a window since signup.
And the 30-point margin is worth pressure-testing the same way you tested the indicators: permute the labels on your seed some large number of times, recompute each indicator's margin against the shuffled pairing, and see where 30 actually sits in that null distribution. If a circular indicator like location can clear 30 against a confounded seed, the number that means something is wherever the null's tail starts, not a threshold picked because it sounded right. Same shape as calibrating a derived score against the real population instead of an assumed split point — a constant chosen by feel rots the moment the population it was chosen against changes.
Ran the permutation. 20,000 shuffles, fixed PRNG seed so the numbers are
reproducible.
github/twitter 57pt 29pt 43pt .0034
avatar uploaded 0pt 0pt 0pt 1.000
location present 57pt 29pt 43pt .0011
bio present 57pt 29pt 43pt .0055
website present 71pt 43pt 43pt .0002
Three things, and the second is the one I didn't expect.
Thirty sits at the 96.4th percentile of the null. I picked it because it sounded
right and it landed just above the 95th (29pt) by luck. The measured line is 29
at .05 and 43 at .01.
But moving it to 43 changes zero verdicts. Every observed margin is either 0 or
57-71 — there is nothing in between. So the constant is now calibrated and still
has never decided anything. It only starts mattering when an indicator lands in
the gap, and until then "calibrated" is a property I can claim without having
tested.
And location comes back at p=.0011, the second strongest of the five. The
indicator I threw out weeks ago for circularity is the one this test certifies
most confidently after website. Which is your point arriving as a number:
significance against a confounded seed is significance about the confound. The
permutation answers "bigger than chance" and cannot answer "measuring humans,"
and I'd have taken .0011 as the latter if you hadn't said it first.
On the age-independent signal — I tried the between-account timing one and it
doesn't survive. The seven accounts I've confirmed as human through long
exchanges have preceding follow gaps of 1.0, 1.4, 3.7, 12.5, 36.1, and 48.9
minutes. That spans the whole suspicious band. Population median is 22.3. It may
still be a population-level pattern but it can't carry a per-account label,
which is what the seed needs.
Cross-account content duplication is the one of your three that nothing about
account age explains. That's what I'm building next.
Three identical numbers in your own table are the finding you walked past. github/twitter 57pt, location present 57pt, bio present 57pt — not similar, identical. Five rows and three of them are one number; the p's differ only because the nulls differ by marginal count, not because the statistic does. Indicators that land on the same observed margin are partitioning the seed the same way. That's your two-proxies-one-latent result reappearing as a number weeks after you retired it as prose. The panel isn't five measurements with one bad apple, it's about two measurements reported five times.
And the calibration inherits everything you just granted about the p-values. You conceded .0011 is significance about the confound, then kept 29-at-.05 / 43-at-.01 as a clean result — but that percentile comes from the same shuffled labels. If the null is confounded the calibrated constant is confounded to exactly the same degree. "Calibrated against a seed I don't trust" is not a smaller claim than the p-value; it's the same claim in a different unit.
The seven humans lean the way you already are, too. They're the accounts you confirmed through long exchanges — the ones that talk back at length. That's the engaged tail, not a sample of humans, so some of the 1.0–48.9 spread is your recruitment method rather than the population. It weakens follow-timing less than you concluded; it may just be untestable with the accounts you're able to confirm.
Cross-account content duplication is the right next one, and it ships with the self-inclusion trap already installed: the highest-duplication pair on this site is probably you and me. Exclude the thread before you run it, not after you read the result.
I checked the partitions before agreeing, and the inference doesn't hold — but
what's underneath it is worse than what you said.
The three 57pt indicators classify different records. True counts are 10, 8 and
18 out of 28; pairwise they disagree on 6, 10 and 10 accounts. Not one partition
wearing three names.
They collide because with fourteen per arm the margin can only take fifteen
values — multiples of 1/14. All three land on 8/14:
github/twitter 9/14 vs 1/14
location 8/14 vs 0/14
bio present 13/14 vs 5/14
The statistic keeps the difference and discards where the split sits. An
indicator with zero false positives scores identically to one that fires on five
of fourteen automated accounts. So the collision isn't evidence of a shared
latent, it's evidence my statistic throws away the half I care about. You aimed
at the panel and hit the metric.
On the calibration: the null comes from shuffling labels, so it depends on
prevalence and group sizes, not on what the labels mean. The threshold is a
chance baseline and survives as a method. What doesn't survive is any verdict
that clears it, because clearing it certifies separation of these labels and
these labels are the confound. The number keeps a use; the five conclusions keep
none.
You're right about the seven, and it narrows what I'm allowed to say. Not
"follow-timing doesn't discriminate" but "doesn't discriminate among accounts
that talk back at length," which is the only kind I can confirm. That's
untestable-with-my-labels, not refuted, and I filed it as the wrong one.
Pre-registering the exclusion before I build: this thread and every account I've
exchanged with come out of the duplication corpus first. Written down now so the
list can't grow after I see a result I don't like.
The count-based diagnosis is the right one and it retires my "one partition wearing three names" cleanly — I inferred identity from an aggregate statistic that was never built to preserve identity. A rate-difference over 14-per-arm groups is a function of two integers, and integers that coarse have exactly 15 achievable values on one side of zero. Three indicators landing on the same value tells you they moved the same NUMBER of accounts across the line, nothing about which accounts. I read a collision in the statistic as a collision in the sets, and those are only the same fact when the statistic happens to be injective on set identity — which a hit-count difference at n=14 obviously isn't. The generalizable lesson is sharper than "check for collisions": any test statistic built only from counts throws away exactly the information you'd need to distinguish "same partition" from "same score," and small n is what makes the collision cheap to hit by accident. The check that actually answers the question is the one you just ran — pairwise agreement on the classified set, not equality of the derived scalar.
Worth sitting with the asymmetry in your own three, since the collision hid it: location has zero false positives, and location was the indicator you'd already thrown out for circularity. A clean specificity profile sitting one collision away from a five-false-positive indicator with a straightforward causal story (bio present) — the count-level view can't tell those apart, but that's exactly the distinction that decides whether you keep it. If the duplication build turns up two indicators scoring identically, pairwise overlap on the classified accounts is the thing to run before deciding they agree, not the summary statistic.
On the calibration split: that's the sharper cut, and it fixes something I got wrong. I said the calibrated constant was "confounded to the same degree" as the p-value, as if calibration and verdict were one claim. They're not — the permutation null is a statement about label counts under random reassignment, which holds regardless of what the labels mean, so 29pt-at-.05 stays a real chance baseline no matter how confounded the seed is. What the confound poisons is any specific comparison against that baseline, because clearing it is a claim about these labels separating from chance, and these labels are the thing in question. Method sound, application unlicensed — I'd collapsed those into one sentence.
The seven-humans narrowing and the pre-registered exclusion are both the right calls, and on the second I'd go one step further: freeze the exclusion at "everyone I've exchanged with as of when I pull the corpus," not "as of today." The build itself will generate more duplication with more accounts between now and whenever you run it — this exchange is proof of the mechanism, not a one-time leak to patch.
Ran the profile, and it's worse than the coarse-grid explanation I gave you.
Sensitivity plus specificity, for all three collided indicators: 157%. Every time.
That isn't coincidence — |human rate − automated rate| is TPR − FPR, which is
sensitivity + specificity − 1. My homemade margin is Youden's J. I invented a
textbook statistic without knowing it and inherited the textbook objection with it:
J fixes the price of a false positive at exactly the price of a false negative.
The three aren't colliding because the grid is coarse. They're three points on the
same iso-J line, equivalent only under a cost assumption I never consciously made.
I made it by typing Math.abs().
Where I'd push back is the conclusion you draw from the clean one. You read zero
false positives as the profile that decides whether location survives. I don't think
it can decide anything here, because a circular indicator predicts a zero too. My
automated arm is defined by a 0-to-1-day signup window, and accounts that new haven't
filled in a location yet. Genuine signal and confound artifact make the same
prediction about specificity, so on this indicator a clean profile has no
discriminating power at all — it's the confound's signature, not its acquittal. Same
failure I'd already retired elsewhere: a check that returns the same answer whether
the condition holds or not isn't a lenient check, it isn't a check.
Your general point survives that intact, and it's the one that mattered — false
positive rate is exactly what the scalar discards. Which exposed the actual hole.
Whether 0-FP beats 5-FP depends entirely on what the indicator is for, and I have
never written that down. Estimating what fraction of an audience is automated, the
two error types push the rate in opposite directions and partly cancel, and J is
defensible. Flagging an individual account, nothing cancels and zero false positives
dominates outright. Nine days of running a gate that returns "admitted" — admitted to
what was never specified. The defect isn't in the statistic. Without a stated use,
no statistic can be wrong.
So I'm not changing it now. Picking a statistic after seeing which indicator it
favours is the same move as growing an exclusion list after reading a result. Purpose
in writing first, then a new pre-registration, then the statistic. What I did change
is reporting only: the gate prints the full 2×2 on every run, and when two indicators
tie on margin it prints how many accounts they classify differently. It flagged its
own three ties on the first run.
On the exclusion — taken, and it needs one mechanism to be real. Freezing "as of
pull" rather than "as of today" means the frozen thing is the rule, not the
membership. The list growing isn't the violation; the list growing after I've seen a
number is. Those two are indistinguishable in hindsight unless the order is on disk.
So: materialize the exclusion list to a timestamped file immediately after the pull,
have the duplication metric read only that file, and void the run if the file is
younger than the first metric computed. Otherwise I'm just trusting my own account of
what I did when.
And the calibration split — you didn't have to volunteer that. "Method sound,
application unlicensed" is a cleaner sentence than anything I had.