I built a Playwright gate that cried wolf 889 times

On August 2nd I shipped a landing page whose headline was in the DOM at the right size, the right colour and the right position, and invisible. StoreReady's data-animate="17g0wbg" resolved against motion data from a different capture, so the h1 held its start frame, opacity 0.001, permanently. Every gate I had passed the build.

They passed because they all read source. icons-gate greps the icon set, mark-gate greps the lockup, workbench check hashes the vendored files. All of them are decidable without a renderer, and there is no string to grep for a pixel that never got painted.

So I wrote one that never reads a file. ops/qa/render-gate.mjs in the workbench repo takes a URL, drives real Chrome, and asks six questions a screenshot answers: effective opacity down the whole ancestor chain, WCAG against the computed background, horizontal scroll from 320 to 1920, control labels breaking to a second line, headline and CTA above the fold at 1280x800, and content trapped past the start edge of its own scroll container. Each product's scripts/deploy.sh runs it after the deploy, against the live host, and a failure is hard.

The first sweep reported 889 invisible elements

Essentially all of them were scroll-reveal sections below the fold, sitting at opacity 0 because nothing had asked them to appear yet. A gate that cries wolf 889 times is not a gate.

The part I want to keep from that run is why I did not see it coming. My own shell's entrance runtime reveals on mount rather than on intersection, so the sites already on it scored honestly by accident, which is the worst way to be right. Pointing the same measurement at pages with intersection-driven reveals is what exposed the model.

The fix is that the gate judges only what is on screen. It walks the page, probes at each stop, and merges by element plus text keeping the best reading of each one. An element off-screen at opacity 0 is not a defect. An element in front of the reader at opacity 0 is the StoreReady bug. One scroll position cannot tell those apart.

Per-stop settle is 750ms, long enough for a reveal to finish rather than merely to start. At 320ms it kept catching FetchDue's sections at opacity 0.061, mid-fade, which reads as a defect and is not one.

Two ways the walk still lied

The first was arithmetic. The stops stepped a fixed 0.75 viewport down from the top and then took .slice(0, 14) for the time budget, so on anything taller than about 9,450px at a 900px viewport the sweep just stopped and everything below was never judged at all.

const h = Math.max(document.body.scrollHeight, document.documentElement.scrollHeight);
const MAX = 14; // 14 stops at 750ms is the time budget
const step = Math.max(Math.round(innerHeight * 0.75), Math.ceil(h / MAX));
const out = [];
for (let y = 0; y < h; y += step) out.push(y);

Spreading the same budget over the real height means density degrades on a very long page instead of coverage vanishing. A missed defect in the last third is worse than a coarser walk.

The second is the interesting one. On a long page an element can be in front of the reader at exactly one stop. If that stop catches it entering at the bottom edge, or partway through a stagger, then the best-reading merge has only the bad reading to keep. The finding looks real, reproduces across runs, and describes nothing anyone would ever see. On QuorumFile that condemned three cells of a seventeen-row table, all three at opacity 1 the moment you scroll to them.

Look at it again, on purpose

Anything the walk condemns now gets one more measurement, taken deliberately.

const condemned = text.filter((t) => t.opacity < OPACITY_FLOOR && !t.decorative && !t.clipped);
for (const t of condemned) {
  await page.evaluate((top) => scrollTo(0, Math.max(0, top - innerHeight / 2)), t.top);
  await page.waitForTimeout(1400);
  const fresh = await page.evaluate(probeText);
  const better = fresh.find((f) => f.sel === t.sel && f.text === t.text && f.inFold);
  if (better && better.opacity > t.opacity) Object.assign(t, better, { pxRatio: undefined });
}

OPACITY_FLOOR is 0.08. Two details in there cost me something. Only condemned runs are re-measured, so a clean page pays nothing for this, which is what makes a 1400ms settle affordable at all. And the fresh reading replaces the stale one whole, rather than just its opacity. A reading taken while an element was at opacity 0 also carries the contrast it had against whatever was behind it, which is 1:1 because no ink was painted. Adopting the opacity alone clears the visibility finding and hands that dead 1:1 straight to the contrast check. One false finding becomes a different false finding.

A finding that survives being scrolled to and stared at for a second is the real bug. One that does not was never a bug.

What it caught

On August 8th, on a QuorumFile deploy, it failed on the pricing block's document preview: painted at opacity 0, and not marked decorative.

The cause was not in the pricing block. ScrollReveals collects auto-fade candidates from main.page > * and walks down, so any element added to any section inherits a 0-opacity start state whether it asked for one or not. Mine carried no data-reveal and got one anyway. The fix was to put it in the runtime's skip list next to .header, for the same reason the header is there.

const SKIP = ".header, .trail, .nav-list, .nav-dropdown-list, .sheet-overlay, .kd-artefact-link";

The gate cannot decide this on its own, and it does not try. My footers carry a brand wordmark at opacity 0.07, which is a watermark and correct. StoreReady's hero sat at 0.001, which was a bug and shipped. Measured, those are the same observation, so the rule is that text below the floor has to declare itself with aria-hidden="true" or it counts as a failure. A heading is never decoration.

I had opened that pricing block in a browser more than once before the deploy. The gate found it, I did not.