Severity belongs to (check, kind), not to the check: fixing 512 false "will not load" verdicts

I run an index that scrapes published Claude Code artifacts — skills, subagents, plugins, marketplaces — scores them on four measured components, and publishes a page whose h1 is literally "These will not work."

That page is the whole product. It is also the only place where a bug costs someone else something. A checker's two error directions are not symmetric: a missed defect is a gap, but a false positive is a public accusation against a stranger's repo. Last weekend I found three of them in the same scorer, and they were all the same mistake wearing different clothes.

1. ?? 0 turned a shape mismatch into a verdict

The marketplace check is one line: a marketplace manifest listing no plugins can't install anything.

const n = a.extra?.plugin_count ?? 0;

extra is a shape assembled on the way into the database — after the scorer runs. The artifact the scraper hands the scorer carries plugin_count at the top level. So the read was undefined, ?? 0 made it a confident zero, and every marketplace in the index — all 55 — published a fatal "empty marketplace, will not load." Including anthropics/claude-plugins-official, which lists 276 plugins.

Optional chaining plus a nullish default is how a missing field becomes a measurement. The property access can't fail, so nothing ever throws, and the wrong answer is the plausible one.

2. The same check is not the same severity on every kind

The bigger one. The scorer flagged no-name and no-description as fatal on anything with frontmatter. Reading Anthropic's own references (skills and sub-agents, 2026-08-02):

A skill with neither field still registers. It takes its command from the directory and its routing text from whatever the first paragraph happens to say. That's worse than a written description — it is not a loading failure.

I had generalized the subagent rule across both kinds, so 494 skills, 6 plugins, 2 marketplaces and 10 subagents sat under "these will not work." The fix is that fatality stopped being a property of the flag:

const FATAL_ALWAYS = new Set(['broken-frontmatter', 'broken-manifest', 'empty-marketplace']);

/** Required frontmatter on a subagent; documented as optional on a skill. */
const FATAL_IF_REQUIRED_FIELD = new Set(['no-name', 'no-description']);

/** Does this flag stop the artifact registering, given what kind it is? */
export function isFatal(flag, kind) {
  if (FATAL_ALWAYS.has(flag)) return true;
  if (FATAL_IF_REQUIRED_FIELD.has(flag)) return kind === 'subagent';
  return false;
}

The flags themselves survive — a skill with no description is still genuinely harder to route to, and it still takes a deduction (8 points instead of 30). What changed is that a deduction stopped being a verdict.

archived came off the fatal list in the same pass. An archived GitHub repo is read-only, not unloadable; the files are there and the skill still installs. It was also already the steepest input to the Maintained component, so it was being charged twice.

3. Thresholds you remember instead of read

long-description fired above 1024 characters, with the reason string "the practical ceiling is ~1024." Nobody wrote that number down anywhere. The documented cap is 1,536 characters, over description and when_to_use combined, applied to the listing text to reduce context usage. A remembered number was penalizing descriptions the documentation explicitly allows.

That one's now in a FACTS.json register: every claim the product makes about the outside world, with the primary source URL, the date it was read, and which files depend on it. Internal arithmetic — score weights, grade boundaries — deliberately stays out. Those are the method, not claims about the world.

The check that would have caught all three

Every one of these had the same tell, and it's cheap to look for: if 100% of a category fails your checker, the checker is what's broken. Fifty-five out of fifty-five marketplaces empty. Effectively every skill on the planet missing a required field. That is never a fact about the corpus.

The generalizable version, for anything that grades other people's work: encode the bar as a sentence in the code next to the list (mine is "Claude Code cannot register the artifact — nothing softer qualifies"), keep severity keyed on the pair, not the check, and treat a suspiciously round failure rate as a bug report against yourself.

This is how we built SkillWorks — a scored index of every published Claude Code skill, subagent and plugin.