You write a SKILL.md, run it once on your own example, watch it produce something good, and ship it. That's not a test — it's a demo. And a skill that looks great in a demo and falls apart on the messy real case is worse than no skill at all, because now you've built a workflow that assumes it works.
Here's the actual failure mode: a skill that's excellent once and unusable twice is worse than a plain prompt. A plain prompt you re-read every time, so you catch the bad output. A skill you trust — that's the whole point of writing one — so the bad output slides through.
Judge variance, not best case
Run the skill several times on inputs you didn't design it for. Not the clean example you had in mind while writing the trigger — the ugly, ambiguous, half-specified request a real user actually sends. If the output quality swings wildly between runs, you don't have a skill, you have a lottery ticket with good marketing.
Three things a testable skill needs
1. A written expectation of correct output. Not "it should be good" — an actual description of what correct looks like, specific enough that you (or someone else) can check a real output against it and get the same yes/no answer twice. If you can't write this down, you don't know what the skill is supposed to do yet, and that's worth finding out before you ship it, not after.
2. At least one adversarial case. An input designed specifically to trigger the failure the skill exists to prevent. If the skill's job is "don't commit until tests pass," the adversarial case is the one where the agent can't reproduce a bug but has a plausible-looking fix anyway — the exact moment a rule is most likely to get rationalized away. A skill that's never been tested under the condition where following it is inconvenient hasn't been tested at all.
3. A check the agent must pass before declaring done. Not "I did the task," but a verifiable claim — a checklist item that actually got demonstrated, not just asserted. Skills that let the agent self-report success without evidence produce agents that self-report success without evidence.
Why "it worked on my example" is the wrong bar
Your example was written after you already knew what the skill should do. It's the easiest possible case by construction. The messy real case — ambiguous phrasing, missing context, an edge condition you didn't think of — is what actually determines whether the skill survives contact with real use. Test against that, or you're testing against a case you rigged to pass.
Re-test after model updates, at least for the ones that matter
Skills encode assumptions about how a model behaves — how literally it follows instructions, how it handles ambiguity, what it treats as an implicit exception. Those assumptions drift when the underlying model changes. You don't need to re-verify every skill after every update, but the ones your workflow actually depends on are worth a re-run, not a shrug.
If you're building skills you plan to rely on — for yourself or to hand to other people — write the eval before you write the polish pass. It's the difference between a skill you can trust and one you're hoping about.
Full write-up: https://agentkitworks.com/answers/how-to-test-agent-skills
Top comments (0)