In a traditional SDLC, validation was someone else's job and it came later. QA teams checked that the software did what it was supposed to. The behavior was repeatable, so eventually you automated the checks and moved on.
So when building Slooster, I did the hard parts first:
- Building the prompts, tuning them, defining schemas for parameters and outputs.
- Building a system to pick the provider, the model, the temperature, and other tuning configs per prompt.
Everything worked great locally, so I told myself the validation layer could wait.
Unfortunately, the traditional SDLC model breaks the moment an LLM enters the equation. Remember, LLMs are producing output based on guessing off the context. They're getting better at it, no doubt, but they are still guessing. The same prompt will give you a different answer next time, or even an incorrect one, with nothing changed, so validation must be part of the software itself.
When a model is in the loop, you have to, from day one, build a system that can absorb the unpredictability without a hiccup.
The (un)predictability factor
After deploying Slooster Guide to staging, I gave it a once-over before demoing to my co-founders.
The exact same prompts (and model, and configs) that had been returning me real, criteria-matched vendors for weeks came back with what looked like placeholder data, literally outputting the words Vendor A, Vendor B, Vendor C. Regenerating then brought Vendor D into the mix.
A model that gives you clean output locally for weeks can still hand you garbage on a real call, and no amount of testing beforehand would be able to prevent it. That is the whole case for validating inline, on every actual output, instead of once before you ship.
The layer I built next
I added the validation layer, with retries built in: bad output gets caught and runs the call again. But retries cost time, and the app needs to stay responsive while a user is waiting on it.
So validation isn't free, and it isn't one-size-fits-all. Where it runs, how hard it retries, and whether a human sees the result all depend on your use case.
Letting a model check a model
The deterministic checks come first: a schema validator rejects anything structurally off-spec. But a placeholder like "Vendor A" is structurally valid. It passes a schema check, yet it's still not a valid output. That is a semantic miss, not a structural one, and you need a separate type of check for it.
So the last gate is a model checking a model: a validation prompt judging the output against the same criteria the first prompt was given.
One AI-gate prompt, roughly:
You are checking another model's answer before a user sees it.
The task it was given: [the original request and selection criteria].
Its answer: [the response to check].
Reply as JSON: 'pass' (true/false), 'reason' (one line), 'confidence' (0 to 1).
Fail it if the answer has placeholder names, generic filler, items that don't meet the stated criteria, or anything that reads like example data instead of a real result.
The gate can run on the same model that produced the answer, or on a different provider and model entirely for an independent second opinion. Make it a config choice, so that you don't need a code change to tune.
The three important takeaways:
- Not every response should take the retry hit, but not every response should be trusted silently either.
- Surface the confidence you have in an answer. Let the user see it.
- Depending on the scenario, show the output with a clear way for the user to push back, and loop that feedback straight back into the system so the next attempt knows the bar it missed.
A model shouldn't get the last word on its own output. Keeping a human in the loop is a feature, not a gap you'll close later.
Same question, different answer
A model won't give you the same response on the next call, which makes naive caching a trap, and no caching expensive.
The answer is caching intelligently:
- Reuse a good result when it makes sense.
- Give the user explicit control to regenerate when they want it refreshed.
Intelligent caching also pays for validations. Structuring prompts so the stable part is cached offsets a good chunk of the extra cost the validation layer incurs.
Not every prompt is equal
By treating all prompts the same, you'll end up with increased costs and an underwhelming performance.
Instead, match each prompt type to the proper provider and model:
- A cheap, fast model for the simple calls.
- A stronger model where the answer actually matters.
- A suitably selected model for the validation gate.
Per-prompt selection isn't a premature optimization. It's how you keep the whole system responsive while still validating the parts that count.
The cornerstone
I had planned to add validation after the MVP. That staging shock pulled it all the way forward, and I'm glad it did, as I otherwise very likely would not have accounted for getting placeholder data in the response.
The validation layer is an integral part of the system, along with transformers, schemas and specs, and it's the part I'd start with for the next system I build.
When a model is doing the work, validation can't simply be a layer you choose to add. It needs to be the cornerstone you build on.
Again, there is no one-size-fits-all solution here. AI is giving you leverage by doing a lot of the work, but you still need to be in charge, and organizing your thoughts up front matters more than ever.
If you're building software with AI workflows as part of the solution and wrestling with the same unpredictability, I'd be happy to share my experiences and learn about yours.
Added after publishing: @fromzerotoship asked three very good questions in the comments that this post had no answer for. What was on the fail list that I hadn't already been burned by, has the gate ever failed on purpose, and does a same-model check count as a second opinion? They needed more room than a comment reply, so they got their own post: Lesson 4b - Validation: Testing the gate itself.
Top comments (3)
The structural/semantic split is the right cut, and "Vendor A" is a good example
precisely because it's obvious in hindsight. The question I'd ask about the
gate: what's on the fail list that you haven't already been burned by?
Placeholder names, generic filler, example data — those are failures someone met
and then named. The case that worries me is three real-sounding vendors that are
plausible, structurally clean, and wrong. It passes every criterion on that
list, and it's exactly where a semantic gate should be earning its keep.
The other thing I'd check is whether the gate has ever failed on purpose. I run
what I think of as a match test on my own guards: deliberately break the exact
thing the guard exists to catch, confirm it goes red for that reason, then
restore. A gate that has never rejected anything under test is indistinguishable
from a gate that always passes, and you find out which one you have at the worst
possible moment. Mine took four rounds with a stranger before it caught the bug
it was written for.
On running it against the same model that produced the answer: I did that for
months with an AI reviewer and read a long run of "nothing found" as evidence
things were clean. Same weights, same priors, same blind spots. A second
signature is not a second opinion.
@fromzerotoship Regarding
It is a multi-step process in the workflow
For
and lastly
So design your system to handle the fact that it has to deal with data that can be inaccurate. How to deal with it depends on the use case you are trying to solve.
"You are now QA'ing the model API" is the right reframe and I'm keeping it.
One push on the bottom line, because I think it gives away more than you meant.
You can't force the model to be wrong — agreed. But you don't need to. The gate's
input is a string. Hand-write "Vendor A, Vendor B, Vendor C" and feed it straight
to the validation prompt: the gate is now being tested with a known-bad input and
the model that produced it is out of the loop entirely. That's your AI-mock step,
and I think it's worth naming explicitly as the forcing mechanism, because
"we cannot force it to be wrong" is exactly the sentence that licenses leaving a
gate untested. The model isn't the thing under test. The gate is, and it takes
input you control.
On capturing flags and feeding them back: that improves the gate on modes it has
already met, and it also measures the gate with the gate. As it gets better at
seen modes the flag rate drops — and a falling flag rate looks identical whether
the outputs got cleaner or the gate went blind to something new. That's the same
shape as my months of "nothing found," one level up.
The only exit I know is a denominator the gate doesn't produce: sample some
percentage of outputs it passed and have a human read them. Small rate, but
without it there's no signal about the modes not on the list — which is where the
plausible-and-wrong vendors live.