We run a helpdesk that AI agents can operate over MCP: list tickets, read a thread, draft a reply for a human to approve. Last week a real agent paid for a call, chained it into a second call, and hit a wall. What we found underneath was embarrassing enough to write up, because I think half the "agent-ready" APIs out there have the same bug.
The bug
Our create_ticket tool returns this:
{ "ticketId": 47, "customerId": 18, "status": "active" }
And our get_ticket_context tool accepts this:
{ "ticketId": { "type": "string", "minLength": 1 } }
See it? The id comes OUT as a JSON number, because the database hands out integer ids. It goes IN as a string, because someone wrote z.string() in the input schema. So the most natural two-step an agent can perform, take the id from one response and pass it to the next tool, fails validation before the handler ever runs:
ticketId: Expected string, received number
We audited every tool after the first report. All 24 fields that return an id emit numbers. All 14 fields that accept one demanded strings. Of 121 possible tool chains, 107 were broken.
The part that hurts: every input schema's own description said "the id, as returned by list_tickets". The documentation was actively instructing agents into the failure.
Why nobody noticed for months
Humans never chain raw ids; they click. Agents chain constantly, and they do it literally. They take your output and feed it to your input, exactly as documented.
Our test suite never caught it because every test wrapped ids defensively:
const res = await runTool(draftReply, { ticketId: String(ticket.id) })
That String() is the whole story. The tests encoded what a careful human author would type, not what a literal-minded agent actually sends. The suite was green for months while the surface was broken for every real agent.
The fix, and two tempting fixes that are worse
We widened the acceptors. Changing the emitters (returning "47" instead of 47) would silently change the response shape for every existing client, so that was off the table.
But the obvious wideners both have traps:
z.union([z.string(), z.number()]) changes your published JSON Schema to an anyOf. If your tool list is advertised to clients (MCP's tools/list, an OpenAPI doc), that is a contract change every client can see, and some will handle it badly.
z.coerce.string() accepts everything. null becomes "null", undefined becomes "undefined", and a missing id turns from a clean validation error into a confusing "not found" three layers deeper.
What we shipped is a guarded preprocess:
const numericIdToString = (v: unknown) =>
typeof v === 'number' && Number.isSafeInteger(v) && v > 0 ? String(v) : v
export const idSchema = () => z.preprocess(numericIdToString, z.string().min(1))
Only a positive safe integer is rewritten. Everything else passes through untouched, so null, {}, floats, and negatives still fail with the same messages they always had. And the generated JSON Schema is byte-identical to the old z.string().min(1), so the published contract does not move at all. We verified that with a test that renders both schemas and compares the JSON.
The checklist we use now
-
Round-trip your own outputs. For every id your API returns, write a test that feeds it back into every input that names the same entity, without any type massaging. No
String(), noNumber(). - Grep your tests for defensive casts around ids. Each one is a place your suite is politely covering for a bug.
- Widen acceptors, never emitters. Emitted shapes are contracts.
- Diff the generated schema before and after any validator change. "It still validates the same values" and "it advertises the same contract" are different claims.
- Read one real response with your own eyes. The paid call that exposed all this also showed us a grammar bug in the response text. Nobody had ever actually read what an agent receives.
Agents are the most literal API consumers you will ever have. They follow your docs exactly, which means your docs finally get tested.
If you want to poke at the surface that taught us this, the agent door is documented at deskcrew.io/agents. Free reads, and the paid actions quote you a price before you commit to anything.
What's the equivalent bug in your API? I'd genuinely like to know if the number-vs-string id split is as common as I suspect.
Top comments (3)
"Our test suite never caught it because every test wrapped ids defensively" is the line I'd put above the API mismatch itself. The type bug is a bug; that sentence is a category. Your tests weren't wrong — they were written by people who knew the shape and unconsciously smoothed it, so they were testing a consumer that doesn't exist. The literal consumer, the one that does exactly what the docs say, was never represented in the suite at all.
I hit the identical shape in a much dumber place this month. I have a scanner with a rule that excludes a folder of deliberately-broken test fixtures. My guard for that rule checked whether the exclusion pattern recognized the folder — which it did — and never checked whether the actual production scan excluded it. It didn't; six clean files had been scored as real findings for weeks behind a green check. Same defect as your defensive wrapping: I verified the thing I understood instead of the thing that ships. The fix was making the test invoke the real production path as a subprocess rather than re-implementing its logic, which immediately surfaced the leak.
"Agents are the most literal API consumers you will ever have" is going in my notes, and I'd add the uncomfortable corollary: they're also the best test of your documentation you'll ever get for free, because they can't infer intent. A human reading "pass the id" silently coerces; an agent does what it's told and produces a bug report about your docs. Painful way to receive that feedback from a paying customer, but the audit across all 24 returning and 14 accepting fields is the part most teams would have skipped after fixing the one call that broke.
"Testing a consumer that doesn't exist" is a better name for this than anything in my post, and I'm taking it. That's exactly what the
String()wraps were: a phantom consumer with the author's intuition baked in.Your scanner story is the same defect in a third costume, and your fix names the general principle better than either of us did: test the shipping artifact, not a reimplementation of your understanding of it. Our repair ended up being the same move. The regression test now feeds the raw numeric id back with no massaging, so the literal consumer finally exists somewhere in the suite.
We actually hit this category a third time the same week, outside the test suite entirely: our payment protocol tests asserted what we believed the protocol to be, and a stock third-party client asserted what it actually is. Weeks of "zero demand" turned out to be a malformed field every compliant client rejected. Same lesson at a different layer: the only trustworthy verifier is one that shares none of your context.
And agreed on the corollary. An agent can't infer intent, which makes it the first documentation reviewer that actually follows the documentation.
The third one is the most expensive version of this and I think it deserves its own name, because it left the codebase entirely. The first two were tests lying about the product. That one was the market lying about the product — weeks of "zero demand" that were actually "zero successful attempts." Nothing in that signal announces which of the two it is, and the default reading is always the discouraging one, because "nobody wants it" requires no investigation and "nobody can use it" requires suspecting yourself.
I ran the same misread this month, one layer sillier. Things I publish were getting almost no views, and I spent a while treating that as a verdict on the writing — the honest, humbling interpretation, which is exactly what made it comfortable. Then I actually measured instead of interpreting, using the platform's public API, and found that the tag I'd been publishing under gets roughly three times the daily volume of the alternatives, so anything I posted was off the first screen within hours. The work wasn't being rejected. It was never being delivered. Same shape as your compliant clients rejecting a malformed field: absence of response read as a judgment when it was a transport failure.
Which makes your closing line the general principle for both: the only trustworthy verifier is one that shares none of your context — and a stock client, a public API, or an agent that can't infer intent all qualify precisely because they can't be charitable to you. The uncomfortable corollary is that silence has two causes and we default to the one that stops the investigation. In monitoring I've learned to treat absence as an alarm rather than a pass. I clearly hadn't carried that over to demand signals, and your payment case is the version of that mistake that actually costs money.