Our AI pipeline runs three step kinds (ai.generate, ai.extract, ai.classify), each independently resolving its own provider, model, and prompt revision at run time. The default model is Sonnet, not Opus — and for a while that felt like a compromise, because Opus was the expensive-but-reliable option and Sonnet needed babysitting to hit the same pass rate.
The fix that closed the gap wasn't a smarter prompt. It was tool_choice plus a tightened output schema, forcing the model to commit to an answer shape instead of spending tokens hedging its way there. That alone got Sonnet to pass-parity with Opus's prior output at roughly a quarter of the cost, in our eval. A second, separate lever: Anthropic's own recommended effort: "xhigh" for agentic tasks produced roughly twice the thinking tokens of "high" for the same pass rate — thinking tokens bill like output tokens, so that's a straight 2x for zero accuracy gain, and it only matters when Opus is used via an explicit override (Sonnet doesn't get adaptive thinking at all). A third, independent lever: max_tokens defaulted to 16000 out of caution; dropping it to 4000 (still comfortably above what any real step needed) cut effective output cost again, because Anthropic bills against the cap as an upper bound in some failure paths, not only against what the model actually emits.
None of these three touched the prompt content or the provider. All three came from looking at per-step token usage, which only exists because every step writes an immutable provenance record on completion: prompt revision, provider, model, token usage. Same idea as event sourcing, applied to LLM calls instead of domain writes. You don't trust "the pipeline probably used prompt v3, on whatever model was configured," you have a row that says so — and you can audit six months of "which prompt touched this tenant" without ever loading the actual LLM payloads (also the DSGVO-friendly shape: provenance rows carry no user content, only call metadata).
The other real trap, found the hard way: adaptive thinking and forced tool-use don't mix. Anthropic 400s with "Thinking may not be enabled when tool_choice forces tool use" the moment both are set, which sounds obvious in hindsight but not when you're setting both because both individually sound like "make the model try harder." The fix has to be automatic — detect a forced tool_choice and disable adaptive thinking before the request goes out — because leaving it to every caller to remember means every eval run using toolChoice: { type: "any" } against an Opus override breaks the same way, all at once.
Separately: prompt caching only pays off if the cache_control breakpoint sits on the last static block, with tools and system prompt as one cacheable prefix. Past roughly 16k tokens without it, requests start hitting the SDK's HTTP timeout before the response streams back far enough to matter. Put the breakpoint on something that changes per request and you get a silent cache-miss that looks identical to a hit in the logs while you keep paying full price.
Full writeup with the pipeline architecture (with diagrams), the provider-resolution code, and the provenance shape: docs.kumiko.rocks/en/guides/ai-pipeline-provenance.
Top comments (7)
The part that's easy to skip here is that none of the three levers were findable without the per-step provenance row. It reads as a cost problem and it was an observability problem first. On the silent cache-miss: the usage block returns cache_read_input_tokens and cache_creation_input_tokens separately, so a hit ratio that quietly collapses is detectable. Do you write those into the provenance record and alert on them, or is it still caught when the invoice shows up?
honest answer: right now it's caught at invoice review, not alerted on. cache_read/creation ratio isn't in the provenance record yet, only prompt/provider/model/usage. good catch though, that's a cheap column to add and an obvious alert threshold once it's there. will add it.
One thing worth deciding before the threshold: a global ratio moves for boring reasons. Every prompt edit or deploy invalidates the prefix, so creation legitimately spikes and a single global threshold ends up paging you on your own releases. Keyed per prompt version and route it gets much quieter, and the signal you actually want is the ratio failing to recover after the first few calls, not the spike itself.
that's the right key, and it's basically free since prompt revision is already on the provenance row. "failing to recover after the first few calls" is the sharper framing too, a deploy-triggered spike that heals in a handful of requests is expected, one that plateaus is the actual signal. will key the alert that way instead of a global rate.
The prompt caching point is the one that bites silently: a breakpoint on anything per-request (user content, dynamic tool args) produces a cache miss on every call that logs exactly like a hit — you have to check the cache_read_input_tokens field in the response to confirm it's actually hitting. The tool_choice and adaptive thinking incompatibility is also easy to fall into because both surface as "make the model try harder" in the docs, and the 400 only happens at runtime. One addition to the max_tokens insight: the billing behaviour you describe specifically applies to server-side validation errors where Anthropic has already loaded the request; dropping the cap also narrows the window in which an unexpectedly long generation runs the clock before you can cancel it, so it doubles as a latency guard on pathological responses. The provenance-as-event-sourcing framing is the structural idea worth keeping — treating every LLM call as an immutable fact rather than a configurable behaviour means your cost audit surface is a query, not a conversation.
good catch on the latency angle, hadn't framed it that way but it's right: the cap isn't just a cost lever, it's the timeout backstop too. thanks for reading closely.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.