Intro
In a demo, one agent that "does everything" looks like impressive economy: one prompt, one API call, one mental model. Nobody's testing it against thousands of edge cases a day, and nobody's paying for the tokens burned parsing forty unrelated instructions to answer a two-line question. In production, that same design is where teams spend their first real outage. Below are four failure modes that show up once a monolithic agent hits real traffic. (Illustrative composites, not case studies from a specific client.)
The system prompt that nobody can safely edit
Picture a support agent that started as "answer product questions" and grew, feature request by feature request, into a 4,000-word system prompt covering billing, refunds, technical troubleshooting, and tone guidelines for three different customer segments. Adding a new instruction for edge case #41 quietly changes how the model handles edge case #12, because both are competing for the same attention budget in the same call. Nobody can point to which line caused the regression, because there's no isolation between concerns, only proximity.
Every request pays for capabilities it doesn't use
A single do-everything agent charges the same token bill for a request that just needs a database lookup as it does for one that needs careful reasoning about a refund policy exception. Say a team's simplest, highest-volume intent, "what's my order status," routes through the full monolithic prompt anyway. That's paying premium reasoning costs, on every call, for a task a five-line function could handle for a fraction of the price.
One bad instruction degrades everything downstream
Because a monolithic agent handles routing, task execution, and formatting in a single inference pass, a subtle drift in one part of its behavior, say it starts hedging more on refund questions after an unrelated prompt tweak, has no boundary to stay contained inside. There's no seam where you could catch it before it reaches the user, because the "component" that's misbehaving is the entire agent.
Testing becomes a probability exercise, not an engineering one
With a single-purpose function, you write a test, you know the input, you assert the output. With a 4,000-word do-everything prompt, "testing" often means running the same conversation a dozen times and hoping the failure rate stays under some tolerable threshold. That's not a test suite. That's a weather forecast.
A multi-purpose agent isn't one decision. It's twenty, made by an LLM, at inference time, with nothing watching any of them.
Every one of these failure modes comes from the same root cause: collapsing routing, task execution, and state management into a single non-deterministic call and hoping the model sorts it out consistently, every time, forever. It won't, not because the model is bad, but because that's not what a single inference pass was ever built to guarantee.
Route it, don't merge it
The fix isn't a smarter prompt. It's separating what genuinely benefits from a language model's judgment from what doesn't. A small classifier decides intent. Specialized, single-purpose functions handle each isolated task, several of which may not need an LLM at all. Deterministic code owns routing, state, and final formatting, the parts where you actually want predictability, not creativity. Each piece is small enough to unit test on its own, which is the property the monolith could never give you.
Where's the line for you, at what point does "just add another instruction to the prompt" stop being the pragmatic choice and start being the thing you'll be debugging at 2am?
Top comments (9)
The 2am debugging point for us was when the same prompt section started controlling both routing logic and tone, so fixing a customer segment's tone broke routing for another. The seam you describe - splitting routing/intent classification from task execution - is what finally gave us reproducible test cases: intent in, intent out, no LLM ambiguity. The token cost argument is underrated too; once we routed the high-volume, simple-lookup intents to deterministic functions, our inference spend dropped without touching accuracy. The last holdout for the monolith is usually the engineer who wrote the original prompt - nobody else feels safe editing it, which is itself a signal.
That failure mode - one prompt section owning both routing and tone - is the clearest example of why "attention budget" isn't just a metaphor. Two unrelated concerns literally competing for the same tokens, with no interface between them. And you're right about the social signal: when only one person feels safe editing a prompt, that's not seniority, that's a bus factor of one wearing a trench coat. The reproducibility win you mention is the part I wish more teams measured before the migration. "Intent in, intent out" is testable in the normal engineering sense. "Did the whole agent behave reasonably" is not. On the token spend, I'd add that the savings usually understate the real gain, because the latency drop on deterministic paths changes what product decisions become viable downstream (retries, fan-out, richer UIs).
The testability framing is the clearest argument for the separation: "intent in, intent out" fits in a unit test matrix, while "did the agent behave reasonably" requires a human rubric that shifts with each reviewer. The downstream product viability observation mirrors ours — once deterministic paths ran sub-100ms, retry budgets became cheap enough to absorb into the UI flow, which unlocks product choices that simply weren't on the table at 3s latency. The bus factor point is the one worth adding to architecture reviews as a concrete signal: if only one person feels safe editing a prompt, the architecture has been encoded into prose instead of interfaces. Worth tracking what fraction of your eval suite still lives in the "behaved reasonably" category — that's the backlog of separations left to do.
I think the hidden cost is change isolation.
A monolithic agent isn't just harder to reason about it's impossible to evolve safely. Every new capability becomes a prompt edit that implicitly retests every existing capability, even if they're unrelated.
We've seen the opposite work much better at IT Path Solutions: keep the LLM focused on decisions that genuinely require reasoning, but let deterministic services own routing, state, permissions, and business rules. That way a change to billing logic can't accidentally alter search behavior or support responses because they don't share the same execution surface.
A good litmus test is simple: if you can't deploy one capability without regression-testing every other capability, you don't have one agent you have tightly coupled production risk disguised as convenience.
Change isolation is the right frame, and I think it's actually the one that lands hardest with engineering leadership, because it maps cleanly onto things they already care about: blast radius, deploy independence, regression surface. Your litmus test is a good one. I'd offer a companion: if you can't describe a single capability's behavior without referencing the prompt of an unrelated capability, they're already coupled, you just haven't paid for it yet. The point about permissions is one I under-weighted in the post. Once auth and business rules leak into a system prompt, you don't just have a testing problem, you have a security posture that depends on an LLM continuing to interpret instructions the same way it did last week. That's not a boundary anyone should be comfortable relying on.
The 'route it, don't merge it' principle is exactly the architecture we landed on for Opportunity Skill. The Skill has 16 callable functions, but they are organised into 6 independent modules, authentication, user representation, human card management, human discovery, human outreach, and lead engagement. Each module has its own process flow, its own guidelines, and its own failure modes. The agent activates the relevant module based on user intent rather than loading a 4000-word monolithic prompt that covers everything simultaneously. The testing point resonates strongly too. Each function returns deterministic types, booleans, lists, dictionaries, so you can assert on outputs without running ten conversations and praying. The one place we deliberately keep a human in the loop is outreach. Discovery and engagement can be automated as recurring tasks, but sending a contact message always requires human confirmation. That boundary exists precisely because a single bad instruction in an automated outreach path would degrade trust globally, with no module boundary to contain the damage.
That module breakdown is a good concrete example of the principle in practice, and the human-in-the-loop boundary on outreach is the part I'd underline for anyone reading. It's the same reasoning as circuit breakers in distributed systems: the cost of a wrong action is asymmetric, so you don't let the automated path own the commit step. The deterministic return types point is underrated too. Once functions return booleans and structured data instead of prose, your assertions look like normal unit tests again, and the "did the LLM phrase it acceptably" question stops being on the critical path. Curious whether you found the six-module split up front or whether it emerged after a couple of painful merges, because in my experience teams almost never get the boundaries right on the first pass.
it is intresting
...
Thanks for reading. If you've hit a version of this in your own work - even a small one - I'd be curious which failure mode showed up first. In my experience the token cost is what people notice, but the testing problem is what actually forces the redesign.