Your multi-agent system isn't hitting prompt cache. Your system prompt is the reason.

I run a multi-agent setup where ten agents analyse the same input. Same document, same market data, same everything. The only difference between them is persona: each one is instructed to look at the material through a different lens.

Ten agents, one shared context. That should be the ideal case for prompt caching. Send the expensive context once, pay full price for it once, and let the other nine reads come back at a fraction of the cost.

My cache hit rate was zero percent on three of the five models I was using, and under seven percent on the other two.

I had been reading the bill for a while and optimising the wrong thing. Here is what was actually happening, because the mistake is structural and I doubt I am the only one making it.

What prompt caching actually matches on

Hosted inference providers cache on a prefix. The provider hashes your request from the first token forward and looks for the longest run it has already computed. If your request starts with the same 3,000 tokens as a recent one, those 3,000 tokens are a cache read, typically around five times cheaper than a fresh read. The moment the token stream diverges, caching stops for the rest of the request. There is no re-syncing later.

That word — prefix — is doing all the work, and I had not thought about it carefully.

The setup that broke it

My call looked like every example in every SDK doc:

messages = [
    {"role": "system", "content": agent_persona},   # differs per agent
    {"role": "user",   "content": shared_context},  # identical for all 10
]

The persona is short. A couple of hundred tokens describing how this particular agent should reason. The shared context is large: several thousand tokens of source material.

Read that message array as a flat token stream, which is what the provider does. The first thing in the stream is the persona. The persona is different for every agent. So the prefix diverges at roughly token one, and the several thousand tokens of identical context sitting behind it can never match anything.

Ten agents. Ten identical copies of the same context. Ten full-price reads.

Proving it rather than assuming it

I did not want to guess, so I hashed both halves of every call for a single work item and counted the distinct values.

SELECT
  COUNT(DISTINCT prompt_sha256) AS distinct_user,
  COUNT(DISTINCT system_sha256) AS distinct_system
FROM agent_calls
WHERE item_id = ?

The answer:

distinct_user   = 1
distinct_system = 10

One user prompt. Ten system prompts. The expensive half was byte-identical across all ten calls, and the cheap half in front of it was unique every time.

This lines up exactly with the provider's own usage report, which broke my spend into cached and uncached input tokens:

Those low non-zero numbers are incidental collisions between unrelated calls, not the structural reuse I should have been getting. If the design were right, nine out of every ten context reads would be cache hits.

The fix

Put the shared, expensive, identical part first. Put the small, varying part last.

messages = [
    {"role": "system", "content": shared_context},          # identical -> caches
    {"role": "user",   "content": f"{agent_persona}\n\n{question}"},  # varies, small, last
]

Now the first several thousand tokens are the same for all ten agents. The first agent pays full price and warms the cache. The other nine read it back at cache rates. The only uncached part is the couple of hundred persona tokens at the tail, which is what you actually want to be paying for.

The general rule, which I now think should be a design constraint rather than an optimisation:

Order your prompt from most shared to most specific. Caching rewards a stable prefix, and every byte that varies early poisons everything after it.

This also composes with how you batch. If you run the same model across many items back to back, you keep hitting a warm prefix. If you round-robin across models for each item, you cold-start the cache on every single call. Grouping by model, not by work item, keeps the cache warm.

The part I am not comfortable with

Moving the persona out of system and into user is not free.

Some models weight system instructions more strongly than user content. That is often the point of a system prompt. If one of my agents is specifically instructed to argue an unpopular position, and I demote that instruction from system to user, it may hedge more. I would be trading spend for behaviour, and I would not necessarily notice, because the output would still be well-formed and plausible.

So this is not a change I would ship straight to production off the back of a cost argument. It needs an A/B on a sample of items, comparing the actual decisions each layout produces, not just checking that the responses parse.

There is a middle path worth trying first: keep a short stable instruction in system that is identical across all agents, and move only the per-agent differentiation into the user message. You get a shared prefix and keep a system-role framing. Whether that is enough depends on how much of your agents' behaviour hangs off the system role, which is an empirical question about your prompts and your models.

What I would take from this

1. Prefix means prefix. Anything that varies early destroys caching for everything after it, no matter how much identical material follows.
2. Instrument it. Hash the components of your requests and count distinct values per work item. It took one query to turn a vague suspicion into a definite structural bug.
3. Read the cached-versus-uncached split in your usage report. A near-zero cache rate on a workload with obvious shared context is not a pricing quirk. It is a design bug, and it is telling you the prefix is broken.
4. The default SDK message shape is not cache-aware. Persona-in-system, content-in-user is the shape in every tutorial. It is exactly wrong for fan-out workloads where many personas share one context.

I had spent real effort choosing cheaper models before I checked whether I was paying for the same tokens ten times over. The model swap was worth doing. It was also the second-biggest lever, and I found it first because it was the one I was looking for.