There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, four connected servers eating 21,077 tokens of context before anyone asks a question. The argument is about what MCP costs. Almost nobody in it has measured what agents actually do with the data a tool returns.
I ended up measuring that, not because I planned to, but because a maintainer refused to let me guess.
The question nobody wanted to answer with opinions
I contribute to CNCF Jaeger's MCP server. Last April I proposed exposing service performance metrics (latencies, call rates, error rates) as an MCP tool, and hit an immediate design fork: what shape should the output be?
Option one, summary rows: pre-aggregated stats per service, compact, cheap. Option two, per-bucket time series: the raw points, roughly 720 of them per service at default resolution, expensive but complete.
I asked which the maintainer preferred. The answer, verbatim, from the issue thread:
This type of decision should not be based on opinion, but on benchmarks with a real agent troubleshooting some issues and using this MCP tool to access metrics, where you could do A/B testing of different output formats.
Fair. So I built the A/B.
The setup
Everything below is public in jaeger-mcp-bench, including the harness, the tasks, the scorer, and a research log of everything that went wrong.
The fixture is Jaeger v2 with the spanmetrics connector, hotrod generating traffic, and Prometheus behind it, snapshotted so every run sees identical metric state. In front of the metrics API sits a thin bench server with exactly one switch: --format=summary|series. No new semantics, just the shape of what comes back.
Six troubleshooting tasks, and this part matters: three were chosen because I predicted summary would win them (point questions: current latency, ranking, threshold checks) and three because I predicted series would win (temporal questions: spike detection, correlation, trend). Designing the tasks to favor both arms keeps the deck unstacked against your own hypothesis.
Two agents, and not stripped-down tool loops: Claude Sonnet through the Claude Code CLI and Gemini 2.5 Pro through the gemini CLI, real system prompts and all, because that's what production agents actually are. Three trials per cell, 72 trials total, cells run in randomized order (seed 42) so fixture drift can't correlate with either arm. Scoring is programmatic against ground truth, not vibes.
What 72 trials said
| arm | correct | wrong | declined |
|---|---|---|---|
| claude / series | 18 | 0 | 0 |
| gemini / series | 17 | 0 | 1 |
| claude / summary | 10 | 1 | 7 |
| gemini / summary | 11 | 0 | 7 |
The result I expected was wrong answers. Bad format, wrong conclusions, embarrassing agent behavior, good story.
That is not what happens. Across 72 trials there was exactly one wrong commitment, and it traced to a bug in my own bench server, not to the format (disclosed in RESULTS.md; it biased against summary, and I'd rather report my own bug than a fake finding).
The real difference lives in the decline rate. Given summary rows, agents said some version of "I cannot determine this from the available data" seven times more often than with series, almost entirely on the temporal questions. And they were right to decline: aggregation had destroyed the time axis the question needed. You cannot localize a spike in an average.
Agents don't get it wrong when you underfeed them. They give up, politely and correctly.
The stats hold up under correction: Claude's summary-versus-series gap is significant after Bonferroni (p=0.001 against a 0.0125 alpha). Gemini's, at p=0.016, sits right on the boundary, and I'd rather say that than round it in my favor. Point queries were a wash, exactly as predicted: when the question needs one number, format doesn't matter.
So series won. But that's not the useful lesson
The tool shipped with per-bucket series as the decision, backed by data instead of my taste. Fine.
The useful lesson is about what "expensive output" actually buys. The whole MCP cost argument treats tokens as waste: big responses bad, small responses good. The benchmark says the relationship is more specific than that. A compact format is cheap right up until it deletes the axis a question depends on, and then its cost is not tokens, it's the agent declining to answer at all. A "cannot determine" costs you the entire investigation loop, plus the retry, plus whatever the human does when the agent shrugs.
Put differently: the token bill of series output is visible and easy to complain about. The failure bill of summary output is invisible until you measure decline rates, and nobody measures decline rates.
If you're designing an MCP tool right now, the transferable version is:
- Match the format to the question class, not to the token budget. Point questions tolerate aggregation. Temporal and causal questions do not.
- Watch declines, not just errors. Models fail politely. Your error dashboards won't see it.
- Balance your benchmark against your own prediction, or you'll build a deck stacked toward whatever you already believed.
- When your bench produces a result you like, hunt for the bug first. Mine had one, and it flattered the winning arm.
Honest limits
Seventy-two trials is three per cell. Two model families, one fixture, six tasks, all in the observability domain. This settles which format a metrics tool in a tracing backend should return; it does not settle MCP philosophy. If you run the harness against other domains I'd genuinely like to see the numbers, and everything needed to do that is in the repo.
The maintainer was right, is the short version. The format decision took a weekend to benchmark and would have taken zero minutes to argue about forever.
I do this work for hire, mostly making LLM agents safe and observable against production infrastructure. Scope and pricing are at roshansingh.systems/#hire, or write to inbox@roshansingh.systems and tell me what your agents touch.
Top comments (11)
Excellent measurement. The natural next arm seems like progressive disclosure: return a compact summary with explicit coverage/window/granularity plus stable handles for fetching buckets only for suspicious services or time ranges.
Then score the whole investigation, not just the first call: correctness, decline rate, total tokens, tool calls, latency, and whether the agent requested the right slice. That could preserve cheap point queries without making temporal questions impossible. Your result also suggests that required analytical dimensions are part of the tool contract, not merely a response-format preference.
This is the right next arm. You've described roughly what the trace side of Jaeger's MCP server already does: topology first (~185 bytes a span), then handles into full detail for the spans you suspect. The metrics tool went binary summary-or-series because that was the question I was asked to settle. Summary plus handles, with coverage and granularity explicit in the response, is the natural third format, and I'd bet it recovers most of the series arm's decline-rate win at a fraction of the tokens. Worth measuring rather than betting.
On scoring the whole investigation: agreed. The harness already drives real CLIs end to end, so the trajectories exist, I just only score the final verdict. Tokens, call count, latency and did-it-fetch-the-right-slice are capture work, not redesign. That last metric is the interesting one, because a progressive tool can fail in a way the binary formats can't: right answer, wrong slice, wasted loop.
Your last line is the one I'd put on a wall. Required analytical dimensions as part of the tool contract is a better statement of what I found than the one I wrote. The time axis wasn't a formatting preference the summary arm dropped, it was a contract term the tool silently broke, and the decline rate was the agent noticing.
Repo's public if you want to sketch the third arm. If I run it, I'll credit the design here.
Benchmarking the disagreement instead of arguing it is the right move, and the decline-rate finding is the part I'd frame as the headline. Everyone worries about the loud failure — the confidently wrong answer — but your data says the expensive failure is silent: the agent declines because you aggregated away the axis the question needed, and nobody notices except the user who didn't get an answer. Decline rate is a much better health metric for a tool than accuracy alone precisely because it's invisible in a demo. What I'd want to see next is whether the summary-vs-series tradeoff is actually a false binary. The cost objection to series (720 points per service is real context pressure at four connected servers) and the completeness objection to summary both dissolve if you split it: a cheap
summarytool for the point/ranking/threshold questions, plus aseries(service, window)drill-down the agent calls only when it detects a temporal question — let the model pay for the time axis when the question needs it, not on every call. Did any of your trials hint at whether agents would reliably reach for a second tool, or do they tend to commit to whatever the first call returned? That reachability question feels like the whole ballgame for a two-tool design.Whether agents reliably make the second call was the next thing I measured. In the follow-up (tiered nine-tool inventory vs one tool that returns everything, 72 trials) captured trajectories showed zero violations of drill-down order, and four of six tasks were solved without ever touching the verbose tier. Full writeup: dev.to/lopster568/does-progressive...
The decline-vs-wrong split matches what I've seen running agents against internal tools: a wrong answer at least shows up in an eval, but a polite "I cannot determine" just quietly burns the retry loop and nobody graphs it. We log declines as their own category now for that reason.
Curious whether you tested any middle arm between the two formats. 720 points per service is the honest answer for temporal questions, but a downsampled series with anomaly markers (top-k spikes, exact timestamps) might keep most of the axis without the full token bill. Was it strictly binary in the harness?
Not tested, and it's a fair gap: summary and series were deliberately the two extremes. A downsampled series with anomaly markers is the obvious middle arm, and my guess is it keeps summary's token profile while fixing the temporal-question failures, but that's a guess, not a measurement. It's on the list if I run a third format arm. What resolution does your internal tooling downsample to?
Picking three tasks you expected summary to win and three you expected series to win is the part of this I would steal, and it is the part most benchmark posts skip. Pre-committing to which arm should win each task is what turns a result into evidence rather than a demonstration, because it gives the outcome somewhere to disagree with you.
The question I would push on is what three trials per cell can actually resolve. With that many, only a large effect separates from run-to-run variance, so a cell that splits two-one is close to uninformative on its own even though it looks like a result. That is not a criticism of the design, it is the honest limit of the budget, and I would rather see it stated than see the per-cell numbers read as rankings.
The thing I would most want out of the data you already have: not the aggregate win rate but the variance within each cell. If the summary arm wins on average but swings wildly on temporal tasks, that instability is the finding, and it is the sort of thing an averaged scoreboard hides completely.
You are right that a per-cell 2-1 isn't a ranking, which is why the significance test runs on the 18 trials per arm rather than on the cells. The per-cell instability you're after is already published as Pass^3 here: results/tables/t6_final.md and the raw correct/decline/wrong counts per cell are here: results/tables/b75f18cd.md
This tracks with what I've seen building introspection-style MCP tools (the kind that answer "what actually happened" rather than "do a thing") — the trap isn't returning too little data, it's returning data in a shape the agent then has to re-derive conclusions from, which burns tokens and introduces exactly the kind of error your 72 trials were built to catch. Pre-aggregating the specific answer to the likely question consistently seems to beat handing over more complete data and trusting the model to summarize it correctly on the fly.
That held for the point, ranking and threshold questions, but it inverted on the temporal ones: pre-aggregating discarded the time axis and agents declined 7x more often as a RESULT. Pre-aggregating the likely question only wins when you already know which question is coming.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.