There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, f...
For further actions, you may consider blocking this person and/or reporting abuse
Excellent measurement. The natural next arm seems like progressive disclosure: return a compact summary with explicit coverage/window/granularity plus stable handles for fetching buckets only for suspicious services or time ranges.
Then score the whole investigation, not just the first call: correctness, decline rate, total tokens, tool calls, latency, and whether the agent requested the right slice. That could preserve cheap point queries without making temporal questions impossible. Your result also suggests that required analytical dimensions are part of the tool contract, not merely a response-format preference.
This is the right next arm. You've described roughly what the trace side of Jaeger's MCP server already does: topology first (~185 bytes a span), then handles into full detail for the spans you suspect. The metrics tool went binary summary-or-series because that was the question I was asked to settle. Summary plus handles, with coverage and granularity explicit in the response, is the natural third format, and I'd bet it recovers most of the series arm's decline-rate win at a fraction of the tokens. Worth measuring rather than betting.
On scoring the whole investigation: agreed. The harness already drives real CLIs end to end, so the trajectories exist, I just only score the final verdict. Tokens, call count, latency and did-it-fetch-the-right-slice are capture work, not redesign. That last metric is the interesting one, because a progressive tool can fail in a way the binary formats can't: right answer, wrong slice, wasted loop.
Your last line is the one I'd put on a wall. Required analytical dimensions as part of the tool contract is a better statement of what I found than the one I wrote. The time axis wasn't a formatting preference the summary arm dropped, it was a contract term the tool silently broke, and the decline rate was the agent noticing.
Repo's public if you want to sketch the third arm. If I run it, I'll credit the design here.
Benchmarking the disagreement instead of arguing it is the right move, and the decline-rate finding is the part I'd frame as the headline. Everyone worries about the loud failure — the confidently wrong answer — but your data says the expensive failure is silent: the agent declines because you aggregated away the axis the question needed, and nobody notices except the user who didn't get an answer. Decline rate is a much better health metric for a tool than accuracy alone precisely because it's invisible in a demo. What I'd want to see next is whether the summary-vs-series tradeoff is actually a false binary. The cost objection to series (720 points per service is real context pressure at four connected servers) and the completeness objection to summary both dissolve if you split it: a cheap
summarytool for the point/ranking/threshold questions, plus aseries(service, window)drill-down the agent calls only when it detects a temporal question — let the model pay for the time axis when the question needs it, not on every call. Did any of your trials hint at whether agents would reliably reach for a second tool, or do they tend to commit to whatever the first call returned? That reachability question feels like the whole ballgame for a two-tool design.Whether agents reliably make the second call was the next thing I measured. In the follow-up (tiered nine-tool inventory vs one tool that returns everything, 72 trials) captured trajectories showed zero violations of drill-down order, and four of six tasks were solved without ever touching the verbose tier. Full writeup: dev.to/lopster568/does-progressive...
The decline-vs-wrong split matches what I've seen running agents against internal tools: a wrong answer at least shows up in an eval, but a polite "I cannot determine" just quietly burns the retry loop and nobody graphs it. We log declines as their own category now for that reason.
Curious whether you tested any middle arm between the two formats. 720 points per service is the honest answer for temporal questions, but a downsampled series with anomaly markers (top-k spikes, exact timestamps) might keep most of the axis without the full token bill. Was it strictly binary in the harness?
Not tested, and it's a fair gap: summary and series were deliberately the two extremes. A downsampled series with anomaly markers is the obvious middle arm, and my guess is it keeps summary's token profile while fixing the temporal-question failures, but that's a guess, not a measurement. It's on the list if I run a third format arm. What resolution does your internal tooling downsample to?
Picking three tasks you expected summary to win and three you expected series to win is the part of this I would steal, and it is the part most benchmark posts skip. Pre-committing to which arm should win each task is what turns a result into evidence rather than a demonstration, because it gives the outcome somewhere to disagree with you.
The question I would push on is what three trials per cell can actually resolve. With that many, only a large effect separates from run-to-run variance, so a cell that splits two-one is close to uninformative on its own even though it looks like a result. That is not a criticism of the design, it is the honest limit of the budget, and I would rather see it stated than see the per-cell numbers read as rankings.
The thing I would most want out of the data you already have: not the aggregate win rate but the variance within each cell. If the summary arm wins on average but swings wildly on temporal tasks, that instability is the finding, and it is the sort of thing an averaged scoreboard hides completely.
You are right that a per-cell 2-1 isn't a ranking, which is why the significance test runs on the 18 trials per arm rather than on the cells. The per-cell instability you're after is already published as Pass^3 here: results/tables/t6_final.md and the raw correct/decline/wrong counts per cell are here: results/tables/b75f18cd.md
This tracks with what I've seen building introspection-style MCP tools (the kind that answer "what actually happened" rather than "do a thing") — the trap isn't returning too little data, it's returning data in a shape the agent then has to re-derive conclusions from, which burns tokens and introduces exactly the kind of error your 72 trials were built to catch. Pre-aggregating the specific answer to the likely question consistently seems to beat handing over more complete data and trusting the model to summarize it correctly on the fly.
That held for the point, ranking and threshold questions, but it inverted on the temporal ones: pre-aggregating discarded the time axis and agents declined 7x more often as a RESULT. Pre-aggregating the likely question only wins when you already know which question is coming.