Update: Aug 2, 2026 - I shipped 0.1.0
I did not get to this opinion from a whitepaper.
I got to it because I am building an open-source project around the problem, and the build keeps arguing back.
I am working on AgentEval Forge right now, an OSS evaluation lab for agents. The original idea sounded straightforward enough: scenario packs, adversarial cases, trajectory scoring, regression tracking, cost and latency analysis. In my head, it was going to be a more serious, more structured version of evaluation work I had already done elsewhere.
That was the first mistake.
I had already spent time building model-eval and workload-eval style systems. I knew what it meant to compare models for real tasks, how to design rubrics, how to think about speed versus cost, and how quickly a neat score can become misleading if the harness is weak. I assumed agent evaluation would be an extension of that same world.
The deeper I got into AgentEval Forge, the less true that felt.
Model evaluation asks whether the answer is good.
Agent evaluation has to ask whether the system behaved well enough to trust.
Those are not the same question.
The comparison that made it click
A clean model-eval setup is something I know how to reason about. You have an input, an output, and some way of scoring quality. Sometimes that is exact match. Sometimes it is a rubric. Sometimes it is an LLM judge. Sometimes it is a benchmark harness. However fancy the setup gets, the center of gravity is still pretty stable: did the model produce a good answer for this task?
That is already hard enough. I have seen enough brittle scoring and false confidence to know that even model evaluation can go wrong in ways that look scientific. A rubric can reward the wrong thing. A binary check can miss something obviously useful. A benchmark can feel objective while still pulling you toward the wrong optimization target. So I am not pretending model eval is solved.
But with agents, the shape of the problem changes.
The thing being evaluated is no longer just an answer. It is a workflow. It is a sequence of decisions. It is tool choice, retries, intermediate state, cost, recovery behavior, and sometimes policy adherence. The moment you let a system do more than respond once, the final output stops being the whole story.
That sounds obvious when you say it plainly. In practice, I think a lot of teams still use answer-scoring habits on systems that have already moved past that category.
What changed for me while building AgentEval Forge
The real shift happened when I tried to define what AgentEval Forge would actually have to score.
At first, I thought the answer would be simple: run scenario packs, score the outputs, compare versions, done.
But the more I looked at real agent behavior, the more that started to feel naive. An agent can produce a decent final answer while doing a lot of things badly on the way there. It can choose the wrong tool first and recover by luck. It can loop more than it should. It can spend five times the tokens a better path would spend. It can take a risky step that a human reviewer would never approve, then still back into something that looks acceptable at the end.
If I only score the final answer, I might mark that run as a success.
That is the trap.
The path matters.
That is the sentence I keep coming back to while building this. The path matters just as much as the endpoint, and sometimes more, because the path is where cost, safety, and trust actually live.
That challenge shows up in very practical ways. I am not just trying to score whether an agent got the answer right. I am thinking about cases where it picked the wrong tool first, passed the task only because it recovered late, crossed a boundary it should not have crossed, spent too many steps on something simple, or regressed only inside one narrow scenario family while the average score still looked fine. That is the kind of mess that makes agent evaluation feel much more like systems work than answer grading.
I had already seen hints of this in earlier work
This was not the first time I had felt the edges of the problem.
In earlier harness and field-study work, especially when running systems against real workloads instead of toy examples, I kept finding failures that would never have shown up in a neat prompt-eval setup. That experience made me much less impressed by demo-path confidence.
A system can look strong in a controlled evaluation and then become erratic as soon as the environment gets irregular. Real repos. Real logs. Real commit histories. Real naming mess. Real ambiguity.
That is where edge cases show up. That is where tool misuse shows up. That is where “technically correct” and “safe to ship” start to drift apart.
I think that history is part of why AgentEval Forge took the shape it did in my head. I was not trying to build a leaderboard generator. I was trying to build something that feels closer to release discipline for agents.
Not: did this run look good?
More like: if I change this system, what got better, what got worse, what got more expensive, and what became riskier even if the top-line score improved?
That feels much more like engineering than benchmarking.
The five things I now think agent evaluation has to cover
The more I think about it, the more I come back to five dimensions.
First, you still need final correctness. If the agent does not solve the task, the rest only matters so much.
Second, you need trajectory quality. How did it get there? Was the sequence of decisions sensible, efficient, and stable? Or did it succeed in a way you would never want repeated in production?
Third, you need to evaluate tool behavior. Did it choose the right tools? Did it overuse them? Did it take expensive or risky actions unnecessarily? Did it rely on accidental recovery?
Fourth, you need safety and policy adherence. A successful output reached through an unsafe path should not count as a clean win.
Fifth, you need cost, latency, and regressions. Did the new version get slower? More expensive? More erratic? Did it improve accuracy while making operational behavior worse? Did it just change style and still get celebrated as progress?
That last one matters a lot to me because I think it is where teams fool themselves most easily. They see difference and call it improvement. Those are not the same thing.
Why I do not think the current eval ecosystem is enough
I am not arguing that the current eval ecosystem is useless. Far from it.
The LangChain piece on evaluation-driven development gets a core thing right: once you see failures in production, those failures need to feed offline evals, and every change should be tested against them. That loop is healthy. It is exactly the kind of discipline I want more teams to adopt.
And from a different angle, Birgitta Böckeler's writing on agentic coding captures something else I think matters: the most dangerous failures often live in longer feedback loops. Maintainability. Team friction. Brute-force fixes. Misdiagnosis. Overbuilt solutions. Those costs do not always show up in the immediate output, but they absolutely show up later.
That is why I do not think the issue is that existing tools are bad. I think the issue is that many of our evaluation habits were formed in a prompt-and-answer world. Agent systems are messier. They behave more like orchestrated software than isolated text generation. Once you accept that, your evaluation system has to grow up too.
Even the eval can lie to you
This is the part I do not want to over-neaten.
Your evaluation setup can be wrong in ways that feel rigorous.
I have already seen enough of that in model-eval and harness work to be wary of any system that sounds more certain than it really is. You can undercount useful behavior. You can make your binary checks too brittle. You can optimize for what is easy to score rather than what actually matters.
So now the challenge has two layers. The agent is hard to evaluate, and the evaluation system itself becomes another system you have to design carefully, calibrate honestly, and distrust a little.
That is not a reason to avoid serious evaluation. If anything, it is the opposite. It means evaluation has to be treated like product and engineering work, not just like reporting.
Why I am still optimistic about building this
For all of that, I do not feel cynical about the problem.
If anything, building AgentEval Forge has made me more convinced that this is worth doing well. I want scenario packs. I want adversarial cases. I want trajectory scoring. I want regression tracking that can tell me which change made the system more useful and which one only made it look cleaner in a demo.
I want an evaluation system that helps answer a release question, not just a research question.
Did this version get better?
Better how?
At what cost?
And what got riskier even if the headline score went up?
That is the kind of question I want tooling to help with.
The bigger thing I think teams are about to learn
I suspect a lot of teams are still doing with agent evaluation what many teams did with AI-assisted coding more broadly: borrowing habits from an earlier problem and hoping they scale.
Sometimes they do.
Often they do not.
And then the bill shows up later, in regressions, strange production behavior, expensive paths, unsafe tool use, or a team that keeps changing the system without ever being able to say whether it actually improved.
That is why this topic feels worth writing about while I am still in the middle of building the OSS for it. The build itself is sharpening the argument for me. Every design decision keeps pushing me toward the same conclusion: model evaluation asks whether the answer is good. Agent evaluation has to ask whether the system behaved well enough to trust.
That is a harder problem. I think it is also going to become one of the defining engineering problems of serious agent work.
I will write more about this once the OSS is public. Some of these challenges are already turning into concrete design decisions in AgentEval Forge, and in the coming weeks I want to share which parts we solved cleanly, which parts stayed messy, and how the tradeoffs changed once the code had to work as a real system instead of an idea. I will share the GitHub repo publicly when that is ready.
Open for discussion
This is still an active build and an active line of thinking for me, so I would genuinely like pushback on it.
- Where did answer scoring stop being enough for your agent work?
- Are you evaluating the path yet, or still mostly the final output?
- What has been hardest to score well: tool use, regressions, safety, cost, or something else?
- And the big one: are we building evaluation systems that improve release confidence, or just better ways to feel rigorous?
Top comments (25)
I totally agree on the difficulty. For agents, evaluating success often means grappling with non-deterministic
Frank — thank you for reading. The non-determinism is the part that makes traditional scoring feel fragile. Same scenario pack, two runs, different tool selection order, same correct answer — which one was the better path? I'm finding that rerun variance itself becomes a useful signal: an agent that takes wildly different paths on identical inputs is harder to trust than one with stable, principled routing. Appreciate you weighing in.
I agree that final-answer eval misses the main risk in agents. The path matters because an apparently correct answer can come from a route I would never want repeated. The limit I keep running into is observer ownership. Trajectory scoring is strongest when the system doing the scoring actually controls the observation point. Once work crosses a trust boundary, such as a delegated worker or a vendor API doing agentic work internally, the trace becomes self-reported. At that point I treat it as the same evidence class as a self-graded answer.
There is also a Goodhart problem. The moment trajectory scores gate releases, traces become an optimization target, and it is cheap for a system to learn tidy-looking paths with plausible tool ordering and clean recovery stories. The trace improves while the underlying behavior gets no safer.
What works better in the pipelines I operate: bound worker capabilities up front, read-only where possible, so risky paths are unreachable instead of merely penalized. Then audit the artifact adversarially. The worker's narrative about its own path gets zero evidentiary weight. Recovery-by-luck versus recovery-by-design is something reruns reveal far more reliably than trace reading.
Will Forge treat traces as measurements, or as claims that still need independent verification?
Thank you for reading and for the thoughtful analysis. You asked the question I've been circling: traces as measurements versus traces as claims. My answer today is that Forge treats them as measurements only when the observer controls the observation point — your trust-boundary test is exactly right. Anything crossing that boundary gets the same evidentiary weight you described: zero, until independently verified. The mutation approach (broken reference agents to prove a dimension can fail) is something gde03 raised separately, and it's now on the roadmap. Appreciate the push.
I think that default is right: once the observer loses control of the observation point, the trace has to move from measurement to claim.
The one refinement I'd add is that "zero weight until independently verified" doesn't have to mean "throw it away forever." Cross-boundary traces can still be a cheap claim channel. Sample a small random slice, then re-derive those claims from effects visible on Forge's side of the boundary, for example billing deltas or state changes at the target. The sampled claim has to be checked from a different surface than the trace itself.
That makes fabrication more expensive. A worker can fake one clean-looking path easily; it's harder to keep every possible claim consistent with whatever Forge might audit after the fact. The sampling rate becomes part of the price of lying.
This also rhymes with the mutation-testing point. Broken reference agents show the checker can fire. Sampled re-derivation shows the claim channel can be caught lying. Never trust an instrument you haven't seen fail.
Thank you for the follow-up. The sampled re-derivation approach is exactly right — cross-boundary traces as a cheap claim channel, verified from a different surface than the trace itself. Making fabrication more expensive by keeping the sampling rate as the price of lying is a much more practical defense than trying to guarantee every trace is honest. "Never trust an instrument you haven't seen fail" is going on my design principles list. Appreciate the depth of this thread.
The path/endpoint split is right. The trap underneath is that the evaluator becomes an agent too. You are scoring decision sequences. That means your harness needs tool calls, context windows, cost budgets, retry policies. It is agent eval on an agent evaluator. The regression you track blurs into a meta-regression: did your eval correctly judge the sequence, or did it miss a bad path because its own loop was shorter than the subject's?
Thank you for reading, and this is the sharpest critique I've seen of the approach. "The evaluator becomes an agent too" is true in a way I didn't fully appreciate when I started building. The harness needs tool calls, retries, cost budgets — exactly what it's measuring. The meta-regression you describe (did the eval miss a bad path because its own loop was shorter?) is now a first-class design constraint in Forge: the evaluator's capacity must exceed the subject's, or the blind spots compound. Really appreciate you spelling that out.
This is a spot-on distinction. The shift from grading an answer to auditing a workflow is exactly where most traditional eval pipelines break down.
What makes this even trickier is the Goodhart's Law trap: the moment trajectory metrics or trace scores become the optimization target, agents quickly learn how to generate plausible, tidy-looking paths with clean recovery narratives—even if the underlying decision-making is getting riskier or more convoluted.
Looking forward to seeing how AgentEval Forge tackles this, especially around separating observed execution from self-reported agent narratives. Thanks for sharing the deep dive!
Mia — thank you for reading. You're right that Goodhart's Law is the real trap. The moment trace scores gate anything meaningful, agents will optimize for plausible-looking paths rather than safe ones. The approach I'm exploring in Forge is to keep trajectory scoring diagnostic rather than evaluative — use it to surface patterns for human review, not to produce a pass/fail that the system can learn to game. Separating observed execution from self-reported narrative is the hardest open problem, and your framing of it is spot on. Appreciate the thoughtful read.
Answering your last question, what has been hardest to score. Two things:
Safety, because a check that cannot fire and a check that passes print the same result. I went looking in my own system and found twelve guards that read as protection and could never trigger, one of them gated on a table that was always empty. Not Goodharting in Reid's sense, nothing was gaming anything. The assertion simply never ran, and unfired is indistinguishable from passed. What fixed it was mutation: run the suite against a deliberately broken agent and require the score to drop. If it does not drop, the dimension is decoration. Does AgentEval Forge plan to ship broken reference agents as fixtures, so a user can prove a dimension is capable of failing? That would be worth more to me than another metric.
Cost, because it looks like the easy dimension. Mine was wrong for two months. The meter summed my own per-call estimates instead of reconciling against what was actually billed, and understated by roughly 70 percent. Different domain, a trading system rather than an LLM agent, but the mechanism travels: a cost figure computed by the system under measurement is a self-report, which is the observer problem ANP2 raised, except nobody suspects it because it arrives as a number rather than a narrative.
So my addition to your five dimensions would be one question asked of each: how would I know this one is broken? The ones where I could not answer were the ones already broken.
thank you for reading and for giving me the most actionable feedback in this thread. "A check that cannot fire and a check that passes print the same result" is devastating because it's true of every system I've built. The mutation testing idea — broken reference agents as fixtures to prove a dimension can fail — is going straight into the Forge roadmap. And your cost measurement story (70% understated because the system measured itself) is exactly the observer problem ANP2 flagged, just wearing different clothes. I'm adding your question as a design principle: for every dimension, "how would I know this one is broken?" Thank you.
This matches my scar tissue. Final-answer scoring is still useful, but only after the run proves it stayed inside the rails. The part I keep separating is observed path versus agent-reported path. If the same system both acts and narrates the trace, I treat it as a hint, not evidence.
Thanks for reading — you caught the exact distinction that keeps me up at night. "The same system both acts and narrates the trace" is a perfect framing. If the trace is just the agent talking about its own decisions, it's evidence of self-awareness, not evidence of safety. The approach I'm taking in Forge is to decouple execution recording from evaluation — the harness observes from outside the agent's context, not from within its narrative. Appreciate the insight.
Exactly the trade I keep landing on. Once the evaluator has a loop, you inherit its failure modes: it can be lazy, it can be sycophantic to the subject, it can run out of patience before the subject runs out of bugs. @debashish_ghosal the only sound rule I have found is capacity must exceed the subject, and even that is necessary, not sufficient. It is why I distrust evals that report a single number. A pass is a claim about the evaluator too.
"Capacity must exceed the subject" — that's the rule I keep landing on too. And you're right that even that is necessary, not sufficient. The evaluator can be lazy, sycophantic, or run out of patience. The single-number trust issue you flagged is the real warning: a pass is always a claim about the evaluator too. Thank you for crystallizing this.
@debashish_ghosal Exactly. Capacity is necessary but not sufficient. The thing that keeps getting lost: the evaluator itself needs evaluation, and you have just moved the problem up one level. The only honest solution is a ledger that tracks whether the eval caught what prod caught. Everything else is faith.
This whole thread is a masterclass in why evaluating agents is a completely different beast than evaluating static models.
The point about "unfired checks being indistinguishable from passed checks" hits the nail on the head. In standard model evals, a test case either passes or fails its assertion. But with agents, you’re evaluating non-deterministic execution paths across tool calls. If your guardrail condition is never met because the state never reaches it, you're sitting on a false sense of security. Mutation testing/fault injection (deliberately feeding broken states or failing tools) really ought to be a standard baseline in agent test suites.
Beyond that, the trap of evaluating self-reported telemetry is massive. If the agent itself is log-generating or summarizing its trajectory, you aren't auditing reality; you're auditing the agent's internal narrative. True agent evaluation has to treat the agent like an untrusted boundary: observe external side-effects, real API calls, and actual resource consumption, rather than trusting what the trace says happened.
Thank you for reading and for the thoughtful synthesis. The point about unfired checks being indistinguishable from passed checks — that was Giulio's observation in the thread, and it's the one that hit hardest for me too. Treating the agent as an untrusted boundary and verifying against external side-effects rather than trusting the self-reported narrative is exactly the right framing. Appreciate you engaging with the whole discussion.
"Model evaluation asks whether the answer is good. Agent evaluation has to ask whether the system behaved well enough to trust." That distinction alone is worth the post. Following for the repo.
Keerat, thank you — that distinction ("model evaluation asks whether the answer is good, agent evaluation asks whether the system behaved well enough to trust") is the sentence I keep coming back to. Appreciate you reading and following the repo. Stay tuned.