Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument: with agents, you are not just judging an answ...
For further actions, you may consider blocking this person and/or reporting abuse
The field-testing discovery is the part that matters most. Searching 150+ repos and landing on 19 testable agents, with most being "too heavy, too coupled, or no clear entry point" — that is not a sampling problem, it is an ecosystem problem. Most agents are not eval-ready because they were not built to be observed.
The adapter contract you described — scenario input, allowed tools, disallowed tools, budget, never seeing the expected answer or scoring thresholds — is the right isolation boundary. No ground-truth leakage is the minimum bar and most eval frameworks do not meet it. The five adapter surfaces (subprocess, Python import, HTTP, LangGraph, PydanticAI) also tell you something about the fragmentation in the agent space: there is no standard invocation contract, so the harness has to bridge every surface.
The 17 deterministic scorers vs 11 LLM-as-judge split is interesting. In my own work on verification gates, the deterministic checks caught the structural failures (missing tool calls, wrong artifact paths) while the LLM judge caught the semantic failures (right tool, wrong intent). The ones that broke the clean version of the story were usually the cases where the deterministic scorer said pass but the LLM judge said the agent solved a different problem than the one posed.
Curious about the adversarial generator — does it produce perturbations of existing scenarios or generate entirely new failure modes? The distinction matters because generated adversarials tend to cluster around known failure types while real-world agents fail in ways nobody thought to test for.
Thanks for reading — and the point that a 9% pass rate across 19 repos is an ecosystem problem, not a sampling problem, is the real takeaway. As I am attempting to build a generic eval system which was my original intent, I am realizing its harder than what I thought it would be. If was solely focused on 1 or handful of agents that I or someone developed, I can make ecosystem that recreates the target env that the agent would be used in. Hence the question lingers on my mind - is it just too hard to build a generic eval or do I need to invest more where ecosystem is created from scratch to fit the agent, in a very short time, and then torn down - I don't know.
On the adversarial generator: today it's mostly perturbative — it mutates existing scenarios (rewording, tool-order, budget stress, injected noise) rather than inventing novel failure modes. You're right that this clusters around known failure types. My plan for the next pass is to seed it with real field-test failures (like the blank PydanticAI completions) so it learns the failure space nobody thought to test for.
Debashish,
The gap between "generic eval system" and "eval system that works on my agents" is where most of the hard problems live. The perturbative vs novel distinction matters here: a generator that perturbs known failure modes will find known failure modes. The 9% pass rate against field agents tells you the perturbations are not covering the actual attack surface — real agents fail in ways that are structurally different from what a template engine generates. Starting with your own fleet's failure catalog and building outward is probably the right sequencing.
Xiao Man, thank you for the follow-up. You're right that the perturbative-vs-novel distinction is where the real gap lives. A generator perturbing known failure modes will only find known failure modes, and the 9% pass rate is telling me the perturbations aren't covering the actual attack surface. Starting from my own fleet's failure catalog and building outward is the right sequencing. Appreciate you pushing on this — it's helping me prioritize the adversarial generator roadmap.
One caution before you build outward from the catalog: the catalog only contains failures you caught, so a generator seeded from it inherits that blind spot. Worth holding back a slice of production traffic as a control, so you can tell 'generator found nothing new' from 'attack surface actually covered.'
Xiao - absolutely, accurate. The synthetic traffic, didn't add the intended value. As it was designed to cover all traffic shapes which were not in the HF corpus, the synth traffic boosted the coverage of all detectors but didn't lead to any groundbreaking gap discovery - so, yes, it continued to inherit the blind spot. I had considered using a technique to generate a smaller corpus to cover more traffic variations, but then decided to pursue real agent integration which turned out to be the right bet that paid off. I agree with your idea of using a control technique. Getting hands on real production agents traffic in public space that's doing some serious workload is a big challenge right now
Hands down one of the most honest and insightful posts on AI Agent evaluation I've read on DEV! 👏 The 9% pass rate finding is a huge eye-opener—it completely shifts the conversation from 'model intelligence' to 'software environment and integration reality.'
To answer your question: currently, we are mostly relying on LLM-as-a-judge for final outputs, but trajectory/tool-call failures are definitely our biggest blind spot. Sticking strictly to the 'Safety > Correctness > Efficiency' hierarchy in your PRD is such a solid architectural choice. Looking forward to part 4 of the series!
Thanks for reading — I hear you on the trajectory/tool-call blind spot; that's exactly why I built deterministic trajectory scorers on top of the LLM judge. A final answer can look right while the tool calls were wrong.
Part 4 is tracing the gap where the LLM judge passes but a deterministic trajectory check flags a problem. If you've found a pattern that catches that cleanly, I'd love to hear it.
Exactly! A polished answer often masks a chaotic trajectory.
What usually catches this for us is tracking state mutation logs between tool calls. If a tool call doesn't actually mutate or advance the context state as expected, it's flagged as a 'phantom step' regardless of what the final LLM response claims.
Excited to see your approach in Part 4!
Hosein, tracking state mutation logs as "phantom step" detection is a genuinely clever approach. If a tool call doesn't advance the context state, flagging it regardless of what the final output claims is exactly the kind of trajectory-level check that catches the gap I'm interested in. I'm going to think about how to incorporate something similar as a deterministic scorer in the harness. Excited for Part 4 as well — appreciate you sharing your technique.
The "field testing broke the clean version of the story" framing is the most honest thing in this piece. Anyone who's built an eval framework against mock agents knows how clean it looks â and how quickly that illusion collapses when you point it at real code.
The database-bound agent problem is the one I'd push back hardest on from a different angle. You note that database-bound agents required infrastructure you couldn't provision per-agent. That's exactly the tradeoff we're building moteDB to eliminate: an embedded database that each agent instance instantiates locally, no server, no credentials, no provisioning latency. The agent gets a consistent store with full query capability; the harness doesn't need to know the database exists.
Where it gets interesting for eval is the state leakage problem. When a database-bound agent runs twice against the same scenario, does it see its own previous writes? In production it would â but in an eval harness, isolation is usually required. With an embedded DB per agent instance, you get natural isolation by construction: each agent gets its own data directory, wiped between runs. That's harder to fake with a shared Postgres instance.
The 19 agents out of 150+ that were actually testable is a real number for anyone building in this space. The selection pressure toward agents that are easy to evaluate is already shaping what gets built â agents that expose clear run() interfaces, that don't need credentials, that have observable state. That's not a bad thing, but it's worth naming explicitly.
Thank you for the thoughtful read. The moteDB angle on the database-bound agent problem is genuinely interesting — per-agent embedded DB with natural isolation by construction is a much cleaner solution than provisioning shared infrastructure per eval run. The state leakage problem you described (does the agent see its own previous writes?) is exactly the kind of isolation question that separates a trustworthy harness from a convenient one. Appreciate you sharing what you're building — the selection pressure toward eval-friendly agent design is a real dynamic worth naming.
A robust evaluation framework should measure more than task completion. Observing how an agent responds to failures, ambiguous inputs, and changing conditions provides a far better indication of production readiness than success rates on controlled examples alone.
Thanks for reading — I completely agree that success on controlled examples overstates production readiness. Observing how an agent responds to failures, ambiguity, and changing conditions is exactly where a harness earns its keep. Appreciate the framing.
The field-test result is the useful warning here. If the 9 percent pass rate mostly measures adapter realism, the eval harness is already telling you something before it ranks agents: your test surface is still too polite. I would probably make "can this agent be invoked outside its author's environment?" a first-class score instead of treating it as setup noise.
Exactly right, and honestly, I was a bit naive and optimistic, the tool will score higher pass rate for higher starred agents and maybe successful in catching issues in lower popular ones - this seemed like a good hypothesis. And, at the same I had pivoted more towards putting this in CI pipeline so I can push the eval more to the left. With the lower pass rate, and adapter realism and the extra shims I had to build, I am considering exactly as you said - will agent eval be challenging if folks use outside my intended environment or do I consider making this part of a common agent development environment? I am definitely not aiming to be a SWE Bench analog to come baseline agents that you have already invested in eval-ing other ways. Anyway, I have to think through these and solidify my use cases as to what this should be and what it should not be. Setup noise turned out to be very high.
Are all 19 agents tested on identical hardware/OS? You mention local MLX on Apple Silicon, but what about CI runners (Linux/Docker)? Do you publish detailed reports per agent (environment vars, dependency versions, adapter logs)? Can readers reproduce or rerun evals on different machines/clouds to verify the 9% baseline? Raw test reports are linked, but how granular is the failure data—can someone tell if their agent fails due to adapter quirk vs. real incompatibility?
Thank you for reading — and these are exactly the right questions, because reproducibility is the real test of a harness like this.
Honest answer: no, the 19 agents did not all run on identical hardware/OS. The rapid sweeps ran locally on Apple Silicon with MLX, while the CI-gated scenarios ran on Linux/Docker runners. So treat the 9% as a floor, not a machine-to-machine guarantee. The cloud LLMs ran on the provider hardware
On granularity: the raw field-test reports in docs/field-test-reports/ cover per-agent adapter, integration step, and failure classification. What I don't yet publish is a full per-agent manifest (dependency versions, env vars, adapter logs) — that's a real gap and it's on the roadmap - I had the files on my mac but I need to go through and sanitize. Each scenario JSON does record the exact invocation payload, so a rerun on your box should pass or fail for the same structural reasons. If you do try to reproduce them, I'd genuinely love to hear what breaks.
The 9% pass rate measuring adapter realism instead of agent quality is the key finding here, it would've been easy to publish that number as "most agents are weak," but correctly diagnosing a blank completion as a compatibility failure, not a quality failure, is the harder and more honest read.
The cheap-vs-better tier result reframes eval work generally: if the expensive judge produces identical outcomes to the cheap one, the bottleneck was never the judge, it was whether the harness faithfully exercised what it was judging.
"Testing inside the author's own environment against the author's own examples is the worst possible way to evaluate whether something actually works" is the line that generalizes furthest past this project.
Curious about the blank-completion PydanticAI wrappers, is the current guess that the adapter's invoking correctly but the agent's own output path silently short-circuits, or is it still unclear where the failure originate
Thank you — and you're right that calling the 9% "compatibility, not intelligence" is the harder and more useful read.
On the blank PydanticAI completions: it's still partly conjecture, but the evidence leans toward the agent's own output path short-circuiting, not my adapter. In the failing cases the adapter receives a valid completion object with empty content and no raised error, and the agent's trace shows it stopping right after tool dispatch — the model call never really happens. My working hypothesis is a schema mismatch between expected structured-output and what the internal loop emits. It's not fully root yet, but I'll try to pin it down in a follow-up.
Real agents broke the clean version of the story" — this is so accurate! 😂
Testing agents in a vacuum is easy, but in production, they hallucinate tool-calls or leak PII. This exact chaos is why my team open-sourced Aegisora to intercept rogue agent behaviors in real-time. I'd love to see how your eval harness handles edge cases like semantic prompt injections!
Eray, thank you for reading. Semantic prompt injection is exactly the class of edge case that makes me nervous about eval completeness. My current adversarial generator handles rewording and tool-order perturbations, but genuine injection attacks that exploit the agent's own reasoning loop are a gap. Aegisora's runtime interception approach is interesting — the eval-at-build-time vs. guard-at-runtime distinction is one I'm still trying to think through clearly. I'd love to learn more about how you handle that boundary.
The honesty in this write-up is refreshing. Hitting a 9% pass rate because the field test mostly measured "adapter realism and environment setup" rather than agent intelligence is the exact wall everyone runs into when trying to evaluate third-party repos.
The point about config chaos—module-scope API calls, hardcoded model imports, and transitive dependencies like ormsgpack—highlights why standard CI/CD paradigms break when applied to AI agents. Making "Safety > Correctness > Efficiency" a non-negotiable hierarchy in the PRD was a great architectural choice. Until we standardize agent interfaces and execution boundaries, harness builders are going to spend 80% of their time writing compatibility shims rather than scoring logic. Congrats on getting agent-eval-forge onto PyPI!
To answer your question at the end: what breaks first when trying to evaluate third-party agents is almost always side-effects and environment assumptions. The moment an agent assumes a specific directory structure, a local database connection, or module-scoped model initialization, standard test runners fall apart.
The decision to build dynamic adapters for LangGraph and PydanticAI while using [unk] fallback patterns and local MLX models for rapid sweeps makes agent-eval-forge super pragmatic. Looking forward to part 4 of the series to see how you evolve trajectory scoring!
Thanks for reading — and for taking a shot at the closing question rather than just reacting to the headline number. Side-effects and environment assumptions breaking first maps 1:1 to what I saw: module-scope API calls, hardcoded directory structures, and transitive deps like ormsgpack were the top breakers, none of them about agent intelligence.
The Safety > Correctness > Efficiency hierarchy was the one decision I refused to compromise on in the scoring layer. Part 4 is in progress; trajectory scoring is the honest hardest problem left, and I'll publish the raw data alongside it.
nice
@debashish_ghosal identical 9/95 on both tiers might say more about the blank completions than about the judge, with no output both models score the same nothing. I trip over that in my own tooling.
nice work
Thanks! If you check out the repo and try it against an agent of yours, happy to help with any edge cases you hit.
Hello there,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
Best regards,
Ono
For the module-scope ChatOpenAI() case, a less fragile route than a pre-import monkeypatch on init is the OPENAI_BASE_URL env var. ChatOpenAI reads it at construction, so running the agent in a subprocess with the env pointed at your MLX endpoint redirects the model call without touching their imports. It does nothing for the ones writing to /root at import time, unfortunately.
The silent no-op cases feel like the bigger long-term risk for a harness like this. uv sync succeeding at the root while the real pyproject.toml sits in a subdirectory means the run looks healthy until the scores come back odd. Did you end up making the harness fail loudly when venv resolution doesn't line up with the declared entry point, or is that still manual?
@jkming the silent no-op is the one that scares me too. Mine was a review step passing against an older draft of the same file, clean report, meaningless. Now it has to prove which version it ran on.
The part that landed hardest is that field testing against real agents forced this into an integration reality check instead of a scoring project. Judging a whole run, not just a final answer, is the right frame, but adapters and sandboxes eat more of the work than the metric layer ever does. I have seen evals look green against mock agents and then fall apart the moment a real tool returns a messy partial result. Did any of the five adapters end up costing more maintenance than the scoring layer itself?
The 9/95 result makes adapter realism look like a first-class evaluation dimension, not setup noise. I would split the CI result into at least three gates: invocation compatibility (import, environment, tool contract), policy/safety compliance, and task quality. Otherwise a blank completion or import-time side effect can collapse into the same score as a genuinely incorrect trajectory.
For the field layer, a useful regression fixture might be a deliberately awkward agent: nested project root, delayed tool response, unavailable optional dependency, and a restart after partial output. If the adapter can resume and preserve the failure classification, the score becomes actionable. The current repo caveat that compatibility is not yet ranking quality is exactly the distinction I would keep visible in CI.
Thanks — I really like the 3-gate split (invocation-compatibility, policy/safety, task quality). Right now those are conflated, and you're right that a blank completion can collapse into the same score as a genuinely wrong trajectory. The deliberately awkward agent fixture (nested root, delayed tool, restart after partial output) is a great regression pattern — I'll be stealing that for the field layer.
Hello there,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
This is my whatsapp number: +81 70-9427-3751
Best regards,
Ono
The shift from judging an answer to judging the whole run is the part most eval setups skip, and it is exactly where agents fall apart. Once the path and tool calls matter, a passing final answer can still hide a broken trajectory that surfaces later under load. When you scored the 20 journeys, did you weight trajectory correctness separately from outcome correctness, or fold them into one number?
The cheap tier and the better tier both scoring 9/95 is the whole post in one line: you can't judge your way out of a broken harness. A smarter judge just scores the blank output with more confidence. Everyone spends on the judge; the money was in the adapter the entire time.
Be cautious when using online investment platforms, even if they have positive reviews. I lost my deposit within a few days. Fortunately, CYBERBERSPY assisted me in recovering $50,000 through what I found to be a professional recovery process.If you're considering any recovery service, always verify its credentials, be wary of anyone requesting upfront payments, and proceed carefully before sharing money or personal information. i highly recommend CYBERBERSPY
The reset boundary is the part that broke our “clean” eval. A subprocess adapter that restarts every scenario is reproducible, but it erases search indexes, provider cache reuse, session memory, and tool-process state; reusing the process has the opposite problem and can leak one case into the next.
We hit this benchmarking BitFun's persistent runtime. In the SWE-Bench-Pro round, the provider-reported KV-cache hit rate averaged 98.67%, so tearing everything down per case would misstate both cost and latency. I'd make reset semantics part of every adapter manifest — fresh process / workspace / session / warm indexes — and report cold-start and steady-state separately. Otherwise the harness quietly rewards stateless wrappers or leaked state depending on one implementation detail. Are your subprocess and HTTP adapters normalized to the same reset boundary, or is that still surface-specific?
Hello there.
I have a good opportunity for you.
If you'd be interested in hearing it, please let me know.
Whatsapp:+81 70-9427-3751
I'm looking forward to hearing from you.
Best regards
Ono