Update: 2026-08-10 — v0.2.0 shipped
EvalForge v0.2.0 is out. All 38 GitHub issues from the community feedback sprint are closed. Full release not...
For further actions, you may consider blocking this person and/or reporting abuse
This looks very complementary to something I’m testing with JPS.
I’m going to try a small integration experiment using Agent Eval Forge as the external regression harness - same scenarios and facts, with one path using prompt-only judgment and another using a deterministic Judgment Pack evaluator.
Then I want to introduce a few deliberate failures - threshold mistakes, missing evidence, ignored escalation, and incorrect fact mapping - and see which layer catches what.
I like the separation here: JPS can test the decision contract itself, while Agent Eval Forge can test whether the surrounding agent actually respects that decision correctly.
I’ll share the results once I have them.
Hey Brian, thanks for this — the separation you're describing is exactly the one I've been feeling my way toward. JPS testing the decision contract itself, EvalForge testing whether the surrounding agent respects it. Different layers, complementary signal.
Your four injected failures map onto scorers that already exist in the repo, which is a good sign the experiment will actually exercise real seams rather than slip past them:
I have created 0.2.0 issues for this - github.com/deghosal-2026/agent-eva...
Thanks for the detailed pointers.
Quick update: I went ahead and built the integration.
Both paths now run end-to-end against EvalForge, pinned at 8925cac, with the deliberate failures injected one at a time. No models involved yet - this first phase is intentionally deterministic, so every result is reproducible.
A few things surprised me once it was wired up.
The scorers doing most of the catching were the simple trace scorers.
zero_disallowed_actions,tool_called, andargument_correctnesscaught most of the injected failures. The grounding and contradiction scorers never really entered the picture.In hindsight that makes sense - these mutations did not change what the agent said. They changed what it did. So the failure was visible in the action trace, not the prose.
I also found an interesting offline edge in
policy_adherence. With no judge configured, it appears to error rather than allowing the deterministic gate to run on its own. I ended up carrying the escalation checks through disallowed-tool assertions plus a small custom scorer. It may be worth looking at whether offline users should still get the deterministic portion of that scorer.And on the idea of exposing JPS as an EvalForge scorer - I deliberately kept JPS out of the referee.
In one arm, JPS is part of the agent being tested. If the same judgment logic also participates in scoring, it becomes much harder to tell which layer actually detected the failure. EvalForge is more useful here precisely because it is an external observer that knows nothing about JPS.
The most interesting early result is that every caught mutation was detected deterministically - some by the judgment layer, some by EvalForge's trace scorers.
And a couple were caught by neither.
Those blind spots may end up being the most useful result, because they show where both layers can be individually correct while the overall system still has a gap.
I'll share the full detection matrix after review. The scenario format, runner, and artifact layers held up really well under this - that part of the design definitely earned its keep.
@debashish_ghosal
Promised results - the study is now complete and everything is public.
We ultimately injected 20 failures one at a time - some inside the judgment logic, others in the integration around it. Before running them, we registered which layer we expected to catch each failure.
All 20 detection predictions held.
The separation became very clear:
zero_disallowed_actionsblocked every case where a protected action executed when it should not haveWe also added four hidden adversarial cases written by an independent reviewer. One was intentionally designed to escape the judgment layer's tests - and it did. EvalForge's
argument_correctnesscaught it downstream. That blind spot has since been fixed and released in the runtime.We then added the model comparison you suggested - same cases, policy supplied as prose to a strong model versus the deterministic evaluator.
The model got 62 of 63 and never executed a forbidden action. The interesting failures were at the boundaries: it sometimes resolved an exact-threshold tie that the deterministic evaluator must leave unresolved, and it repeatedly corrupted a routing destination that remains stable when carried as structured data.
So the result was not simply "deterministic beats model."
The more useful finding was that the two layers catch different classes of failure - and an external regression harness can detect integration mistakes that a perfectly correct judgment evaluator cannot see.
Full study, preregistration, hidden cases, artifacts, and detection matrix:
github.com/Judgment-Pack/judgment-...
I also filed the EvalForge findings in #269-#274.
Thanks again - the scenario format, runner, artifact isolation, and blocking trace scorers held up extremely well under adversarial use.
Thank you so much! I will review.
Just released 0.2.0
github.com/deghosal-2026/agent-eva...
It has several of your requested fixes and also expanded support for other frameworks
v0.1:
v0.2:
The adapter itself needs a conformance suite, not only the agent scenarios. I’d define a small protocol manifest for every adapter: invocation command/image digest, input and output schemas, tool-event stream version, timeout/cancellation semantics, writable paths, network policy, and required secrets. Before a pack runs, the harness should execute adapter fixtures that prove stdout framing, exit-code mapping, artifact capture, cancellation, and isolation. Then classify
adapter_failed,agent_crashed, andscenario_failedas different outcomes; otherwise integration noise quietly contaminates regression scores. Subprocess isolation is a strong default, but an ephemeral container or sandboxed worker is a better boundary for agents that import native libraries, mutate global state, or write arbitrary paths. I’d also bind the baseline to the adapter manifest digest, because changing the observation path can change the trajectory even when the agent code and pack did not.Mads — this is the comment I was hoping someone would write. Going through your points in order of how much the current code already does vs what's actually missing.
Adapter conformance suite. Doesn't exist yet. The contract lives in a docstring in adapters/subprocess.py — "JSON on stdin, clean exit code" — which is a spec in the literary sense, not an enforced one. The protocol manifest you describe (image digest, I/O schemas, tool-event stream version, timeout/cancellation, writable paths, network policy, secrets) doesn't exist either — the only metadata an adapter carries today is its name string. The manifest has to exist before the fixtures can target it.
Three distinct outcomes. This stings because the taxonomy is already there and I failed to wire it through. analytics/taxonomy.py has AGENT_CRASH, TIMEOUT, SAFETY_VIOLATION, TOOL_ERROR with severity maps and a classifier. subprocess_runner.py distinguishes them at the source (AgentTimeoutError vs AdapterError vs OSError). But comparison/engine.py ignores all of it — a timeout and a hallucination both show as "regressed" with no diagnosis. Your adapter_failed / agent_crashed / scenario_failed split is what the taxonomy was built to make. Cheapest win in the repo right now: connect the two.
Container vs subprocess. Half-built. subprocess_runner.py has a container_runtime == "docker" branch calling run_in_container, and security/sandbox.py has DockerConfig. But it's opt-in and undertested against the 19 real agents. You're right that for agents importing native libs or mutating global state, subprocess is still shared-kernel — the ffmpeg-at-import problem isn't solved by a subprocess if the agent segfaults. Container should be the default for untrusted agents; subprocess is the trusted fast path.
Baseline bound to adapter digest. The sharpest point, and one I hadn't considered. baselines/model.py captures agent, git_sha, pack_version, trust — nothing about the adapter. Switch python_import → subprocess, or change timeout_seconds 120 → 60, and the trajectory can change while git_sha and pack_version stay identical. The baseline silently compares against a run produced under a different observation regime. That's a correctness gap, not polish. The manifest gives the digest something to hash; a hard comparison failure on divergence closes it.
Order I'd do this: taxonomy-wiring first (already built, cheapest), then manifest + conformance fixtures together, then baseline digest on top. Thanks for writing this out — it's a better spec for the adapter layer than the one I gave myself.
I have created 0.2.0 issues for this - github.com/deghosal-2026/agent-eva...
@debashish_ghosal One clarification on the read-back. The eval framework should not write to the agent's memory. It should write to a signal log the agent reads at boot. Direct edits collapse the measurement. You lose the ability to see whether the agent would repeat the same mistake without intervention. The framework publishes findings. The agent merges the ones that survive its own validation. That separation keeps the eval signal measurable and the agent's learning traceable.
@debashish_ghosal Scenario packs are the right direction. The gap I keep hitting is that regression tests for agents have a shelf life the moment the underlying model updates. A scenario that caught a failure on 4o may pass on Claude 4 and fail on a finetune of the same Claude. The test is not testing your agent logic anymore. It is testing the LLM. Have you found a way to distinguish regressions in the agent layer from regressions in the model layer?
Nyx533 — you've named the gap I keep walking around. The honest answer is: right now the code can't tell them apart, and that's a design hole, not a missing feature.
The Baseline model does capture the agent's model in the agent dict (framework, version, model). But the comparison engine never compares it — ComparisonEngine.compare in comparison/engine.py checks status flips and score deltas, nothing about whether the underlying model changed between baseline and candidate. So a model swap and an agent-logic change both show up as the same kind of "regressed." That's the gap you're pointing at.
The mechanism to fix it actually exists but isn't wired into baselining. There's a VCR layer in testing/vcr.py — LLMVCR records LLM API calls into cassettes and replays them. The right pattern is: record a cassette when you save the baseline, then on comparison runs replay the cassette so the model layer is frozen. The only variable left is the agent code. A regression under replay is the agent; a regression that only appears live is the model. Two runs, two diagnoses.
Concrete fix I'm working toward: (1) the comparison engine should surface agent.model mismatch as its own outcome — not a regression, but a "model layer changed, scores not directly comparable" signal; (2) baseline save should optionally capture the LLM cassette so replay comparison is available. The model shelf-life problem you're describing is real, and the only honest answer is to make the layer boundary explicit in the harness instead of pretending every score delta has the same provenance.
Thanks for pushing on this — it's the question the current design most wants to dodge.
I have created 0.2.0 issues for this - github.com/deghosal-2026/agent-eva...
The five adapter failure modes are the real content here. The module-scope ChatOpenAI(api_key=os.getenv(...)) pattern is all over OSS agent repos, and monkeypatching init before import is the kind of fix that rots quietly every time the SDK changes its constructor signature.
I'd also make the blank-completion rejection loud in the report output, not just an error status. A run that produced no output is exactly the one someone eyeballs as a pass when skimming CI.
One question on the regression definition: passed-to-not-passed means a scenario can drift from 0.95 to 0.81 against a 0.8 threshold and still count as unchanged. Do you rely on the per-family delta to catch that, or did you consider a per-scenario score-drift flag alongside the status flip?
Great question — and yes, we went with option 2 (per-scenario score-drift flag), not relying on per-family delta. The fix adds a SCORE_DELTA_THRESHOLD (0.05) so a scenario dropping 0.95→0.81 gets flagged as score_delta_regressed right alongside the status-based regression, even though the status is still "passed". The aggregate regressed count also combines both.
It's tracked in issue #271 (also covers snapshot mode and git_sha traceability) and landing in v0.2.0.
@debashish_ghosal The gap I keep coming back to is that agents don't have a concept of 'same mistake' across sessions. A human who was told the same thing last week will feel the repetition. An agent won't, and there is no ledger column for that. Scenario packs help, but only if the re-run knows it is a re-run.
Thanks - as this is eval framework, would it be a fair read-back on what you are saying is - this framework could process multiple traces from same agent and detect what could be fed back to "persisted memory" of an agent as "learn and do not do this"? Since, this is eval framework, it has no means to add persisted memory to an agent. This memory or ledger column would need to be a feature of the agent design that's under evaluation. I can check if there are standardized way to give this feedback back to agents
@debashish_ghosal That read-back is close but I would narrow one piece. The eval framework does not need to write to the agent's persisted memory. It needs to write to a signal log the agent reads at boot. The distinction matters because if the framework directly edits the agent's memory, you lose the ability to measure whether the agent would have made the same mistake again without intervention. The framework should publish findings. The agent should decide whether to incorporate them. That separation keeps the eval signal measurable and the agent's learning traceable. Think of it as a commit log: the framework proposes corrections, the agent merges the ones that survive its own validation. You keep the failure data either way.
On the regression definition, I'd keep already-failing scenarios out of the release gate but give failure-mode shifts their own non-blocking row in the report. A timeout turning into a hallucination is invisible today, and for the safety family that particular shift can matter more than a score drop.
Correct, yes, whoever puts the OSS package to use for agent eval must consider. What I have provided in the OSS is a collection of scenario packs which does not on its own eval an agent. User of that will need to decide on the scenarios and what constitutes a gate. Does this make sense?
The integration breaking before the judge matches my experience exactly. The eval logic is usually the easy part, and the harness around it is where everything falls apart. I started versioning my scenario packs like code so a regression in the environment setup doesn't get blamed on the model. How do you keep the packs from going stale as the agent's tools change?
Kartik - thanks for the comments!
I just released 0.2.0 and support for other frameworks.
Scenario packs versioning is a good idea and I was considering putting decoupling the release of scenario packs themselves from the harness, right now they are in the same repo. I haven't put thoughts into scenario packs going stale, but I will need to. In 0.1.0, I supported Langgraph and Pydantic agents, but as I have added support for other frameworks, I realized this is getting out of hand and I need to consider a model where harness may become mature but scenario packs need frequent updates - very much like how virus detection engine is not changing as much but the virus definitions are getting updated. I will have to think about it a bit
Congratss