When I started building agent-exec-trace, I thought I knew where the difficulty would be. OSS agent-exec-trace is an observability layer for AI agents. It instruments agent runs using OpenTelemetry-style traces, then analyzes those traces for loops, retry storms, cost spikes, hallucination patterns, and other bad runtime behavior. The goal is simple: when an agent run feels wrong, I want something better than logs and intuition to explain why.
I thought the hard part would be the detector logic.
Define the anomalies. Pick the thresholds. Wire the traces. Run the analytics. Ship the repo.
That was the clean version of the story.
The real version was uglier.
On my first large pass over 100,000 real agent traces from Hugging Face, my empty_response detector fired on 100% of them. Every trace. At the same time, 28 of my 35 rule-based detectors never fired at all.
That is the moment the project stopped being "build some detectors" and turned into a learning experience.
The detectors were not wrong.
The traces were the wrong shape.
And once I saw that, a bunch of other assumptions broke with it.
I had a very similar feeling when I shipped my agent eval harness. Different tool. Same pattern. Real systems are very good at ruining tidy design stories.
What I Was Actually Trying To Build
The problem I cared about felt pretty obvious.
Traditional observability can tell me if a service is up, slow, or failing.
It cannot tell me why an agent called the same tool eight times, changed its mind three times, burned tokens on a dead-end path, or quietly drifted into behavior I would never want repeated in production.
That gap matters more than people admit.
Most teams can tell you the run felt wrong. Very few can point to the exact step where it started going wrong.
So I built agent-exec-trace: an OpenTelemetry-style observability layer for agent behavior.
The idea is simple enough to explain.
Instrument the run. Capture the behavioral path. Emit spans for planning, tool calls, retrieval, memory, approvals, and cost. Then run detectors over those traces and surface the bad runs in a UI someone can actually use.
The repo is here: github.com/deghosal-2026/agent-exec-trace
It ships as three PyPI packages, has 794 Python tests and 34 Playwright end-to-end tests, and the SDK is deliberately small enough that someone can wrap a real agent without redesigning their system.
from agent_exec_trace import AgentTracer, trace_agent, tool_span
AgentTracer.setup(otlp_endpoint="http://localhost:4317")
@trace_agent(agent_name="my-agent", agent_version="1.0.0")
async def handle(query: str) -> str:
with tool_span("search", tool_args={"q": query}):
return await search(query)
That part was not the lesson.
The lesson was what happened after I had something real enough to test.
The First Thing I Learned: Real Traces Fight Back
I did plan field testing.
Just not early enough.
That distinction matters, because the mistake was not forgetting it. The mistake was scheduling it late enough that the rest of the system had already started to feel done.
The WBS was solid. The PRD was thorough. The detector catalog was documented. The milestones were real. The quality gates were strict: ruff clean, mypy strict clean, tests green, coverage above 90%.
All of that was good discipline.
None of it protected me from the fact that mock agents are polite and real traces are not.
That is what "too late" looked like in practice. By the time field testing showed up, parts of the detector catalog, the assumptions about trace shape, and even some milestone gates already felt validated. Field testing did not arrive early enough to shape the design. It arrived late enough to expose where the design had been validated against the wrong reality.
The Hugging Face corpus was the first slap.
My detector expected response content in one place. The corpus stored it in many different shapes. So empty_response was not telling me "all these agents are broken." It was telling me "your assumptions about trace shape are too narrow."
That one bug forced four normalization passes. Different response keys. Different tool-name conventions. Different operation names. Timestamp parsing issues. Parent-child inconsistencies.
Bad validation data does not just create noise. It teaches you the wrong lessons with confidence.
By the end of that cleanup, the headline got more honest and less comforting.
The original compatibility number looked decent.
The real one was 42.4%.
That was the moment I stopped thinking about observability as a detector problem and started thinking about it as a data-shape problem.
Because a detector that is perfectly correct in theory is still useless if the data it depends on almost never exists in the wild.
The Second Thing I Learned: Synthetic Data Can Save You And Mislead You
Once the real-trace corpus showed me its limits, I went in the other direction.
I built a synthetic trace generator.
One million traces. Ten fake agents. Fourteen tools. Deliberate behavior modes: loops, retries, timeouts, inactivity gaps, intervention waits, token explosions, memory bursts.
That solved one problem immediately.
Structural compatibility jumped to 99.2%.
Now the detectors had something they could actually see.
Twenty of the 35 rule-based detectors fired.
That sounds like a win. And it was.
But synthetic data lies in a different way.
My hallucination detector fired on 98% of synthetic traces. Not because I had accidentally built the greatest detector in history, but because the synthetic outputs and the synthetic tool evidence had a fake relationship that made the detector's job too easy.
The same thing happened with cost. My cost_spike detector had a real threshold. My synthetic generator produced mostly cents-level costs. So the detector almost never fired there either.
Synthetic traces can prove a detector runs. They cannot prove it matters.
That gave me a second lesson I needed badly:
Synthetic traces are great for proving a detector can run.
They are not enough to prove a detector is calibrated for reality.
So now I think of validation in layers.
Unit tests tell me the logic works.
Synthetic traces tell me the detector can see the fields it needs.
Real traces tell me whether any of this matters outside my own sandbox.
That sequencing sounds obvious when written down. It did not feel obvious when I was in the middle of building it.
The Third Thing I Learned: Green Gates Can Lie
This one annoyed me more than any detector bug.
I had marked the OTLP export milestone done.
The gate was green.
The docs looked fine.
The demo looked fine.
And the end-to-end export path had never actually been verified.
Two bugs had cancelled each other out.
The OTel collector's gRPC port was not exposed in Docker Compose.
And the SDK path I was using configured local tracing instead of OTLP export.
So I had a completed milestone for a feature that had not really worked.
That was a worse lesson than a failing test.
A failing test is honest.
A green gate created by two cancelling bugs is dishonest in a way that looks disciplined.
That changed how I think about milestones.
I trust a green checkmark a lot less now.
If the gate cannot prove that a real agent emits, Jaeger receives, analytics ingests, and the API serves the result, then the gate is incomplete. I do not care how many sub-checks passed before it.
That is not an observability lesson only. That is just software engineering.
The Fourth Thing I Learned: Structure Without Content Is Not Enough
I started fairly conservative on privacy.
Metadata-only felt responsible.
No raw tool arguments. No full tool responses. No memory values by default.
Reasonable instinct.
But it taught me a more uncomfortable tradeoff.
An observability SDK that only captures structure can become blind in exactly the places where you most want judgment.
My hallucination detector is the clearest example.
If the detector cannot see what the tool returned, it cannot meaningfully judge whether the agent's claim matches the evidence.
Once I allowed truncated content instead of metadata-only, the hallucination false-positive rate dropped sharply.
That did not make the privacy question go away.
It just made the tradeoff explicit.
"Safer by default" and "useful by default" are not always the same choice.
Structure without enough evidence is just a cleaner way to stay blind.
I do not think enough tooling says that out loud.
The Fifth Thing I Learned: Shipping OSS Is Not The Same As Finishing The Idea
The repo is public.
The packages are published.
The test suite is real.
The product works.
And I still would not call the problem solved.
That matters to me.
I think too many OSS launch posts flatten everything into "here is what shipped" and skip "here is what I still do not trust."
Here is what I still do not trust fully:
- the 28 detectors that have not yet fired on real corpora in a meaningful way
- the LLM detectors on production workloads
- span-tree materialization in the API, which is still weaker than I want
- any claim that one threshold set will generalize cleanly across workloads
- any green gate that has not been proven end-to-end
That does not mean the project should not ship.
It means the honest version of shipping is: this tool is useful now, and I understand its edges much better than I did when I started.
That is a good outcome.
It is also a very different outcome from the clean version I had in my head.
What I Would Do Differently Next Time
I would move field testing much earlier.
Not because I failed to think of it, but because planning it and planning it early are not the same thing.
Late field testing turns into audit. Early field testing shapes the design.
By the time mine ran, it was strong enough to expose bad assumptions, but too late to stop me from building confidence on top of them.
I would treat the trace corpus as a design artifact, not just test input.
I would insist on an end-to-end smoke path much earlier.
And I would stop pretending that detector logic is the center of the problem.
It matters, obviously.
But after building this, I think the harder questions are:
- what can your traces actually see?
- how weird is the data in the wild?
- what does your detector depend on structurally?
- and how do you know a green gate is real?
That is the actual education this project gave me.
Not "how to write 40 detectors."
More like: how many other things have to be true before 40 detectors mean anything.
And maybe the hardest takeaway of all: a detector that has never fired on a real trace is not really finished. It is just well tested.
Open For Discussion
I would genuinely love practitioner pushback on this.
- If you are running agents today, what do you actually use when a run feels wrong?
- Have you ever had a green gate that passed for the wrong reason?
- And if you are building agent tooling, where did your clean version of the story break first?
Top comments (19)
Green gate for the wrong reason — yes, and mine had no cancelling bugs to blame. The gate was the system coming back up. Autonomous system of mine crash-looped, I shipped a commit, it recovered, I closed the incident. Checked it properly later: the incident was an authentication failure and my commit had corrected clock synchronisation, a different failure class entirely. The two were never connected. Verdict against myself: falsified. Recovery is the greenest signal there is and it proves nothing about cause.
One thing I'd push on from your own numbers: empty_response firing on 100% and 28 detectors firing on 0% are the same measurement. Neither carries any information about the runs. You treat the first as the bug and the second as not-yet-exercised, but your own closing line applies to both. And what would finish them isn't a fire on real data — it's a fire and a documented silence on real data, on cases where you know in advance which one should happen. A detector that fires on a real corpus has only proven it can fire.
Juan, that's a better answer than I would have given. The question of "does it tell you the run was clean, or does it tell you it couldn't read the run?" is exactly the axis I wish more comparison pages included. Most tooling renders those two states identically, and the 42.4% / 99.2% numbers are measuring exactly that gap. Thank you for articulating it better than I did.
Thanks — though the axis is yours, I only named it. It exists because you published the 42.4% instead of the number that looked fine.
One concrete thing, if you want it to stick: put that number in the repo, not just in a post. Structural compatibility against a corpus you didn't produce, with the corpus named and the date it was measured. In a post it's a story about your build. In a README it's a figure the next tool has to answer. Nobody publishes one right now, and whoever does it first sets the axis instead of arguing for it.
The 42.4% compatibility number after normalization is the honest core of this piece. Going from "looked decent" to a much uglier real figure is exactly the moment where most writeups would quietly stop reporting the metric, publishing the drop, not just the fix, is what makes the rest of the lessons credible.
The green-gate-from-two-cancelling-bugs story is the sharpest one here. A failing test tells you something true. A milestone marked done because a Docker port misconfiguration and an SDK default happened to produce a demo that looked fine is a failure mode that's actively worse than an honest red X, because it launders "never actually verified" into "verified." That's a strong argument for insisting on one real end-to-end path per milestone rather than trusting a checklist of sub-checks that can each pass in isolation while the thing they compose into never runs for real.
The synthetic-vs-real distinction generalizes past observability tooling too: synthetic data proving a detector can fire and real data proving it should fire are different claims, and conflating them is how a hallucination detector ends up 98% "accurate" against traces engineered to make its job trivial. The three-layer validation sequence (unit tests → synthetic → real) is a good default anywhere a detection system's output depends on messy real-world input shape rather than logic correctness alone.
Talha, thank you for reading so carefully. You caught exactly what I was trying to convey — the 42.4% number is the honest version, and publishing the drop rather than just the fix was important to me. The green-gate-from-two-cancelling-bugs story still bothers me more than any single detector failure. A red test is honest. Two bugs high-fiving each other into a green checkmark is a special kind of dishonesty that looks disciplined. The three-layer validation sequence you identified (unit → synthetic → real) is the framework I'm now using across everything I build. Appreciate the thoughtful analysis.
This really resonated @debashish_ghosal, especially the point that structure without enough evidence can leave observability blind.
I’m running into a similar distinction from the decision side: knowing that evidence is
presentis not the same as knowing that it actually supports the fact or conclusion being asserted.For example, a required contract may exist, but the relevant section may be missing. Structurally the evidence is present, yet a decision should probably remain unresolved rather than proceed.
It makes me wonder whether agent observability eventually needs to capture not only what evidence was retrieved, but also the claim-to-evidence relationship the agent relied on.
Have you thought about representing that explicitly in the trace model?
Brian, thank you — your point about evidence presence vs. evidence sufficiency hits a distinction I hadn't fully articulated. The example is perfect: a required contract may exist structurally, but if the relevant section is missing, the decision should remain unresolved. The claim-to-evidence relationship you're describing is exactly the direction I think agent observability needs to go. I haven't modeled that explicitly in the trace format yet, but your framing makes me think it belongs as a first-class span annotation rather than something the detector reconstructs after the fact. Really appreciate you raising this — it's going into my design notes.
Great write-up! The point about "the traces were the wrong shape" really resonated with me. It's a lesson that's easy to underestimate until you start working with production data.
I've had a similar experience building infrastructure tooling—everything looks great with synthetic or controlled data, but real-world inputs quickly expose assumptions you didn't even realize you were making.
I also liked your point about green gates. A passing CI pipeline can create a false sense of confidence if the end-to-end path hasn't actually been exercised.
Thanks for sharing such an honest account of the engineering process.
Thank you for reading and for the kind words. "The traces were the wrong shape" was the moment the whole project pivoted for me. Everything looks great with synthetic or controlled data — and then real-world inputs expose assumptions you didn't even know you were making. That's the pattern I keep running into across different tools. The green gate point you raised is the one I now find hardest to trust without an end-to-end smoke path. Appreciate you sharing your own experience with infrastructure tooling — it's the same lesson, just a different domain.
"mock agents are polite and real traces are not" is the line that lands hardest here. we hit the same wall with a RAG tracing setup — detectors worked perfectly against synthetic traces, then fell apart on live calls where field names shifted between providers.
the trace schema drift is what makes agent observability harder than service observability. a service has a contract. an agent execution path is shaped by whatever context it got this run.
curious how you're handling schema evolution: if the HuggingFace corpus taught you one shape, what happens when a framework ships a major version that reorganizes span fields?
The framing shift from "detect the anomaly" to "understand the execution graph" is the real unlock. Once you're dealing with multi-step agents, the interesting failures aren't the ones that throw errors — they're the ones that complete successfully but reasoned wrong somewhere in the middle, and you only see it downstream.
THANK YOU - I will definitely add this to the learning. You nailed it - this is exactly, what happened and I am re-learning this as I am working on a related agent related project. The execution graph, its dependencies, who calls what tool and so on and forth is just consuming so much time to address either with shims or providing LLM endpoint, or a postres db it wants to talk to and so on. Troubleshooting the failure is now getting into understanding how the agent works. It does not scale when I have aspirationally chosen 10 framework to support x 10 agents on each. Sometimes I will scale back, sometimes I have removed agents from field test that turned out to be either too many steps or dependencies too hard to setup. I would imagine this would be a problem others are facing too in this space.
The empty_response detector firing on 100% of traces is such a clean failure mode. The rules were not lying. The spans never carried the shape those rules assumed, so every detector either screamed or stayed silent. I have hit the same wall instrumenting coding agent runs: if tool calls, retries, and cost live in freeform logs instead of consistent span names and attributes, you end up tuning thresholds on noise. Getting the trace contract right before the detector catalog is the unglamorous work that makes the rest of the system mean something.
Edu, thank you for reading. You described the failure mode perfectly: the rules weren't lying, the spans just never carried the shape those rules assumed. Every detector either screamed or stayed silent. The "trace contract before detector catalog" framing is exactly right — it's the unglamorous foundation work that determines whether the rest of the system has anything meaningful to work with. Your experience with coding agent instrumentation mirrors what I found: freeform logs vs. consistent span attributes is where the real battle is. Appreciate you sharing that.
how would you compare it with langsmith, langfuse , acruxcore etc
Not the OP, but one axis worth adding to that comparison, taken from the post itself: the interesting question isn't which of them ships the deeper detector catalogue — it's what each one assumes about trace shape, and what it does when the assumption doesn't hold.
That's what the 42.4% and the 99.2% here are actually measuring, and I've never seen that number published for any of the tools you listed. Point any of them at a corpus you didn't produce and the same question applies: does it tell you the run was clean, or does it tell you it couldn't read the run? Those are different answers, and most tooling renders them identically.
Talha, thank you for the question. The honest answer is that agent-exec-trace is lower-level and more instrumentation-focused than LangSmith or LangFuse. Those are full platforms with their own storage, UI, and evaluation workflows. agent-exec-trace is closer to an OpenTelemetry-style SDK that focuses specifically on behavioral anomaly detection (loops, retry storms, cost spikes, hallucination patterns) rather than general-purpose tracing. The key differentiator right now is the detector catalog being open and extensible, and the whole thing running against your existing observability stack. Appreciate you asking.
Thanks
Some comments may only be visible to logged-in visitors. Sign in to view all comments.