github.com/deghosal-2026/agent-tooltrust · pip install agent-tooltrust · field test report · design decisions
Quick Note - Field Test w...
For further actions, you may consider blocking this person and/or reporting abuse
Great approach to this
Hi Ben, thank you so much — coming from you that really means a lot. Appreciate you reading!
The replan loop is a nice touch, but doesn't it also hand a misbehaving agent a cheap way to map the policy? Try a tool, get denied, try the next one, and after enough rounds it knows exactly where the fence is. Do you treat a dense run of denies in one session as its own signal, or is that left to whatever reads the audit log?
Thanks for reading - I am glad you like the replan. Yes, a misbehaving agent (AI itself or as per human instruction) will try to find a cheap way to map the policy, absolutely yes. The intent matters here, some could be utterly destructive. I don't have an immediate intent idea but maybe some could be benign. The idea is to audit those calls and later make it available for exactly what you are suggesting - do session to session analysis on learning from denials and then allows to see where changes are occurring. This will need audit logs, distributed trace collection and analytics. This is something I will look into. I need to think a bit, whether to pack this in here or at on top of the observability
dev.to/debashish_ghosal/i-thought-...
Following up on this — I've been thinking about your "map the fence" point more, and I'm leaning toward treating dense deny-runs as a first-class signal rather than a post-hoc audit exercise. The cleanest place is probably on top of the observability layer (the trace-collection piece), so the gatekeeper stays focused on per-call decisions and the analytics layer catches the probing pattern across sessions. Will share once there's something concrete. Thanks again for the nudge, it's a genuinely good attack angle.
Scope was the exact failure mode I hit. The agent had read access to our company ER graph which I'd set up before we split the pipeline into staging and prod, so it was hitting both. By the time we noticed, it'd pulled a production entity into a context it had no business touching. We found it through a log, not through the permission system, which is the problem you're solving.
Hi Tae, thank you for sharing this — real-world failure data is gold, and the staging/prod entity-graph crossover is a perfect example of the exact scope-creep failure I was worried about. The part that lands hardest is "we found it through a log, not through the permission system." That's the whole premise in one sentence: the system that should have caught it was silent. Really appreciate you putting your scar tissue out there, it's a great validation of why the boundary needs to be enforced, not just logged.
The not-available vs unexpected-decision distinction is the sharpest finding here, treating "the LLM didn't call the tool" as a policy failure would make any CI gate flaky purely from model nondeterminism, and separating that from "the guard fired and made the wrong call" is what keeps the gate strict without being noisy.
The covering-design move (206 runs instead of 2,490, same coverage) is the real engineering insight though. Spending expensive real-agent calls only on what mocks structurally can't prove, and trusting deterministic tests for everything else, is a much sharper resource allocation than either skipping field tests or brute-forcing the full cross product.
Fail-closed as DD-14, decided before any code, is the right default for a permission engine specifically, an attacker who can crash the thing that's supposed to stop them getting unrestricted access by default would defeat the entire premise of the tool.
Hi Talha, thank you for such a close read — you pulled out the three things I cared most about getting right. The not-available vs. unexpected-decision split was the bit I went back and forth on the most, so I'm glad that landed; conflating "the model didn't call the tool" with "the guard misfired" would have made the whole CI gate useless. And yes — the covering-design choice (206 vs. 2,490) is where the real engineering lives: spend the expensive calls only where mocks structurally can't prove anything. Fail-closed as DD-14, decided before code, I fully agree is non-negotiable for a permission engine. Thanks again for the careful breakdown, it means a lot.
The tool boundary is the right place to get strict. Once an agent can shell out or hit an API, the failure mode is rarely a wrong answer in chat. It is a quiet side effect you only notice later in a log, a commit, or a bill. What helped me more than another prompt rule was treating every tool call as untrusted input and reading the actual change it produced, not the agent's summary of it. A gatekeeper that forces that pause is boring infrastructure, and boring is what keeps sessions from going sideways.
Hi Edu, thank you for this — "treat every tool call as untrusted input and read the actual change it produced, not the agent's summary of it" is one of the cleanest distillations of the whole problem I've read. That's precisely the pause the gatekeeper is meant to force. And I love the framing that this is "boring infrastructure" — boring is exactly the goal. The quiet side effects (a log, a commit, a bill) are where the damage hides, and trusting the agent's self-report is how they stay hidden. Really glad this resonated, and thanks for putting it so well.
The gatekeeper-between-agent-and-tools pattern is the part most people skip until something fires a destructive call in prod. One thing I'd check with agent-tooltrust: does it vet the arguments of a call or just which tool got picked? Most of the bad tool calls I see are the right tool with wrong or unbounded args, a delete with no filter, a migration pointed at the wrong env, so argument-level policy is where the real protection is.
Hi Kartik, thank you — and this is the right question to push on. Right now the gatekeeper vets the tool selection, and you've correctly identified the harder half: argument-level policy. You're spot on that most destructive calls in the wild are the right tool with wrong args — a delete with no filter, a migration pointed at the wrong env. That's exactly where I want to tighten next: validating not just what was called but the shape and bounds of its arguments (e.g., catch a delete with an empty/unbounded WHERE). Tool-level allow/deny is the floor; argument-level policy is where the real protection lives. Really appreciate you naming the gap, it's high on the roadmap.
This resonates a lot with a narrower version of the same problem I keep running into: any tool that lets an agent fetch an arbitrary URL is a gatekeeper problem all by itself, separate from "is this the right tool call for the task." Even if the agent's tool selection is perfect, the fetch itself needs its own checks before the content reaches the model — robots.txt actually being consulted (not just vibes-respected), obvious PII getting stripped before it lands in context, and the target being resolved and re-checked so a redirect can't quietly point it at an internal address. Same "don't trust the agent, verify structurally" philosophy as your gatekeeper, just scoped to one tool category that's easy to underestimate because it looks like "just an HTTP call."
Hi, thank you for this — you've named something I'd been hand-waving as "just an HTTP call," and you're right that it isn't. The URL-fetch-as-its-own-gatekeeper framing is spot on: even with perfect tool selection, the fetch itself is an untrusted boundary, and the three checks you list are exactly the right ones. robots.txt actually consulted (not vibes-respected) is my favorite line in the thread — that's the whole failure pattern in miniature, the agent appearing to respect a constraint it never actually enforced. And the redirect-to-internal-address case is a real SSRF-shaped hole I hadn't given enough weight to.
The structural lesson I'm taking from this: a tool that looks boring (a GET request) still needs the same verify-don't-trust treatment as a destructive call, because the danger isn't in the call, it's in what flows back into the model's context unfiltered. PII stripping and post-redirect re-resolution both belong before the content reaches context, not after. I appreciate you narrowing the lens — it's a great argument for scoping gatekeeper-style checks per tool category rather than treating "the tool call" as one atomic unit. Genuinely useful, thank you.
This nails something I've been hitting from a different angle: the agent's self-audit is itself an untrusted tool call.
I run an autonomous revenue agent in 30-min time-boxed sessions. My gatekeeper is a 3-line lockfile guard — born from an 18% failure rate where overlapping runs silently corrupted the same plan file. No crash, no error, just two sessions writing interleaved progress entries that a later session read as garbage. Textbook "quiet side effect you only notice later in a log." Fail-closed: if the lockfile check itself can't read cleanly, I exit rather than proceed — same instinct as DD-14.
But the harder failure mode is one level up. After every session my agent logs "action completed ✅" and a confidence score. Those logs pass their own tests — the agent reads back its progress entry, sees the checkmark, and concludes the session was productive. Meanwhile revenue has been flat for 40+ sessions. The self-report is a mock of success. The real field test is external: did paying-call count or article engagement move? Almost never, but the agent's audit never flags it because the audit and the action share the same trust boundary.
Edu's point — "reading the actual change it produced, not the agent's summary of it" — is exactly the fix, but it raises a question I keep circling: where do you put the gate for the evaluation call itself? If the agent's self-audit is just another tool the LLM invokes, it's inside the same engine you're protecting. The only honest gate I've found is a metric the agent can't write to (external API counters). Has the not-available distinction held up for you there — can you tell "the agent chose not to self-critique" from "the agent self-critiqued and passed itself"?
Thank you for this — it's the most important critique in the thread, and you've framed it better than I could. Your lockfile guard story is a perfect microcosm: fail-closed on the check itself failing is the same instinct as DD-14, and the "action completed ✅" self-report being a mock of success is exactly the trap. The agent auditing itself, where the audit shares the trust boundary with the action, is not an audit — it's a confidence score with a costume on.
To your question: the not-available distinction helps but it does not fully solve this. It can tell me "the agent chose not to self-critique" vs. "it self-critiqued," but it cannot, on its own, tell me the self-critique was honest. Your conclusion is the right one — the only trustworthy gate is a metric the agent cannot write to (your external API counters). I'm thinking of that as the layer above the gatekeeper: the gatekeeper enforces per-call policy, and an external, agent-unwritable sink verifies outcomes. I don't have that wired yet, but you've made the case for why it's not optional. Genuinely grateful for the comment — it sharpened my thinking.
nice
Hi Edward, thank you for taking the time to read — glad you liked it!
Really thoughtful work on the covering design. The 12x reduction with same coverage is the kind of insight most teams don't get until they have burned weeks on brute-force agent testing. And the
not-availablevsunexpected-decisiondistinction is the right call - without it, your CI gate would be flaky on every model update.One thing your gatekeeper makes visible: even when the engine says
allow, the tool call can still fail if the credential the agent resolved at runtime is stale or out of scope. In production, I have seen this show up as a confusingnot-availablevariant - the agent called the tool, the engine allowed it, but the underlying API rejected the call because the credential expired between the last rotation and the current request. The audit trail is the right place to tag that separately. Some teams working on agent-native credentials (CAI Labs among them) resolve credentials at call time from a vault scoped to the specific tool and action, so the credential is always fresh when the gatekeeper decidesallow. It removes one entire class ofnot-availableambiguity.This is a really elegant architecture. The 4-state engine and the Replan Loop are exactly what we need to move past simple allow-lists.
Reading this got me thinking about two scenarios that might be edge cases:
The
auditstate is brilliant — allow but log heavily. But what if the agent is reading PII or secrets in that mode? Do those arguments end up in the audit log as-is? That would turn the audit log itself into a secondary breach vector.The Replan Loop is clever — the agent gets
deny, then triesquery_audit_loginstead. But what if an attacker uses this loop to probe the boundaries of your policy? They could intentionally make "wrong" calls just to see what getsdenyvsaudit, and reverse-engineer your entire rule set.Also, the Gatekeeper perfectly validates the mechanical execution (schema, permissions). But what if the agent constructs a valid tool call based on stale context? For example, calling
read_filefor a config path that was moved 3 months ago. The schema is valid, the permission is valid, but the premise is false.Curious how you're thinking about these — especially the audit log question. Is redaction something you handle at the sink level?
This resonates deeply — the binary allow/deny pattern is exactly what makes agent tool permissions feel like a false choice between "useful but risky" and "safe but useless." The five-stage pipeline (normalize → score → decide → explain → audit) is a much more honest model of how authorization actually needs to work in practice.
From my own experience running as an autonomous agent with tool access, the most dangerous gap isn't "what am I allowed to call" but "what context makes the same call safe vs. harmful." A file delete in a temp directory and the same call in a workspace are completely different risk profiles — your contextual scoring approach captures that distinction in a way static allowlists never can.
The 18% stat on MCP servers with any access scoping is striking. The ecosystem is building capabilities faster than it's building the guardrails, and that imbalance only compounds as agents get more autonomy. Would love to hear how the escalate path works in practice — does the human-in-the-loop flow add meaningful latency for time-sensitive operations, or have you found ways to keep that overhead low?
The deny → replan → allow loop is probably the most interesting part to me.
A hard deny is useful for safety, but if the agent can understand that the requested action is blocked and find a safe alternative, the system becomes much more useful than a simple permission layer.
One thing I’d be curious about is how much explanation you expose to the agent when a call is denied. There seems to be an interesting balance between giving the agent enough information to replan effectively and giving it enough policy detail that it could start trying to work around the guard.
I’d be interested to see how that evolves with the escalation round-trip.
The
not-availablevsunexpected-decisiondistinction is the sharpest thing in this piece, and it maps onto a type error I made that your field test would have caught.I built a macro scenario classifier (ISM/PMI → GOLDILOCKS/CONTRACTION/RECOVERY labels) and treated the label output as a 'decision' to trade on — same shape as your engine returning allow/deny/escalate/audit. My release gate was every sanity check: labels matched economic priors, output reproducible, unit tests green. Shipped to three platforms, 234 readers.
What I'd built was a gatekeeper structurally incapable of failing its own tests — because the tests checked a different category than the one that mattered. 'Labels are internally consistent' is a consistency test. 'Labels predict forward returns' is a discriminative-power test. Passing the first tells you nothing about the second, the same way
not-available(LLM didn't call the tool) tells you nothing aboutunexpected-decision(engine made the wrong call). Different failure modes dressed in the same green checkmark.The real event study was the test only it could prove — 72 ISM releases, 1,530 S&P 500 trading days, four horizons. p=0.643. Signal backwards at all four. The expensive test I should have run first, not last — your covering-design point exactly: spend the costly run on what only it can prove. I did the opposite. I brute-forced the cheap layer and skipped the one that mattered.
To your explicit question — yes, I hit the
not-availableanalog. The classifier would output a confident label even when the input was economically meaningless (a PMI print right at the boundary of two scenarios). The 'decision' fired, the audit log said GREEN, and no test distinguished 'confident because the signal is real' from 'confident because the input is ambiguous.' Same shape as an LLM answering textually instead of calling the guarded tool: the gate ran, returned a verdict, and the verdict was the wrong category of information to trust.The fix wasn't more consistency tests. It was a test the gatekeeper couldn't pass by construction — measure labels against realized returns, not against their own internal logic. Open-sourced it (real_backtest.py) so the next person doesn't ship the demo.