If you think logging your LLM responses is enough for compliance, this post is for you.
August 2026. The EU AI Act enforcement begins for high-risk systems. FINRA moved AI agents to "active supervisory priority" in its 2026 annual report. Banks and fintechs running agents in production have stopped asking "how do I build it?" — they're asking "how do I audit it?"
And the market's answer is uncomfortable: no single piece closes the compliance circuit end-to-end out of the box.
Auditable is not the same as logged
90% of teams confuse traceability with auditability. A LangSmith trace with 47 agent steps is not an audit trail. A regulatory audit trail requires:
- Hash-chaining between steps (alter one, break the entire chain downstream)
- Cryptographic signatures on every decision (Ed25519, not log.info())
- WORM storage (Write Once Read Many — S3 Object Lock, not mutable Elasticsearch)
- Compliance metadata: policy version, human approver, signed timestamp
- This isn't a nice-to-have. It's Article 12 of the EU AI Act. Minimum 6-month retention. 24 months for law enforcement and biometric systems. Tamper-evident logging is not optional.
The stack that actually works today (July 2026)
LangGraph as the runtime
LangGraph is the most mature framework in the ecosystem for native auditability. Three features set it apart:
Pause before sensitive actions
graph.interrupt("before_transfer", wait_for_human=True)
Checkpointing for deterministic replay
graph.checkpointer = SqliteSaver.from_conn_string("checkpoints.db")
Time travel: reconstruct state from any previous checkpoint
graph.get_state(run_id, checkpoint_id=42)
It's not perfect. Tracing is complete within the graph, but regulatory metadata (signed timestamp, input/output hash, policy version) you have to build yourself.
Approval gates — not a "confirm" button
A serious approval gate isn't if user_clicks_ok: proceed(). It's a stateful door with:
- Compact action packet: tool, args, expected side effect, risk score
- Contextual approver: refund < $50 → auto. Refund > $50 → finance manager. Compliance rule change → DPO + legal
- Decision recorded as a signed receipt
result = request_approval(
agent_name="billing-agent",
action="Transfer $5,000 to vendor #4892",
risk_score=0.82,
assignee="cfo@company.com"
)
if result["status"] != "approved":
raise PolicyViolation("transfer blocked")
The langgraph-approval-hub project (MIT) already provides an approval dashboard, exportable audit log, and email/Slack routing. It deploys in 5 minutes on Vercel + Supabase.
The receipt: the minimum unit of audit
Every agent step should produce a receipt shaped like this:
{
"run_id": "<uuid>",
"step": 3,
"timestamp": "2026-07-18T15:22:00Z",
"input_hash": "sha256:<hash>",
"reasoning": "<model's internal reasoning>",
"tool_call": {
"tool": "transfer",
"arguments_hash": "sha256:<hash>"
},
"decision": "approved_by_human",
"human_approver": "cfo@company.com",
"policy_version": "v2.3.1",
"previous_hash": "sha256:<hash_step_2>",
"signature": "ed25519:<signature>"
}
Key properties:
previous_hash creates a hash-chain. Alter one receipt, break the entire chain.
policy_version tells you under which rules the decision was made.
signature proves who (or what) authorized it.
The stack they're not selling you as a bundle
No vendor packages this end-to-end. You have to assemble it:
Plus three things nobody has built yet:
Reasoning drift detection
Your agent can degrade silently: same task, same tools, worse decisions. Tracing tools show you the reasoning. Nobody tells you if it's worse than last week. No production-ready tool measures whether your agent's reasoning quality is drifting over time. This is a genuine open gap.Automated incident reporting (24 hours)
The EU AI Act requires incident reports within 24 hours (life/safety) or 72 hours. No tool automates "generate a regulatory report from the receipt chain." You're writing that yourself.Cross-framework receipt standard
An IETF draft exists for agent audit trails (AutoGen RFC + Nobulex), but it's not implemented composably across LangGraph, CrewAI, or Microsoft Agent Framework. You're in DIY territory until the standard solidifies.
Where to start
- If your agent touches fintech, banking, or hiring: start with the receipt schema. Define it before choosing tools. You'll be able to swap frameworks without redoing the audit layer.
- If you're in Europe and your agent falls under high-risk: Article 12 is not optional. Tamper-evident logging with cryptographic signatures. Not console.log().
- If you want to contribute: the automated incident reporting gap is wide open. The reasoning drift detection gap is even wider. Both are greenfield.
Top comments (19)
A per-run hash chain gives you tamper evidence over the receipts that exist; completeness is a separate property and
previous_hashdoes not carry it. The signer here is also the audited party:signatureproves a key claimed a step, andstepis a counter chosen inside arun_idthe same key minted. So the cheap attack is not forgery. Under pressure a runtime can close one run and open another, and the inconvenient run leaves no numbered hole anywhere. What closes that is a monotonic sequence scoped to the signing key rather than to the run, so a discarded run shows up as a gap, plus a tool-side enforcement layer holding its own key and its own counter and countersigning what it actually executed. Audit then becomes reconciliation between two independently kept chains, and the discrepancy is the finding. A single self-signed chain has no discrepancy available to it.With the stale-state gap covered upthread,
reasoningis the field I would be most careful about signing. It is model-generated prose about model behavior, so signing it takes an unverified narrative and gives it the shape of evidence, when the signature only attests custody. The drift-detection gap you name sits downstream of the same problem: the model is free to vary how it narrates an identical decision, so measuring drift against that field measures style as much as judgment. It also runs straight into WORM. Free-text model output is where pasted secrets and personal data land, and Object Lock exists precisely so nothing can be removed during the retention window, which puts Article 12 retention and an erasure request in the same field with only one possible winner.Cheaper pattern: commit
reasoning_hashin the receipt and keep the prose in ordinary deletable storage addressed by that hash. The chain still proves the text existed and has not changed, and the text can still be deleted. Structured decision facts go in the receipt as fields: rule fired, inputs consulted, alternatives rejected, and a policy content hash rather than justv2.3.1, since a version string can be rebuilt under the same name.Self-signed chain and run grafting
You're right: if the same key mints run_id and chooses the step counter, a compromised runtime can drop a run and start another without leaving a hole. A proper implementation should enforce a monotonic sequence scoped to the signing key across runs, so any gap in step numbers becomes detectable.
Tool-side countersigning
The two-chain reconciliation model you describe is exactly the right target: one side signs what the agent claims, the other signs what the tool actually executed. Without that, you have no discrepancy to detect.
reasoning as signed evidence
This is the most important point. Signing model-generated prose gives it the shape of evidence while only attesting custody. That's actively misleading in an audit context. The right move is to store reasoning_hash in the receipt, keep the prose in ordinary mutable storage addressed by that hash, and put structured decision facts in the receipt itself — rule_fired, inputs_consulted, alternatives_rejected, and a policy_content_hash computed over the actual policy bytes, not just a version string.
WORM vs. erasure
The conflict between retention obligations and erasure requests is real, and putting free-text model output into immutable storage makes it worse. The pragmatic fix is boundary-based: structured audit facts in WORM, free-text reasoning outside it, with the hash linkage preserved.
Thanks for the thorough breakdown.
Reconciliation only produces a finding if the two chains are able to disagree. If one party deploys and operates both the runtime and the tool-side enforcement layer, that is a single trust root holding two keys, and both chains go quiet together under exactly the pressure the audit exists to catch. The useful question to ask about a second signer is whether it has any reason to keep signing at the moment the first would prefer it stopped. That is what decides where the enforcement layer belongs: the far side of a network hop, or a credential domain the audited party does not administer. It also reframes the API limitation raised upthread. When a target offers nothing to check against, what is absent is custody of an independent witness, and no signature scheme supplies that.
The boundary split moves the retention conflict off the immutable side, and it is worth writing down what that costs. Once the prose is deleted, the committed reasoning_hash still shows that something existed and was unaltered, but the preimage is gone permanently, so that field degrades from a record of what was said into a proof that a string of some length once hashed to this value. Probably the right trade. It does mean an erasure request granted years later quietly changes what a future audit can conclude, without touching the chain at all. So the structured facts have to carry the claim unaided, which is a heavier requirement on that field list than it looks. Anything recoverable only from the prose is a field that can disappear.
The receipt is still missing the state the action was based on.
input_hashandarguments_hashpreserve the recorded request, but they do not show whether the target was still in the same state when the tool ran. A signed receipt chain can preserve a stale decision perfectly.For a sensitive action, I would bind the request to a verified target version and stop if that state moved.
I wrote about that gap here: The reasoning was right, but the world shifted.
This is a sharp point, and you're right — input_hash and arguments_hash prove the request, not the world it landed in. A receipt chain can be internally consistent while every decision in it was based on stale state.
The gap you're describing is a TOCTOU at the agent level: the reasoning step happened against state S₁, the tool executed against S₂, and nothing in the receipt captures that shift.
I like the direction you propose — binding the request to a verified target version. In an auditable agent pipeline, that would look like:
The challenge is making this practical without snapshotting the entire world. For file-based agents, it's tractable. For live APIs, you'd need versioned reads or conditional requests, which not all APIs do.
Read your post — the "reasoning was right but the world shifted" framing is exactly the right way to put it. Thanks for the pointer.
The API limitation is exactly where the system has to be honest. The agent can state which version it read, but a trusted component still has to verify that version before the action happens.
Email is a simple example. A new outbound message often has no existing target version to compare. You may be able to bind it to the thread or draft state it was based on, but the send itself remains a new external effect.
If the target API offers no reliable state check, the system cannot honestly claim strong state-bound execution. The enforcement layer has to evaluate which guarantees the target can actually support, not only audit what the agent requested.
That is one of the design constraints behind MCP Boundary: no state-binding claim unless the target gives the enforcement layer something real to verify. If it does not, the action has to be treated as broader instead of pretending the state was verified.
Thanks for reading the post. TOCTOU is exactly the systems term for the gap I was trying to describe. Now we just need to convince every API provider to give us something real to check :D
Are you running agents in production ? How you would handle compliance ?
We do, but I try to keep the evidence layer independent from the orchestration framework as much as possible.
In practice, I treat the agent framework as an execution engine, not as the source of audit truth. Every sensitive action produces its own signed receipt with immutable metadata (policy version, timestamps, actor, input/output hashes and a link to the previous receipt). That way the evidence survives even if the orchestration framework changes in the future.
One lesson I learned is that emergency bypasses also need the same level of discipline. A permanent “temporary” bypass can silently invalidate an otherwise sound security design. Time-limited, fail-closed escape hatches with explicit audit records have worked much better for us operationally.
I think that’s the balance we’re all trying to reach: keeping the audit model framework-agnostic while still making it practical enough that teams actually adopt it.
This is gold. The emergency bypass point is something I should have included in the post.
You're right that the "temporary permanent bypass" is where most audit models collapse. Everyone designs for the happy path — agent proposes, human approves, receipt gets signed. But at 3 AM when something is broken and someone needs to override the policy gate now, that override either gets logged properly with the same hash-chain discipline, or it gets done through a backdoor that nobody documents and the audit becomes fiction.
The time-limited fail-closed escape hatch you describe is the right pattern. It forces the bypass to be intentional and temporary by design, not by policy. The system reverts automatically — no human has to remember to turn the bypass off. And the audit record of the bypass itself becomes part of the evidence chain, not a gap in it.
This is exactly the kind of operational reality that framework vendors don't document. Theory is clean; 3 AM production is messy.
Have you written about your implementation anywhere? The combination of framework-agnostic receipts + disciplined bypass handling would make a great case study.
Thanks, I appreciate that. I haven’t written it up yet, but I probably should. The implementation evolved from solving production problems rather than trying to design a perfect audit model from day one. The emergency bypass was one example where operational reality forced a change in the architecture. I’ll try to document the approach once the remaining rollout work is complete.
Please, keep me posted once you do it. I'd love to hear about real examples outside
Thanks, I definitely will. The discussion here actually changed how I’m thinking about the article. I was originally going to focus on the audit architecture itself, but I think the more interesting story is what production taught us after the architecture was “finished.” The emergency bypass ended up exposing assumptions we didn’t realize we had, and from there we started redesigning things like verification, ADRs, and operational evidence rather than just the receipt format. If that experience is useful to others, it’ll be worth documenting.
The time-limited-fail-closed hatch solves one problem: it closes itself.
The audit question is separate. A record that a bypass existed is not the same as an action receipt pointing to the exact bypass that allowed it. Without that link, both records can be valid and the chain still does not prove why the action was allowed.
It has the same shape as stale state. There the target moves. Here the rule moves.
If you write this up, that is the part I would read first.
That’s a great distinction. I completely agree that the bypass itself has to become part of the authorization evidence, not just an audit event.
In our case, the action receipt would explicitly reference the bypass receipt (or authorization token) that enabled it, including its scope, approver, expiry, and policy exception. Without that cryptographic linkage, you can prove a bypass happened and you can prove an action happened, but not that this specific action was legitimately covered by that specific bypass.
I like your analogy of “the rule moved.” It’s essentially another form of TOCTOU, except the changing object is the authorization policy rather than the target resource. That’s definitely something I’ll cover when I write the case study.
If I had to make this auditable, I would separate the evidence layer from the agent framework. Traces are useful for debugging, but audit evidence needs stable event IDs, policy version, approver, external write, and a tamper-evident chain that survives a vendor swap. Otherwise the compliance story depends too much on whichever orchestration UI happened to record the run.
100% agreed. That's exactly why the receipt schema in the post includes policy_version, previous_hash, and a standalone Ed25519 signature, not just whatever the orchestration UI happened to log. The hash-chain survives a vendor swap because each receipt is self-contained — you can replay the chain from any storage backend, not just LangSmith or LangFuse. The hard part we're finding is making this lightweight enough that small teams actually adopt it. Everyone agrees on the theory; the implementation friction is the real barrier. Have you built something like this in production? Would love to hear what evidence layer you landed on.
I'm actually building something like that. V.E.L.O.C.I.T.Y. IDE, which uses a merkle root SiteMap and VC system. Essentially a drift-prevention mechanism, with a full audit trail, including agent context, while also allowing multi-tenancy with realtime discourse handling, instead of arbitrary git-merges.
Auditable is not nice to have feature anymore, its a must feature
Some comments may only be visible to logged-in visitors. Sign in to view all comments.