Adding error monitoring nearly leaked my users' API keys

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

I found four places where my app fails without telling anyone. Fixing them meant sending those failures to Sentry, and the test I wrote to prove the fix worked failed for a reason I did not expect at all: the report would have carried the learner's whole conversation and a decrypted API key along with it.

The near miss turned out to be the more useful half, but first the bug.

Project Overview

CodeTrain is an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote and grades it. The control plane is FastAPI on Fly, Postgres on Neon, and it has had Sentry wired in since June.

"Wired in" is roughly just that. The entire integration was two calls: sentry_sdk.init() gated on a DSN, and one capture_exception inside a catch-all handler. No custom tags or spans, and no context beyond the environment name. It had never caught anything, which means there was no obvious issues right?

Bug Fix or Performance Improvement

None of the four is sloppiness. Each one is a piece of defensive code doing exactly what it was written to do, but with invisibility as an unintended side effect.

1. Every model provider failure. run_managed_turn is the single path every metered model call goes through. It catches provider exceptions and re-raises them as a clean 502:

except Exception as exc:
    raise HTTPException(
        status.HTTP_502_BAD_GATEWAY, f"model provider error: {str(exc)[:300]}"
    ) from exc

That is good API behaviour and a total blind spot. FastAPI routes HTTPException to its own handler, which is not my catch-all handler, so capture_exception never sees it. Every provider outage this product has had was a clean 502 for the learner and complete silence for me.

2, 3 and 4. Three places that swallow malformed model output. When a model returns something that is not the JSON the prompt asked for, the parser falls back to a plausible default:

except (ValueError, json.JSONDecodeError):
    raw_steps = []

No log line. The learner gets a lesson with no steps, or a "retry" verdict assembled from raw model prose, and nothing anywhere records that the model went off contract.

This is the same shape as the bug your test cannot see, which I wrote about last week: a failure that throws tells you where it is, and a failure that returns a plausible value does not. You cannot grep for something that was never written down.

Code

The provider site now reports before it converts. Three lines:

except Exception as exc:
    observability.capture_exception(
        exc, model_tier=model_tier, surface=surface,
        provider="managed" if used_managed or key_provider is None else key_provider,
        session_id=session_id or "none", repo=str(repo),
    )
    raise HTTPException(
        status.HTTP_502_BAD_GATEWAY, f"model provider error: {str(exc)[:300]}"
    ) from exc

The three parse sites were harder, because of a constraint specific to my codebase. The tutor engine exists twice: once in the control plane, and once in a CLI agent that ships as a tarball and has no Sentry wired in. Importing a control plane service into the shared engine would have broken that. So the engine declares a hook and defaults it to nothing:

def on_parse_failure(site: str, **meta) -> None:
    """Called when model output could not be parsed and a fallback was returned instead.

    A no-op by default, and deliberately so: the engine stays free of any control-plane
    dependency, and importing it bare must never need Sentry. `app.main.create_app`
    rebinds this to `services.observability.note_parse_failure`.
    """

and each fallback reports through it:

except (ValueError, json.JSONDecodeError) as exc:
    on_parse_failure("course", chars=len(text), lang=lang,
                     found_object=isinstance(exc, json.JSONDecodeError),
                     error=_parse_error(exc))
    raw_steps = []

Note what is in that payload and what is not. Length, whether a JSON object was located at all, and json's own positional complaint. Never the model's output. That constraint is the whole next section.

My Improvements

What actually changed:

Plus 24 tests, six of them on the rate limiter alone, and a script that reproduces all four failures on demand.

Best Use of Sentry

I wrote the tests for this against a real Sentry client with a capturing transport, rather than mocking my own wrapper. A mock would only have proven that my code calls the function I told it to call. I wanted to see the actual bytes.

The test asserted that a captured provider failure contains no prompt and no key. It failed immediately.

Sentry attaches every stack frame's local variables to an exception event, and send_default_pii=False does not seem to cover them. That flag governs request and user data. Frame locals are a separate option, include_local_variables, and it does default to on.

Look at where that lands. The frames around a failed provider call hold messages, which is the learner's entire conversation, and org_creds, which is a customer's decrypted BYO API key.

Here is that event, sent with the SDK's default configuration:

key_override in plaintext. org_creds in plaintext, same key again. The full conversation. The system prompt.

The part that almost fooled me

My first version of that reproduction named its variables api_key and credentials. Sentry came back like this:

[Filtered], [Filtered]. If I had stopped there I would have written a very confident paragraph about how Sentry protects you automatically, and I would have been wrong.

Sentry's server-side scrubber matches on field name. api_key and credentials are on its list. org_creds and key_override are not, and neither is messages. Two things follow. The protection evaporates the moment you rename a variable, and it happens after transmission anyway, so it is redaction at rest rather than a control over what leaves your server.

Look again at that second screenshot, though. prompt is sitting there in plaintext, directly underneath two [Filtered] rows. Even in the case where the scrubber works, it never had any opinion about user content.

The fix

One option, in one place, shared by production and the test suite so they cannot drift:

def client_options(settings) -> dict[str, Any]:
    return {
        "environment": settings.app_env,
        "traces_sample_rate": settings.sentry_traces_sample_rate,
        "send_default_pii": False,
        "include_local_variables": False,
    }

Same failure, same frame, after:

The test that found this generates its secrets at runtime, so they cannot appear in the source context Sentry also attaches, and it fails if anyone flips the option back:

def test_frame_locals_never_reach_sentry(monkeypatch):
    prompt = "learner asked: " + uuid.uuid4().hex
    api_key = "sk-live-" + uuid.uuid4().hex
    ...
    assert prompt not in blob, "the learner's prompt reached Sentry via frame locals"
    assert api_key not in blob, "a decrypted BYO API key reached Sentry via frame locals"

I checked it is not a vacuous guard by flipping the option back and watching it go red.

Keeping the events useful

Scrubbing locals is only correct if what remains still tells you something. Tags carry the operational shape of the turn, and a parse failure carries enough to diagnose it without a byte of model output:

Expecting ',' delimiter (char 52) is json's own positional message. It says where the model's output broke without repeating any of it.

Not spending the month's quota on one bad afternoon

I am currently on the free plan, 5k errors a month, and Sentry's own per-key rate limiting is a paid feature. A provider outage fails every turn at once, so uncapped, one bad afternoon spends the month and then Sentry drops everything afterwards silently. That is the bug I just fixed from the perspective of my codebase.

So the ceiling lives in my code: a token bucket per failure kind, five events immediately because an incident should be visible on its first failure, then one an hour while it continues. A simulated 24 hour outage of 86,400 failures sends only 28 events.

The cap is only allowed to be quiet about volume, never about the fact of it. Every event that gets through reports what it held back:

201 failures, 6 events, and the last one says 195. The local log line still fires every time, because Fly's logs are free and an incident should stay fully reconstruct-able.

Agent tracing

The tutor is an agent loop, so I wrapped one turn in a manually created transaction. Sentry's AI instrumentation picked up the model call on its own:

Nineteen spans for one turn: the entitlement checks, the budget queries, the model call, the usage insert. gen_ai.chat, claude-haiku-4-5, 971 in and 248 out, cost to four decimal places, context utilisation at 1%.

Worth noticing what the Input panel says on that span: "No input for this span." The trace tells me the shape of the turn, which step, which model, how long, what it cost. It does not tell me what anybody typed, which is the same boundary everything else here is drawn on.

That is also why the transaction is created by hand rather than by raising the global sample rate. Raising it auto-instruments every request the app serves, which is unbounded volume for visibility I only want on model turns. Production runs at 0.0, so the transaction is created unsampled and never sent, and the traces above come from a dev environment. My privacy policy discloses Sentry for error monitoring, and I would rather keep the code inside that promise instead of widen it.

About the impact, honestly

I can tell you exactly how many learners have been hit by the parse bugs, because the fallback feedback is persisted and I can count it with a query like:

SELECT count(*) FILTER (WHERE r->>'fail_md' LIKE '%"verdict"%') AS parse_fallbacks
FROM study_sessions s
CROSS JOIN LATERAL jsonb_array_elements(coalesce(s.state::jsonb -> 'record', '[]'::jsonb)) AS r;

And the grand total of real customers affected = Zero. Across every completed step since late July. I then ran a real lesson through the instrumented build, four model calls including a wrong answer and a follow-up question, and got zero there too.

I could have left that out. It is a better story if the bug was hurting people. But the honest result is that instrumentation turned "I have no idea whether this happens" into a number, and the number is zero, and knowing that is worth something on its own. The parse sites were structurally unable to report, which is the bug, whether or not it has fired yet.

The leak is the one with teeth, and even there I want to be precise. My catch-all handler has had include_local_variables on since June, so the exposure was real for about seven weeks. I audited every event in the project before publishing this. Nothing sensitive had been captured. It never fired in a frame that held a real key, so nothing needs rotating.

What I would tell you to go check

If you run the Python SDK with send_default_pii=False and assume that covers you, open any exception event you already have and expand a frame. Everything in scope at the moment it threw is in there.

Then check it with your real variable names, not with something called api_key. That was the difference between a paragraph that was wrong and one that was right, and it took one extra test to find out which one I was writing.