Context window management is the invisible cost in every multi-step agent pipeline.
I see this pattern across production consulting work. An agent that works cleanly in testing — ten steps, coherent reasoning, correct output — starts degrading six weeks after ship. No errors. No crashes. Just drift. Outputs get shorter. Reasoning steps that used to show clear logic start reading like vague summaries. Users start noticing that the agent "feels slower."
The root cause is almost always the same. Context grows with every step and nobody measured it.
How context accumulates
A multi-step agent accumulates its full conversation history by default. Every tool call result gets appended. Every intermediate reasoning step gets stored. The conversation object passed to each model call grows linearly with the number of steps executed.
By step 8 or 9 in a ten-step pipeline, the model is receiving 90% of the context window on each call. Technically within limits, but attention spread thin across thousands of tokens of earlier reasoning that no longer affects the current step.
Here is what this looks like in a real pipeline. Suppose a research agent runs a search, extracts entities, resolves ambiguities, retrieves documents, ranks them, synthesizes a draft, and fact-checks it. That is seven steps. If each step produces an average of 800 tokens of tool output and the model includes its full reasoning trace for each step:
- After step 1: ~1,200 tokens in context
- After step 3: ~4,800 tokens
- After step 5: ~9,600 tokens
- After step 7: ~15,600 tokens
For a 16K context model, that last step is operating with almost nothing left for the actual synthesis task. For a 128K context model, the numbers look more comfortable until you realize you are paying for all of that context on every call, and the model's attention is being diluted across material that stopped being relevant three steps ago.
Why testing does not catch this
Single-query testing never reveals context saturation. A developer runs the pipeline ten times against a test set, sees correct outputs, ships it. What they tested was ten fresh conversation threads, each starting at zero tokens.
Production load looks nothing like that. Users run sequential tasks. A pipeline that runs five requests across the same session accumulates context from all five runs if the session history is not explicitly bounded. An orchestration layer that retries failed steps passes the failure reasoning back into context. Long-running agents that call external APIs and wait for results accumulate tool logs while they wait.
Context growth under real query volume is not a scaling problem in the usual sense. It is a state management problem. The pipeline has no concept of which historical context is still relevant to the current step.
Three things that work in production
Allocate a fixed token budget per step, not per conversation.
If step 3 needs the output of step 1, pass the structured result only — not step 1's full chain-of-thought. A search step that returns ten results does not need to carry its internal ranking rationale forward into the synthesis step. What the synthesis step needs is the ranked list. Truncate aggressively at stage boundaries.
This requires a deliberate design decision about what each step's output contract is. If the output is unstructured text, everything gets passed forward by default. If the output is a typed schema, only the schema fields travel between steps. The type boundary is what enforces the token budget.
Summarize and compact before handoff between pipeline stages.
The detailed reasoning from a retrieval step does not need to be present when the synthesis step runs. A structured summary does. Before passing control from one stage to the next, run a compaction step that takes the full output and produces a fixed-size digest. The digest carries the key facts forward. The full reasoning trace stays available for debugging but does not enter the production call chain.
The compaction step is itself a model call, which costs something. It is almost always worth it. A 3,000-token reasoning trace compacted to a 400-token structured summary saves 2,600 tokens on every subsequent step in the pipeline. On a ten-step pipeline running thousands of queries, the compaction cost is negligible against the savings.
Set a hard context limit per step and fail loudly when exceeded.
Silent context overflow produces subtly wrong outputs that reach users. A model operating with 2% of its context window available for the actual task will not return an error. It will return a plausible-sounding but degraded answer that passes surface-level quality checks. The failure mode is invisible unless you are actively measuring context size per step.
Adding a hard limit is straightforward: measure the token count before each model call and raise an exception if it exceeds a threshold. The threshold should be well below the model's maximum, not a fraction of the maximum. If the step budget is 2,000 tokens and the accumulated context is 8,000, the pipeline is not "within limits" — it is already broken.
Failing loudly means the engineering team sees the problem. Silent degradation means users see it first.
What instrumentation you actually need
Three metrics catch context growth problems before they affect users:
- Tokens-in per step: how many tokens the model receives at each pipeline step. Log this per step, not as a conversation total.
- Context utilization rate: tokens-in divided by the model's context limit. Alert when this exceeds 70% for any step in a production pipeline.
- Step-over-step growth rate: how much context is added between consecutive steps. A step that adds 5,000 tokens is probably passing its full reasoning trace forward.
These are not difficult to collect. A wrapper around the model call captures the prompt token count before sending. The difficulty is that most orchestration frameworks do not expose this by default. You have to add the instrumentation yourself.
The deeper design issue
The reason context growth catches teams off guard is that it does not look like a bug. The pipeline architecture that worked in a stateless test environment is also the architecture that fails under stateful production load. Nothing changed between test and production except the accumulated state.
The fix is to treat context as a resource with a budget, the same way you treat memory or API calls. Each step has an input budget and an output budget. The input budget caps what the step can receive. The output budget caps what the step passes forward. The pipeline enforces both.
An agentic pipeline that has no explicit context budget is running without a key operational constraint. It will work until it does not, and the failure signal will be subtle enough that users notice before dashboards do.
Context window management and instrumentation for production agentic systems are part of the consulting work I do at hannune.ai.
Top comments (9)
The "no errors, just drift" framing is the part teams miss, because every dashboard they have is watching for the wrong signal — latency, error rate, token count in aggregate — and none of those move sharply when quality bleeds out.
The mechanism I'd add: it's not only how much context, it's what fraction of it is stale. A 15K context where 12K is superseded tool output degrades worse than a 40K context that's mostly still-relevant. Attention gets spent proving old steps are irrelevant instead of doing the current one.
What's worked for us is treating each step's input as an explicit budget rather than an append-only log — carry forward a compacted state object, not the full transcript, and let each step declare what it actually needs upstream. It's more plumbing, but it makes the growth visible in code instead of hidden in a conversation array. Do you find summarization-based compaction reliable enough for that, or does it introduce its own drift?
The useful addition here is a replay test that separates context growth from model drift. Persist the compact state and raw event log for each step, then replay the same workload with three policies: append-only history, typed handoffs, and typed handoffs plus bounded excerpts. Record input tokens, output tokens, compaction loss, latency, and accepted-outcome checks per step.
I’d also make the budget failure actionable: stop before the model call when the next step cannot fit, emit the step name and largest contributors, and route to a recovery path rather than silently truncating. That turns “the agent feels slower” into a regression test for state policy. The tradeoff is more durable logging and schema maintenance, but it preserves the raw evidence without making every future call pay for it.
The testing vs production gap you describe is exactly what I keep hitting with agent pipelines that run many steps. Fresh sessions look sharp because every tool result is still relevant, then by the third retry in the same thread the agent starts restating earlier decisions instead of checking the files again. Fixed token budgets per step helped more than a bigger model did. One thing I still struggle with is deciding which intermediate tool output still matters when a step fails mid pipeline and you have to resume.
The typed schema point is the one that paid off most in my own pipeline work. Once each step's output is a typed contract, most of the compaction problem disappears on its own, because you never carry the reasoning trace forward in the first place. The only stage where we still pay for a compaction call is right before the final draft, where the input genuinely is unstructured.
I'd push back a little on compaction being almost always worth it as the default. It is a model call, so it can drift too, and it sits on the critical path for latency. I treat it as the fallback for stages whose output can't be typed, not the standard handoff.
On the alert threshold: 70% utilization was too late for us on a 128K window. Outputs were already getting vague before the counter crossed it. We moved the alert to per-step growth rate instead. A step that suddenly adds 4-5K tokens has been a better leading indicator than any utilization percentage.
The typed handoff is the strongest control here, because it turns “compact the transcript” into a narrower contract: each stage declares the fields it may emit and the next stage declares what it accepts. I’d add two checks to the wrapper: record input tokens, output tokens, and stale-field bytes per step, then run a replay fixture where retries and pagination intentionally inflate the state. Alerting only on context utilization can miss a sharp growth event or a mostly-stale payload. What schema or redaction strategy has held up best when a stage genuinely needs a few verbatim excerpts for auditability?
Interesting perspective. Bigger context windows don't actually solve context management—they often just postpone the problem. In long-running agentic workflows, selective memory, retrieval, and context pruning seem far more scalable than continuously expanding prompts. Curious to hear your thoughts on balancing context retention with performance in production systems.
The six-week degradation with zero errors is the hard part, because nothing in the logs pages you until users start calling the output vague. Context length is one of the few failure modes that gets worse purely with usage, independent of any code change, so it never shows up in a pre-ship test. Are you tracking token count per step as a first-class metric now, or capping history with summarization once it crosses a threshold?
Compaction at stage boundaries is exactly the pattern we landed on. Opportunity Skill's lead engagement does the same thing in chats. When a thread worth following up on passes 10 messages, the agent stops appending to it and opens a new chat that opens with a compacted summary, so the key facts about the lead travel forward and the noise does not. Impressions work the same way, with each one capped at 512 characters. Structured units beat unbounded transcripts. Context discipline is a design decision, not an afterthought.
In my experience particularly when use claude opencode its not context window, context window remain shorts, its multiturn agentic appraoch that consume a lot of tokens