Context Compaction: Summaries, Scratchpads, and Handoffs

In long-running agent architectures, context compaction is essential because the context window acts as a shrinking bottleneck. As interactions span dozens of turns, the input sequence grows, but the model’s ability to attend to relevant information does not scale linearly. This structural inefficiency causes token costs to rise quadratically while retrieval accuracy degrades due to noise accumulation. Context compaction solves this by actively managing the lifecycle of information, treating memory as a managed resource rather than a passive log.

Part 3 of the Context Engineering series.

The Mechanism of Attention Dilution

To understand why compaction is necessary, one must look at the attention mechanism. Transformers process all tokens in the context window simultaneously, calculating attention weights between every pair of tokens. As the sequence length $N$ increases, the computational complexity grows quadratically ($O(N^2)$) in self-attention implementations, and more critically, the density of relevant signals dilutes. Research indicates that models exhibit a "lost in the middle" phenomenon, where information placed at the beginning or end of a long context is retained, but information in the middle suffers significant performance degradation.

Naive context accumulation—simply appending every user turn and agent response to the prompt—forces the model to attend to stale, irrelevant, or contradictory historical data. This increases latency and cost without improving output quality. Compaction techniques intervene by restructuring how data enters the context window, ensuring that only high-signal state is present at inference time.

Summaries as State Compression

The most common form of compaction is summarization, but it must be implemented as a state update, not just a text reduction. In a long-running session, the agent periodically compresses the conversation history into a concise summary that captures the current state of knowledge, goals, and constraints.

Consider an agent managing a multi-step software deployment. Instead of passing the last 50 turns of dialogue, the system triggers a summarization step when the context window reaches a threshold (e.g., 80% capacity). A specialized LLM call generates a structured summary:

{
  "current_goal": "Deploy v2.1 to staging",
  "status": "waiting_for_approval",
  "key_decisions": ["Used PostgreSQL 14", "Disabled auto-scaling"],
  "pending_actions": ["Review security scan results"]
}

This summary replaces the raw history in the context window. The mechanism here is lossy compression: trivial conversational filler and resolved tasks are discarded, while critical state changes are preserved. The agent continues execution using this compressed state as its "working memory." This reduces the context size from thousands of tokens to a few hundred, drastically cutting cost and improving the signal-to-noise ratio for subsequent turns. However, this approach risks losing fine-grained details if the summarization model is not prompted to preserve specific entities or code snippets.

Scratchpads for Transient Computation

While summaries handle persistent state, scratchpads manage transient computation. In complex reasoning tasks, agents often generate intermediate steps, such as chain-of-thought reasoning or tool call attempts. These intermediate tokens consume context window space but do not constitute persistent knowledge.

A scratchpad is a designated section of the context window used for these ephemeral thoughts. Crucially, the scratchpad is cleared or overwritten after each turn, rather than accumulating. For example, when an agent needs to calculate a complex financial projection, it might write out step-by-step arithmetic in the scratchpad:

[Scratchpad]
Step 1: Calculate base revenue: $1000 * 1.05 = $1050
Step 2: Apply discount: $1050 * 0.9 = $945
Final Answer: $945

Once the final answer is extracted, the scratchpad content is removed from the context for the next turn. This prevents the context window from filling up with repetitive reasoning steps. The mechanism here is isolation: by separating transient computation from persistent history, we ensure that the context window only carries forward what is necessary for the next decision, not the entire path taken to get there. This is particularly effective in multi-step reasoning where the path is long but the state change is minimal.

Handoffs as Context Partitioning

The most sophisticated form of compaction is handoff, which partitions context across multiple agents rather than within a single one. In a monolithic agent architecture, the context window must contain all knowledge relevant to the current task, leading to massive windows when dealing with diverse domains.

Handoffs involve transferring a sub-task to a specialized agent with a fresh, clean context window. For instance, a primary coordinator agent might receive a request involving both code generation and legal compliance review. Instead of handling both, it hands off the legal review to a "Legal Agent" with a specific prompt and relevant documents, while the coordinator waits. The Legal Agent operates in an isolated context window, free from the noise of the code-related history.

This mechanism reduces context size by limiting the scope of each agent’s window to its specific domain. The handoff itself is a compact instruction, typically under 500 tokens, that transfers the necessary state. Implementation requires serializing the current context state into a structured payload (such as JSON) that the receiving agent can parse and initialize its internal state machine. The key trade-off is increased latency due to multiple model calls, but the benefit is significantly higher accuracy and lower token cost per agent, as each operates in an optimized, small context. This approach is particularly critical for long running agents that must maintain high precision over extended operational lifecycles.

Common Pitfalls

When implementing context compaction, developers often encounter several recurring issues:

1. Over-Summarization: Aggressive compression can strip away nuanced context or critical edge cases, leading to hallucinated responses in subsequent turns.
2. Scratchpad Leakage: Failing to properly clear or overwrite scratchpad content can cause previous reasoning steps to bleed into new tasks, creating logical contradictions.
3. Handoff Latency: While handoffs reduce context size, the overhead of serializing state and making multiple API calls can introduce noticeable latency if not batched or optimized.
4. State Inconsistency: If the summarization model fails to capture a recent user correction, the agent may proceed with outdated instructions, requiring manual intervention.

Practical Takeaways

To effectively implement context compaction in your agent architectures, consider these actionable steps:

• Implement Dynamic Thresholds: Trigger summarization based on token usage percentages (e.g., 70-80%) rather than fixed turn counts to adapt to varying context lengths.
• Isolate Transient Logic: Use dedicated scratchpad sections for intermediate reasoning steps and ensure they are cleared after each inference cycle.
• Standardize Handoff Protocols: Define clear schemas for state serialization during agent handoffs to ensure seamless context transfer between specialized agents.

FAQ

Q: How do I decide between summarization and handoffs? A: Use summarization when the task remains within a single domain and the context is growing linearly. Opt for handoffs when the task spans distinct domains (e.g., coding vs. legal) or when the context size would otherwise exceed practical limits for a single model.

Q: Can I use RAG instead of context compaction? A: RAG is complementary, not a replacement. Context compaction manages the immediate conversation history within the window, while RAG retrieves external knowledge. For long-running agents, you should use compaction to manage the chat history and RAG to fetch relevant documents as needed.

Q: What is the impact of compaction on model accuracy? A: Properly implemented compaction generally improves accuracy by reducing noise and focusing attention on relevant signals. However, poor implementation (e.g., over-summarization) can degrade performance by losing critical context.

Conclusion

Context compaction is not a single technique but a layered strategy. Summaries compress history, scratchpads isolate transient computation, and handoffs partition responsibility. Together, they transform the context window from a passive log into an active, managed resource. For developers building long-running agents, ignoring compaction is equivalent to ignoring memory management in traditional software engineering. The result is always eventual system failure, whether through cost overruns or degraded performance.