Short-Term Memory: Summarization and Compaction

In Large Language Model (LLM) architectures, short-term memory is often mischaracterized as a simple queue of past messages. In reality, it is a constrained, expensive, and noisy resource. As an agent interacts with a user, the context window—the sequence of tokens fed to the model—grows linearly. This growth introduces two critical engineering failures: latency (processing time increases with sequence length) and cost (token pricing is applied to every input). Furthermore, the attention mechanism within the Transformer architecture suffers from "attention dilution," where early, potentially crucial instructions become statistically less influential as the sequence lengthens.

To build viable agents, we must move beyond simple storage. We need mechanisms to actively manage this memory through context management strategies, specifically summarization (semantic reduction) and compaction (structural optimization).

The Mechanics of Decay

Before implementing compaction, we must understand what we are trying to preserve. An agent’s short-term memory consists of three types of information:

1. System Instructions: Static rules defining the agent's role. These rarely change.
2. Recent Context: The last 3–5 turns. These are high-fidelity, low-latency requirements for immediate response generation.
3. Historical Context: Older turns. These provide background but are low-priority for immediate logical deduction.

If we naively append every new message to the context window, we eventually hit the model’s maximum token limit. Simple truncation (cutting off the beginning) is dangerous because it can delete critical system instructions or initial constraints. Instead, we use compaction to reduce the token footprint of historical context without losing its semantic essence.

Summarization as State Compression

The core mechanism for managing long-term short-term memory is conversation summarization. This involves introducing an intermediate step in the agent’s loop: before generating the final response, or periodically during the conversation, the system calls the LLM (or a smaller, cheaper model) to summarize the conversation history up to that point. This process is often referred to as compaction when applied to the structural flow of the agent's state.

How It Works

Consider a dialogue with 50 turns. Instead of sending all 50 turns to the next inference call, the agent performs a "compact" operation:

1. Extract History: Retrieve all messages from the start of the session up to the most recent turn.
2. Prompt for Summary: Send a prompt to the LLM requesting a concise summary.

   summary_prompt = f"""
   Summarize the following conversation history. 
   Focus on key facts, user intents, and decisions made. 
   Do not include filler or conversational pleasantries.
    
   History:
   {conversation_history}
    
   Summary:
   """

3. Generate Summary: The LLM returns a condensed text block.
4. Replace History: The raw history is replaced in the context window with: [System Instructions] + [Summary] + [Recent 5 Turns].

The Trade-off: Fidelity vs. Efficiency

This approach achieves a significant reduction in token count depending on the conversation style. However, it introduces information loss. The summary is a probabilistic reconstruction of the original text. If the user provided specific, obscure details in turn 10 that are critical to turn 50, the summary might omit them.

Opinion: For most general-purpose agents, this loss is acceptable because the LLM can infer missing details from the summary if they are semantically relevant. For highly structured tasks (e.g., legal review), summarization should be avoided in favor of retrieval-augmented generation (RAG) or explicit vector storage.

Compaction and Recency Bias

A more sophisticated approach combines summarization with recency bias. The human mind (and the Transformer’s attention mechanism) prioritizes recent information. Therefore, we do not summarize the entire history uniformly. We apply a sliding window strategy.

The Hybrid Data Flow

1. Raw Window: Keep the last $N$ turns (e.g., 6 turns, ~1000 tokens) in their raw form. This ensures the model has precise access to the most recent context, including any code snippets, JSON structures, or specific user edits.
2. Summarized Buffer: Everything before the raw window is passed through the summarization process described above.
3. Merge: The final context sent to the LLM is:

   [System Prompt]
   [Summary of Old History]
   [Turn N-5]
   [Turn N-4]
   [Turn N-3]
   [Turn N-2]
   [Turn N-1]
   [Current User Input]
   

This structure minimizes the risk of losing critical recent details while keeping the total token count stable.

Implementation Considerations

When implementing this, you must handle state transitions carefully. If the summarization step fails or returns an empty string, the agent must fallback to a safe state (e.g., truncating from the end or asking the user to restart). Additionally, the summary itself becomes part of the context for the next turn. This means the summary is effectively "compressed again" if the conversation continues, leading to recursive degradation. To mitigate this, some systems reset the summary periodically or use a "rolling summary" that updates incrementally rather than regenerating from scratch.

Conclusion

Short-term memory in agents is not about storing everything; it is about retaining enough signal to make accurate decisions while minimizing noise and cost. Summarization and compaction are not just optimization tricks—they are fundamental architectural components that allow agents to scale beyond fixed context windows. By separating raw recent context from summarized historical context, engineers can build systems that are both efficient and semantically coherent. This approach is a cornerstone of effective LLM optimization, ensuring that agents remain responsive and cost-effective over long interactions.