Track
Window · Summarize · Retrieve · Evict
LLMs have a hard limit on how much text they can process at once — the context window. Once the conversation grows beyond that window, older messages get dropped. This is fine for a single Q&A, but agents often need to maintain context across many turns, or remember facts from earlier in a long session.
Memory systems solve this by selectively deciding what to keep, what to compress, and what to discard. Instead of one fixed window, you build a tiered system: recent messages stay intact, older ones get summarized, and important facts get pinned so they're never evicted.
The simplest memory strategy is a sliding window: keep the most recent messages within a token budget, always preserve the system prompt, and drop the oldest non-system messages when the budget is exceeded.
When the budget fills, the oldest messages are evicted. The agent can still see the recent ones plus the system prompt. This keeps memory bounded and predictable.
Sliding windows work by recency, but sometimes the most relevant piece of information isn't the most recent. Query-based retrieval lets the agent search memory by meaning rather than time.
The agent stores facts as they accumulate, and when it needs something specific, it queries the store. This turns memory from a simple FIFO queue into a searchable knowledge base.
Short-term memory covers the current task: the sliding window, the scratchpad, intermediate state. It lives entirely inside the context window and resets between conversations.
Long-term memory holds durable facts that survive across sessions: user preferences, credentials, project config. Retrieved memory stores facts in an external index and pulls them in on demand.
Choose the tier by lifespan. Session state belongs in short-term. Facts the user expects the agent to remember tomorrow belong in long-term storage.
Memory pollution happens when you store facts that should have stayed ephemeral: saving a temporary observation as a permanent preference, treating a single mention as confirmed, keeping stale data.
A memory write policy prevents pollution: classify data before storage. Ephemeral facts stay in session context. Persistent facts need a clear source of truth and a timestamp for expiry.
Layer your strategies: keep recent N turns, pin system instructions (never evicted), summarize old turns, retrieve only relevant facts from long-term storage. Bigger windows dilute attention and increase cost — use a minimal sufficient context.
Concrete Example
class SlidingMemory:
def __init__(self, max_tokens, count_tokens):
self.max_tokens = max_tokens
self.count_tokens = count_tokens
self.messages = []
def add(self, role, content):
self.messages.append({"role": role, "content": content})
def render(self):
system = [m for m in self.messages
if m["role"] == "system"]
others = [m for m in self.messages
if m["role"] != "system"]
budget = self.max_tokens
for m in system:
budget -= self.count_tokens(m["content"])
kept = []
for msg in reversed(others):
tokens = self.count_tokens(msg["content"])
if budget - tokens >= 0:
budget -= tokens
kept.append(msg)
return system + list(reversed(kept))System messages are pinned — never evicted. Non-system messages are scanned newest-first; each fits within the remaining budget or is dropped. The result is always within max_tokens while retaining the most recent context.
Keep the newest messages within a token budget; drop oldest first.
System prompts are always preserved and never evicted.
Compress dropped context into a compact summary before removal.
Search memory by semantic relevance, not just recency.
Automatically expire facts that are too old, keeping the store fresh.
Classify data as ephemeral or persistent before writing to long-term storage.
Explore these deeper articles to master specific topics in Memory.
Interview revision
6 problems. Sign in to start solving.
Sign in to open a workspace and solve these problems.