Memory · Deep Dive
Every LLM has a hard limit on how many tokens it can process at once. Once the conversation grows beyond that limit, something has to give. The naive approach is a bigger window, but that only delays the problem and introduces attention dilution, latency, and cost.
Effective management treats the window as a scarce resource. Allocate a budget, prioritize what matters, compress what you can, and discard what you do not need.
The system prompt is always present. Recent conversation turns stay intact. Pinned facts (preferences, credentials) survive pruning. Active tool results belong in context until consumed.
Old turns are compressed into a single summary message via a faster model call. Retrieved facts are fetched from external store on demand. Both compete for the same token budget — keep them concise.
Larger windows cause attention dilution (more noise), higher latency (more tokens), and greater cost (per-token pricing). A smarter 4K window beats a noisy 128K window.
Concrete Example
class ContextManager:
def __init__(self, max_tokens, count_tokens, summarizer, retriever):
self.max_tokens = max_tokens
self.count_tokens = count_tokens
self.system_prompt = None
self.pinned_facts = []
self.recent_messages = []
self.summary = None
def prune_to_budget(self):
alloc = {"system": 0.20, "pinned": 0.05, "recent": 0.50, "summary": 0.15, "retrieved": 0.10}
output = []
budget = self.max_tokens
if self.system_prompt:
output.append({"role": "system", "content": self.system_prompt})
for fact in self.pinned_facts:
t = self.count_tokens(fact["content"])
if budget * alloc["pinned"] - t >= 0:
output.append(fact)
if self.summary:
output.append({"role": "system", "content": f"[Summary: {self.summary}]"})
for r in self.retriever(self.query, top_k=3):
t = self.count_tokens(r)
if budget * alloc["retrieved"] - t >= 0:
output.append({"role": "system", "content": f"[Retrieved] {r}"})
for msg in reversed(self.recent_messages):
t = self.count_tokens(msg["content"])
if budget * alloc["recent"] - t >= 0:
output.append(msg)
return outputDivides the token budget into five tiers: system (20%), pinned (5%), summary (15%), retrieved (10%), and recent (50%). Each tier fills independently within its allocation.
Allocate fixed percentages of the context window to different content types.
Compress old turns into a compact summary preserving the gist.
Fetch relevant facts from external store only when needed.
Remove completed results and irrelevant turns first. Keep pinned items.