State and Context

The FIFO sliding window is the simplest context management strategy: maintain a deque of entries with a running token count, evict from the left when the total exceeds the limit. Its main strength is predictable O(1) behaviour and implementation simplicity. Its critical weakness is that early context, especially the original user goal, may be evicted after enough steps, causing the agent to lose sight of its objective. I have seen this manifest as the agent suddenly asking "What was the original task?" halfway through execution. Two mitigations work well in practice. First, goal pinning: reserve a non-evictable section at the top of the context that always contains the original goal. This adds a fixed token overhead (typically under 100 tokens) regardless of step count. Second, periodic summarisation: every N steps, call the LLM to compress the full context into a brief summary that captures the goal, key findings, and current state. Replace the raw context with the summary. This is more token-efficient but risks losing nuance. An alternative strategy is structured state: maintain a JSON object with well-defined fields like current_goal, findings, pending_tasks. This is more predictable but less flexible and requires schema design upfront.

This is context drift from unbounded growth. Each step adds text to the context; around step 8 the accumulated history exceeds the token limit, and the sliding window evicts the oldest entries — including the system prompt with the original goal. The agent then reverts to base behaviour, asking for a goal it already received. Two solutions: first, goal pinning — a reserved context section that is never evicted. The original goal stays in pinned storage across all steps. This is simple but consumes tokens permanently (maybe 50 tokens per step, negligible). Second, periodic summarisation: every 3-5 steps, call the LLM to compress accumulated context into a brief summary that includes the original goal and key findings. Replace raw context with the summary. This is more token-efficient but risks losing nuance during compression. In production, I use a hybrid: pin the original goal and the most recent 2 steps, summarise the middle history.

Each strategy optimises a different axis. Sliding window is simplest — keep last N tokens, drop the rest. Best for short, linear interactions where only recent context matters. Fails when early information is needed later — common in research and debugging. Summarisation compresses history via LLM calls. Best for long-running assistants where token budget is tight. Fails when the summary omits a detail that becomes important later, like a specific user constraint. Structured state maintains a JSON object with well-defined fields (current_goal, findings, pending_tasks). Best for data-processing pipelines where state shape is pre-defined. Fails for unpredictable tasks like creative writing. For an adaptive manager, I monitor three signals: context growth rate per step, early-reference frequency (how often the LLM mentions early entries), and error rate from repeated questions. When early-reference frequency drops, sliding window suffices. When it stays high, I switch to summarisation. When tasks have clear structure, I suggest structured state as a configuration option.