roiscale

Long context windows relocated the memory problem rather than solving it; the teams shipping reliable agent systems in 2026 are doing explicit context engineering — working memory design, retrieval scaffolding, and deliberate forgetting strategies.

BUILDS ON

Stop Calling It Prompt Engineering (V1)

The Mistake Most Teams Make

A healthcare client came to me nine months ago with a problem they were framing as a model quality issue. Their clinical-note summarization agent was producing inconsistent outputs — sometimes excellent, sometimes factually confused, occasionally fabricating medication names. Their instinct was to upgrade the model. I asked to see the prompt first.

What I found was a context window stuffed like a carry-on bag before a budget flight. The agent was receiving the full patient record, the last fourteen days of nursing notes, the admission note, the discharge criteria, the formulary list, and a six-paragraph system prompt — all concatenated in a flat string, passed to a 128K-context model with no structure, no prioritization, and no signal about what was actually relevant to the current question.

They were not doing prompt engineering. They were doing context dumping. The two are not the same thing, and conflating them is costing teams a staggering amount in wasted inference spend and avoidable failures.

Context engineering is the explicit discipline of deciding what information occupies which position in a language model's context window, at what time, and for how long. It includes working memory design, retrieval scaffolding, structured forgetting, and priority signaling. Most teams have never sat down to design any of it.

Why It Persists

The problem has a clear origin: long context windows were marketed as a memory solution. When Anthropic extended Claude to 100K tokens in 2023, and Google later pushed Gemini to 1M tokens, the implicit narrative was that teams could stop worrying about context management. Just put everything in. The model will figure it out.

This narrative was commercially convenient and architecturally catastrophic.

The research caught up faster than the marketing. Liu et al.'s "Lost in the Middle" study (2024) demonstrated that language models systematically underweight information positioned in the middle of long contexts, with recall accuracy dropping roughly 41% on tasks requiring integration of facts that appear far from the beginning or end of the input. The effect is model-agnostic and robust across architectures. Putting more tokens in does not make the model better at attending to all of them; it makes the effective attention window more uneven.

Stanford's work on many-shot in-context learning (2025) showed a parallel problem: when example sets grow beyond a certain density, the model begins weighting recent examples disproportionately, essentially producing recency bias in an architecture that was supposed to be recency-blind. Both findings point to the same architectural conclusion: context is a scarce, unevenly-valued resource, not a limitless buffer.

The persistence of the anti-pattern has a second, more human cause. Context dumping is fast to implement. In a sprint-driven team trying to ship a feature, the path of least resistance is to grab every piece of possibly-relevant data and concatenate it. Designing a proper working memory layer takes architecture time that most teams haven't budgeted.

What the Literature Actually Says

The right mental model for context in a deployed LLM system is not a filing cabinet. It is closer to a register file in a CPU — a fast, limited resource that holds what the current operation actually needs, backed by slower memory tiers for everything else.

Anthropic's internal work on context management (published in their engineering cookbook, Q4 2024) introduced the concept of contextual retrieval — pre-processing document chunks to generate context-aware summaries that explain each chunk's relationship to the overall document before embedding it. This addresses one of the core retrieval failures: a chunk that makes sense in context is retrieved, but arrives in the model's window stripped of the framing that made it sensible. The result is a 49% reduction in retrieval failures on the benchmarks Anthropic published, at modest additional cost.

The broader principle is that retrieval and context management are not separate concerns. The retrieval system decides what enters the context window. The context design decides where in the window it goes, what surrounds it, and what gets evicted when space is constrained. Treating these as two independent systems is the root of most of the failures I've seen in production.

The Architecture That Works

A properly designed context stack for an agent system has four layers, each with different latency and eviction characteristics:

┌─────────────────────────────────────────────────────────┐

│ CONTEXT WINDOW (active working memory) │

│ ┌──────────────┬───────────────┬───────────────────┐ │

│ │ System+role │ Task scaffold │ Retrieved chunks │ │

│ │ (pinned) │ (pinned) │ (scored+ranked) │ │

│ └──────────────┴───────────────┴───────────────────┘ │

│ ┌──────────────────────────────────────────────────┐ │

│ │ Conversation history (compressed/summarized) │ │

│ └──────────────────────────────────────────────────┘ │

├─────────────────────────────────────────────────────────┤

│ EPISODIC BUFFER (session-scoped, not in-context) │

│ Full turn history, tool outputs, intermediate states │

├─────────────────────────────────────────────────────────┤

│ SEMANTIC MEMORY (vector store) │

│ Embedded documents, indexed by relevance │

├─────────────────────────────────────────────────────────┤

│ PERSISTENT STORE (database/KV) │

│ User facts, preferences, long-horizon state │

└─────────────────────────────────────────────────────────┘

The cardinal rule: nothing enters the context window without a recency or relevance score. Pinned sections (system role, task scaffold) are fixed and minimal. Retrieved chunks are ranked by a cross-encoder reranker before insertion, not by embedding cosine similarity alone — similarity is a proxy for relevance, not relevance itself. Conversation history is compressed by a summarization pass once it exceeds a configurable token budget, with a loss-tolerance threshold that triggers only when compression would remove information the task planner has flagged as critical.

Here is the context budget management logic I use in production Python:

 
  
  class ContextBudget:

      def
  __init__(self, max_tokens: int, reserve_for_output: int = 2048):

         
  self.budget = max_tokens - reserve_for_output

         
  self.pinned = 0     # system +
  task scaffold

         
  self.history = 0    # compressed
  conversation

         
  self.retrieved = 0  # ranked
  retrieval chunks

   

      def
  can_fit(self, tokens: int, tier: str) -> bool:

         
  used = self.pinned + self.history + self.retrieved

         
  if tier == "pinned":

             
  return tokens <= (self.budget * 0.20)

         
  if tier == "history":

             
  return (used + tokens) <= (self.budget * 0.50)

         
  if tier == "retrieved":

             
  return (used + tokens) <= self.budget

         
  return False

   

      def
  evict_oldest_history(self, summarizer_fn, turns: list) -> list:

         
  # Compress oldest N turns until history fits budget

         
  while self.history > int(self.budget * 0.30):

             
  summary = summarizer_fn(turns[:4])

             
  turns = [{"role": "system", "content":
  f"[Summary]: {summary}"}] + turns[4:]

             
  self.history = count_tokens(turns)

         
  return turns

The forgetting strategy is as important as the retrieval strategy. In most production failures I've diagnosed, the agent's context was full of information that had been relevant five turns ago but was actively misleading now. A user who says "I changed my mind — let's go with option B" at turn 8 still has "option A is preferred" sitting in the raw conversation history at turn 2, pulling the model toward a stale interpretation.

Failure Modes

Positional confusion: Rotating which chunks appear in which position across different queries introduces noise. Use a consistent template — system/role always first, task-specific context second, retrieved chunks third, history fourth. The model's attention machinery benefits from positional consistency.

Summarization hallucination: History compression via LLM summarization introduces its own error surface. The summarizer can hallucinate details, especially for tool outputs and numerical data. Compress prose exchanges; preserve structured data (tool calls, API responses, numerical facts) verbatim in a structured block.

Over-pinning: Teams that "just pin everything important" end up with a pinned section that consumes 40% of the budget. Anything that doesn't change across turns belongs in a pre-computed system prompt, not dynamically injected on every call. Pinned should mean genuinely invariant.

Retrieval without position management: Inserting retrieved chunks anywhere convenient, rather than ordering them by relevance descending (most relevant closest to the task statement), falls directly into the lost-in-the-middle failure mode. Rank matters, not just selection.

Decision Criteria

When should you reach for a larger context window rather than investing in a retrieval layer? The honest answer: almost never, at scale. Larger windows are the right tool for single-document deep analysis — contract review, paper summarization — where the document is the only relevant input. For any agentic system that operates across a growing knowledge base or a multi-turn session, investing in a proper context engineering layer will outperform window expansion in both cost and accuracy.

The test I apply is simple: if you removed 60% of the tokens currently in your context window at random, would the model's output quality degrade? If yes, your retrieval is doing its job. If no, you are paying for tokens that aren't contributing to correctness.

Closing

Prompt engineering was always misnamed — it was always interface design for model behavior. Context engineering is the deeper layer: it is memory architecture for inference-time systems. The teams that get this right are shipping agents that are cheaper, faster, and more accurate than the teams still treating the context window as a filing cabinet. The research has been clear since 2024. The production gap between teams who've acted on it and teams who haven't is now measurable.

The V1 piece in this series argued that teams should stop thinking of prompts as code and start thinking of them as interface contracts. This piece extends that framing to the full information architecture: the prompt is the interface, and the context is the memory system that backs it. Design both with the same rigor you'd apply to any production data layer.

Production Readiness Checklist

Before shipping any system that relies on context window management, walk through these checks:

• [ ] Token budget is instrumented and emitted as a metric per request (total, pinned, retrieved, history)

• [ ] The summarization pass is tested independently on the most information-dense conversation types

• [ ] Retrieved chunks are position-sorted by reranker score before insertion, not by retrieval order

• [ ] A test exists that deliberately overfills the context budget and verifies graceful degradation

• [ ] Context eviction logic is idempotent — running it twice on the same context produces the same result

• [ ] The system prompt is version-controlled and its token count is tested in CI

What I Would Build Differently

The architecture described above is correct but not complete. The piece I most often wish I had built earlier is a context replay system — the ability to take any production request that produced an incorrect output, reconstruct the exact context window it received, and re-run it with a modified context configuration. Without this, debugging context-related failures requires manual reconstruction from logs, which is slow and error-prone. A context replay system closes the debugging loop the same way eval-driven development closes the quality loop.

Figure 1. Four-tier memory architecture diagram: active context window at top (subdivided into system/pinned, task scaffold, retrieved chunks, compressed history), episodic buffer below, semantic vector stor…

REFERENCES

1. Lost in the Middle: How Language Models Use Long Contexts. arXiv (2024).

https://arxiv.org/abs/2307.03172

2. Contextual Retrieval. Anthropic Engineering Blog (2024).

https://www.anthropic.com/news/contextual-retrieval

3. Many-Shot In-Context Learning. Stanford / arXiv (2025).

https://arxiv.org/abs/2404.11018

4. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). arXiv (2023).

https://arxiv.org/abs/2309.06180

5. MemGPT: Towards LLMs as Operating Systems. arXiv (2023).

https://arxiv.org/abs/2310.08560

6. Anthropic Context Management Cookbook. Anthropic Developer Documentation (2024).