What is a context window in AI, and why does it fill up?

In detail

What goes in the window

A context window is measured in tokens, the word pieces a model reads and writes. Anthropic’s context window documentation defines it as all the text a model can reference when it generates a response, including the response itself. Everything in a request counts toward it: the system prompt, every message so far, tool results, images and documents, and the definitions of the tools the model may call.

Limited space is one reason to retrieve information as needed. LlamaIndex’s Jerry Liu recalls that the models builders were using when his framework took off had a 4,000-token context window, and a company’s knowledge base in Notion, Jira or Salesforce was never going to fit in it. He connects that problem to the use of retrieval-augmented generation in 2023. The technique predates that period: Lewis and colleagues published the RAG paper in 2020. Retrieval also supplies private or current information that a model did not learn in training.

Why it fills up

For each response, the model uses the history the application supplies. Anthropic’s standard API example keeps earlier messages in full, so every user message and response accumulates in the window. Applications can instead remove older messages, clear tool results or replace history with a summary. An agent adds more material through tool calls and their results.

That growth is fast for agents. SwirlAI’s Aurimas Griciūnas says anyone building a multi-turn agentic system has felt the pain of an agent running too long, with the context window exploding to a few hundred thousand tokens in a single run.

Why a full window is a problem

The first cost is money and time. Retained history counts as input again on later turns. Prompt caching changes what you pay, while cached tokens still occupy the window; a larger input can also take longer to process.

The second cost is accuracy. Anthropic’s documentation says more context isn’t automatically better: as the token count grows, accuracy and recall degrade, a pattern it calls context rot. The 2023 Lost in the Middle study found that, for the models and tasks it tested, performance was best when the relevant information came at the start or end of a long input and significantly worse when it sat in the middle. That result describes those tests, rather than measuring today’s million-token models.

Kapil Chhabra of WisdomAI describes the same trade-off from the product side. If you put the whole company history and every document into the window, he says, the model has to guess which part is the right context, and you burn tokens. If you send too little, you still leave room for the model to guess, which is where hallucination comes in. His answer is just the right amount of context for each prompt.

How applications make room

Three moves cover most of what applications do, and choosing between them is the work of context engineering:

• Retrieve instead of preload. Keep documents outside the window and fetch only the passages a task needs, which is what RAG does.
• Summarize. When a conversation nears the limit, replace the older turns with a summary and continue. Anthropic calls this compaction and says the hard part is choosing what to keep, because an aggressive summary can drop details that matter later.
• Store and fetch later. Move material out of the window into files or a database and leave a pointer the model can search.

Richmond Alake of Oracle draws the line between the last two. Summarizing the window and putting the summary back is, in his terms, context engineering. Once you move the uncompressed material to an external store and give the agent a way to retrieve it on demand, you are doing memory engineering, and new questions follow about when and how to retrieve it. He initially contrasts “compaction” as store-and-retrieve with summarization, but also refers to a compact command generating a summary. Anthropic calls summarization compaction, so check the mechanism a speaker means rather than relying on that label alone. The fuller comparison is in prompt vs. context vs. memory engineering, and the kinds of agent memory are in the 4 types of AI agent memory.

Disclosure: Episode 75 was sponsored by WisdomAI, Kapil Chhabra’s company; Conor thanks the company at 4:13. Episode 53 is sponsored by Galileo, the Season 3 presenting sponsor; the read is at 33:32. Episode 38 was sponsored by Galileo, where Conor then worked; he points listeners to Galileo at 28:13.