Skip to content

AI Glossary · Concepts

What is Context Window?

Context Window

A context window is the token budget a model can use for a request and its response. Instructions, conversation history, documents, tool results, and generated output compete for that budget; exact accounting depends on the model and API.

Also known as: context length

· Updated · Chain of Thought

Context Management

  • 4 explainers
  • 8 episodes
  • 2 guest clips
  • 1 source

The context window limits what a model can work with during a response. It includes more than the user’s latest message: instructions, conversation history, tool definitions, and tool results also consume space. Generated output needs room too.

Exceeding the budget does not always silently drop the oldest text. Behavior depends on the API, model, and application. A request may be rejected, generation may stop at the limit, or the application may shorten the input through truncation or compaction. Compaction changes the information the next request receives; it does not enlarge the model’s window.

Anthropic’s context-window documentation explains its input/output accounting and model-dependent overflow behavior. Check the documentation for the particular model you deploy.

For builders, the useful check is whether the request preserves the instructions and evidence needed for the task while leaving room for the answer. A larger budget gives you more space, but does not tell you which information belongs there.

See AI vocabulary in plain English for this term alongside the other language of models, data, agents, and infrastructure.

Try the token-budget exercise below: compare a request whose input fits but leaves no room for output with one that keeps some headroom.

Leave room for the answer

Try an invented 16,000-token window. Instructions, the current request and tool definitions use 2,000 fixed tokens. Change the other allowances and check what remains for output.

These assigned counts teach the arithmetic; they are not estimates or recommended settings for a real model. Output allowance covers the visible answer and any reasoning tokens together. Actual model and API accounting still needs checking.

The worked example below is exactly full. Enable JavaScript to adjust it; the two contrasting examples and explanations also work without JavaScript.

Input
14,000 tokens
Output allowance
2,000 tokens
Planned total
16,000 / 16,000

Exactly full, with no headroom. 14,000 input + 2,000 output allowance = 16,000 tokens.

The bar stops at the window’s capacity; any excess is counted in the text. Nothing here removes tokens or simulates how an API handles overflow. A budget that fits does not establish that the evidence is relevant or the answer correct.

Compare two worked budgets

Input fits; output doesn’t: Over budget by 2,000 tokens. 16,000 input + 2,000 output allowance = 18,000 tokens. The input alone fills the window. It leaves no room for the planned output.

Leave headroom: 2,000 tokens of headroom. 10,000 input + 4,000 output allowance = 14,000 tokens. This request uses a smaller input and reserves more for output. Choosing what to omit still requires checking what the task needs.

Does prompt caching free space in the window?

No. Cached input still occupies context. Caching can change the processing or price of reused input; it does not make that input disappear. Shortening or compacting the input changes what the next request receives and needs a separate check that important information survives.

Primary references: Anthropic’s context-window accounting and OpenAI’s reasoning-token accounting.

Hear it from the guest

“The math is never gonna change. You want to deliver to the context window exactly what the model needs, nothing more as efficiently as possible”
Brian Raymond, Unstructured.io EP 2 · around 13:01 · read the transcript
“So working memory is your context window. It's short term, it's limited.”
Richmond Alake, Oracle EP 53 · around 22:00 · read the transcript
Watch on YouTube

Quotes lightly edited to remove filler words.

Sources

  • Claude Docs: Context windows — Explains input and output token accounting and model-dependent behavior when a request exceeds the window.

Go deeper

From the conversation

Context EngineeringContext PoisoningContext LayerLong Context