Skip to content
Adrythm
Book 20 minutes
AI and automation

Context window

AI context window / token limit / context rot

In short

A context window is the maximum amount of text, counted in tokens, that an AI model can work with in one request, including its own reply. Everything counts toward it, from instructions to the conversation so far, and Anthropic notes that accuracy and recall degrade as it fills.

OpenAI defines the context window as the maximum number of tokens that can be used in a single request, counting input, output and reasoning tokens. Anthropic describes it as all the text a model can reference when generating a response, including the response itself. Both describe working memory for one request, separate from what the model was trained on.

Everything in a request uses the space: the system instructions, documents and tool results, and every earlier turn of a conversation that is sent back each time. As a chat grows, each new message carries more of the past with it.

That growth has two costs. OpenAI says that when a conversation is chained, all previous input tokens are billed as input tokens, so later turns cost more. And Anthropic says accuracy and recall degrade as the token count grows, a phenomenon it calls context rot. Its conclusion is that curating what goes into context matters as much as how much room there is.

In practice

A support assistant built on an AI model sends its full chat history back with every reply. By the fortieth message in a long conversation, each request carries every earlier message, costs more, and is more likely to lose track of a detail from the start. Summarizing older turns, or retrieving only the documents a question needs, keeps each request small.

Not the same as

Training data
Training data is what a model learned from before release. The context window is its working memory for the request in front of it.

Why it matters to you

A vendor quoting a huge context window is describing capacity. How well the tool uses that capacity is a separate question. For custom AI features, how the system decides what to send each time drives both the running cost and how reliable the answers stay in long conversations.

What to ask or check

  1. 01What does the system send to the model on each request, and does that grow with the conversation?
  2. 02How are long conversations handled when they approach the model's limit?
  3. 03Does the tool retrieve only the documents a question needs, or send everything?

What people get wrong

That a bigger context window always makes an AI tool better. Anthropic says more context is not automatically better, because accuracy and recall degrade as the token count grows.

Keep reading

Prompt caching

Prompt caching lets an AI provider reuse its work on the opening part of a prompt when the next request starts the same way. It lowers cost and speeds up replies. OpenAI applies it by default, while Anthropic charges extra to write a cache and a tenth of the normal price to read one.

Retrieval augmented generation

Retrieval augmented generation is how an AI answers from your documents without being trained on them. OWASP describes it as combining a pre-trained model with external knowledge sources at answer time. So the documents sit in a store the system reads from, and who can read that store is the question.

Hallucination

A hallucination is AI output that sounds right and is not. OWASP describes the model filling gaps in its training data using statistical patterns, without understanding the content, so the answer can be fluent and unfounded at once. Its own first example is an airline that was successfully sued over its chatbot.

Training data

Training data is what a model learned from. The question owners ask is whether their own data joins it, and there is no single answer: OWASP names three separate stages, pre-training, fine-tuning and embedding. A commitment worth having names the stage and comes in writing.

AI agent

An AI agent is a model that has been granted the ability to take actions, not just produce text. OWASP says the damage one can do comes from three grants: excessive functionality, excessive permissions and excessive autonomy. What it is allowed to do matters more than how good it is.

Prompt injection

Prompt injection is when text an AI system reads becomes an instruction it follows. OWASP notes the text does not have to be visible to a person, only parsed by the model, and that no fool-proof prevention is known. The UK's national cyber security body says the same.

Want this explained against your own numbers?

Twenty minutes, a straight answer, and no follow-up sequence if you decide not to work with us.

Book 20 minutes