Skip to content
Adrythm
Book 20 minutes
AI and automation

Prompt caching

cached input tokens / prompt cache / AI API caching

In short

Prompt caching lets an AI provider reuse its work on the opening part of a prompt when the next request starts the same way. It lowers cost and speeds up replies. OpenAI applies it by default, while Anthropic charges extra to write a cache and a tenth of the normal price to read one.

Many AI features send the same long instructions with every request: a system prompt, a list of tools, a company's policies. That repeated opening is the prefix, and caching works on it. The saving applies only when a new request begins with the same prefix as an earlier one.

OpenAI enables prompt caching by default for supported models. Reused tokens are billed at a reduced cached-input rate, discounted up to 90%. Caches are not shared across organizations.

Anthropic's pricing shows the trade more plainly. Writing to its 5-minute cache costs 1.25 times the base input price, writing to its 1-hour cache costs 2 times, and reading from either costs 0.1 times. Content that is written to the cache and rarely read again costs more than sending it uncached.

Caches also expire. Anthropic offers 5-minute and 1-hour lifetimes. OpenAI says in-memory entries on earlier models typically stay active for around 5 to 10 minutes of inactivity, up to an hour.

In practice

A booking assistant sends 4,000 tokens of instructions and business details before every customer question. Using Anthropic's multipliers, the first request writes that block to the 5-minute cache at 1.25 times the base price. The next nine questions, arriving within minutes of each other, read it at 0.1 times. Across all ten, the instructions cost 2.15 times the base price instead of 10 times. The figures are a worked example.

Why it matters to you

In custom software that runs AI all day, repeated instructions can be a large share of the input bill. Keeping the fixed part of each prompt identical and at the start lets caching cut that cost. A design that reshuffles that part on every request gets no saving, and on Anthropic a cache that is rarely reused adds cost.

What to ask or check

  1. 01Does the system keep its fixed instructions at the start of every prompt, identical each time?
  2. 02Is prompt caching on, and what share of input tokens is read from the cache?
  3. 03Is cached content reused often enough to cover the higher price of writing it?

What people get wrong

That caching always saves money. On Anthropic, writing to the cache costs more than normal input, so a prefix that is rarely reused costs more with caching than without.

Keep reading

Batch API

A batch API lets software send a large group of AI model requests to be processed together within a set window, instead of waiting for each answer. OpenAI and Anthropic both charge half the normal price for batched requests, in exchange for results that can take up to 24 hours.

API

An API is the way one piece of software talks to another without a person in the middle. MDN describes it as a contract between the application offering it and whatever uses it. The part that matters commercially is that the contract belongs to whoever publishes it, and it changes.

AI agent

An AI agent is a model that has been granted the ability to take actions, not just produce text. OWASP says the damage one can do comes from three grants: excessive functionality, excessive permissions and excessive autonomy. What it is allowed to do matters more than how good it is.

Retrieval augmented generation

Retrieval augmented generation is how an AI answers from your documents without being trained on them. OWASP describes it as combining a pre-trained model with external knowledge sources at answer time. So the documents sit in a store the system reads from, and who can read that store is the question.

Training data

Training data is what a model learned from. The question owners ask is whether their own data joins it, and there is no single answer: OWASP names three separate stages, pre-training, fine-tuning and embedding. A commitment worth having names the stage and comes in writing.

Hallucination

A hallucination is AI output that sounds right and is not. OWASP describes the model filling gaps in its training data using statistical patterns, without understanding the content, so the answer can be fluent and unfounded at once. Its own first example is an airline that was successfully sued over its chatbot.

Want this explained against your own numbers?

Twenty minutes, a straight answer, and no follow-up sequence if you decide not to work with us.

Book 20 minutes