Prompt Caching: How It Works, and How to Design Prompts for It
Prompt caching lets a model provider store the processed form of the beginning of a prompt and reuse it on later calls that start with exactly the same tokens. Cached tokens are billed at a steep discount and processed faster. To benefit, put stable content such as instructions, tool definitions and reference material first, and variable content such as the user's message last, because any change early in the prompt breaks the cache for everything after it.
Most production prompts repeat themselves. The same system instructions, the same tool definitions, the same reference documents, sent again on every call with a different question at the end. Without caching you pay full price to process that identical prefix every time. Prompt caching lets the provider remember it.
How prompt caching works
When a model reads a prompt, it turns the tokens into internal state before generating anything. Prompt caching stores that state for the start of the prompt. If the next call begins with exactly the same tokens, the provider skips the work for that part and bills it at a discount.
The details differ by provider. At the time of writing, Anthropic's prompt caching uses explicit cache breakpoints, bills cache reads at a small fraction of the normal input price, charges somewhat more than normal to write to the cache, and keeps entries for a few minutes by default with a longer option. OpenAI applies caching automatically to long prompts and discounts the cached part. Check current pricing pages before you plan around exact numbers, but the shape is the same everywhere: identical prefix, cheaper and faster call.
What breaks the cache
The cache matches on an exact prefix. Change one token near the start and everything after it is a miss. Common cache breakers:
- A timestamp or the current date at the top of the system prompt.
- The user's name or account ID inserted into the instructions.
- Tool definitions that are generated in a different order each time.
- Retrieved documents placed before the stable instructions.
- Small edits to the system prompt between deployments, which is expected but worth knowing.
How to design prompts for caching
- Order by stability. Most stable first: system instructions, then tool definitions, then long-lived reference material, then session memory, then the current request last.
- Move variables to the end. Dates, names and IDs go in the user message, not the system prompt.
- Freeze the tool list. Define tools in a fixed order from a fixed source.
- Keep a stable core. In the page builder I run, a short core prompt is always loaded and never changes between calls, which makes it the ideal cached prefix. Knowledge entries that vary per request come after it.
- Measure hits. Providers report cached token counts in the response. Log them per step. A hit rate near zero means something early in the prompt is changing.
Caching is a design decision
Prompt caching rewards the same discipline that good context engineering asks for: a small stable core, variable content kept separate and late, and nothing in the prompt that does not need to be there. If a system is designed that way, caching is close to free money. If it is not, caching barely helps, because every prompt is different from the first line.
It is one of the most direct examples of a classic idea, cache what repeats, carrying straight over into AI system design.
A worked example: what caching saves
Suppose an agent's prompt is 8,000 tokens of stable instructions, tool definitions and reference material, followed by 1,000 tokens of variable content, and it is called 10,000 times a day. Without caching, all 90 million input tokens are billed at full price. With a well-designed prefix, 80 million of them can be billed as cache reads, which on providers that discount cache reads heavily cuts the input bill for that agent by most of its value. The exact saving depends on the provider's current rates and how often the cache expires between calls, so measure it rather than assume it.
A prompt layout that caches well
| Order | Block | Changes |
|---|---|---|
| 1 | System instructions | Only when you deploy |
| 2 | Tool definitions | Only when you deploy |
| 3 | Reference material shared by all users | Rarely |
| 4 | Account or brand context | Per account |
| 5 | Conversation summary | Per session |
| 6 | Current request | Every call |
Blocks one to three form the shared cached prefix. Block four can be cached per account if the same account makes several calls in a short time.
Caching and multi-agent systems
Small agents with stable prompts cache better than one large agent whose prompt keeps changing. That is one more reason the designs in agentic AI system design cost less in practice: each agent's prefix is short, stable and called often.
How I use it
In the page builder on my systems page, the always-loaded core prompt never changes between calls, so it is the cached prefix, and knowledge fetched per step comes after it. It is a small design choice with a large effect on the bill.
Prompt caching across providers
| Question | What to check in your provider's docs |
|---|---|
| Is caching automatic or explicit? | Some providers cache automatically, others need cache markers in the request. |
| What is the minimum prompt length? | Short prompts are often not cached at all. |
| How long do entries live? | Default lifetime and any longer paid option. |
| How are writes and reads billed? | Reads are discounted; writes may cost extra. |
| How are hits reported? | The usage fields that show cached tokens per call. |
The same applies when you go through gateways, managed cloud platforms or libraries that wrap several providers: confirm the caching behaviour passes through, and log cached tokens to prove it.
Prompt caching in coding agents
Coding agents re-send large amounts of context on every turn: instructions, tool definitions, files already read. They rely heavily on caching to stay affordable, which is why changing system instructions or tool lists mid-session can make a session suddenly slower and more expensive.
Frequently asked questions
What is prompt caching?
Prompt caching is a feature of model APIs that stores the processed beginning of a prompt and reuses it when later calls start with exactly the same tokens. The cached part is billed at a discount and processed faster.
How does prompt caching work?
The provider saves the model's internal state for the prompt prefix. On the next call, if the prompt starts with the identical tokens, that work is skipped. Any change early in the prompt breaks the match for everything after it.
How do I get more cache hits?
Put stable content first and variable content last, keep dates, names and IDs out of the system prompt, define tools in a fixed order, and log the cached token count each call returns so you can see when the prefix changes.
How long does a prompt cache last?
It depends on the provider. Cache entries typically last a few minutes after the last use, and some providers offer a longer paid option. Frequent calls keep the cache warm.
Does prompt caching change the model's answers?
No. Caching only skips repeated processing of an identical prefix. The model sees the same prompt and produces output the same way.
Is prompt caching automatic?
It depends on the provider. Some cache long prompts automatically, while others require you to mark where the cacheable prefix ends. Either way, the prefix must be identical between calls to get a hit.
AI developer. I design and build AI agents, AI automation, CRM architecture and the web apps that tie them together. Five of my systems run in production, and one has been acquired. About me · My systems · LinkedIn · GitHub
Want an AI system designed this way?
I design and build AI systems where each agent has one job, sees only what it needs, and hands clean context to the next. Five are running in production right now. See what I have shipped, or book a call and bring the process you want automated.