Prompt Caching Explained: How It Works and What It Saves

Prompt caching is the highest-leverage cost optimisation available to most LLM applications, and it is frequently left unimplemented because the mechanism is not obvious from the API documentation.

#What it does

When you send the same content at the start of many requests — a system prompt, tool definitions, a document, a style guide — the provider can store the model's internal representation of that content and reuse it rather than reprocessing from scratch.

You pay a small premium to write to the cache, then roughly a tenth of standard input pricing on every subsequent read within the cache lifetime.

#The critical constraint: it is a prefix cache

This is the part that determines whether your implementation works.

Caching matches on an exact prefix — the content from the start of the prompt up to a marked boundary. If a single character differs anywhere before that boundary, the cache misses entirely and you pay full price.

The practical consequence is that anything variable must come after everything cacheable:

✅ Works:
[ system prompt ][ tools ][ documents ] | [ user query ]
                                     cache boundary

❌ Does not work:
[ user query ][ system prompt ][ tools ][ documents ]

The second layout produces a cache miss on every request, because the prompt begins with content that differs every time. This is the single most common implementation error, and it is invisible — the code works, it just never saves anything.

#What to cache

Always worth caching:

• System prompts of any substantial length
• Tool and function definitions
• Few-shot examples
• Style guides, brand voice documents, coding standards
• Reference documents queried repeatedly
• Large stable schemas

Sometimes worth caching:

• Conversation history in long sessions, cached incrementally as the conversation grows
• Retrieved chunks, if your retrieval returns the same documents frequently

Not worth caching:

• Anything below the provider's minimum cacheable length
• Content used once
• Content that changes on every request

#Minimum lengths and lifetimes

Providers set a minimum number of tokens below which caching does not apply — typically around a thousand tokens, varying by provider and model. Short system prompts fall below it and cannot be cached.

Cache entries expire after a period of inactivity, commonly around five minutes, with longer options available at higher write cost. Each read within the window refreshes the timer, so a steadily used cache persists indefinitely while an intermittently used one repeatedly expires and must be rewritten.

This matters for cost modelling. A workload with sustained traffic gets near-perfect cache hit rates. A workload with a request every ten minutes may pay the write premium repeatedly and save nothing — potentially costing more than not caching at all.

#The economics

Consider a RAG application with 18,000 input tokens per request, of which 15,000 are a stable document context, at 5,000 requests a day.

Without caching: 90M input tokens a day at full price.

With caching: 15M tokens at full price, 75M at roughly a tenth. Effective input cost falls by around 70%.

Set the cacheable share in the LLM API cost calculator to model your own numbers — the slider moves total spend more than almost any other input.

#Structuring an application for it

Standardise your prompt template. If every feature builds its system prompt slightly differently, each one has its own cache entry and hit rates suffer. One template with a shared stable prefix is far more efficient.

Put dynamic values last. Timestamps, user IDs, session data, request-specific parameters — all after the cache boundary. A timestamp injected at the top of a system prompt guarantees a permanent 0% hit rate.

Cache incrementally in conversations. As a conversation grows, extend the cache boundary to include completed turns. Each new message then only pays full price for itself.

Monitor hit rates. Providers return cache read and write token counts in the response. Log them. A hit rate below 70% on a workload you expected to cache well indicates a prefix instability worth finding.

#Where it does not help

Highly variable prompts. If every request has genuinely different context, there is nothing to cache.

Very low volume. Below roughly one request every few minutes, entries expire between uses and the write premium may exceed the savings.

Short prompts. Below the minimum cacheable length, caching simply does not apply.

Output-dominated workloads. Caching only affects input. If you send 500 tokens and generate 3,000, input is a small part of your bill and caching will not move it much.

#A quick check

If your application sends a system prompt longer than a thousand tokens, or reuses documents across requests, and you have not implemented caching, you are very likely paying two to four times more than necessary. It is typically a few hours of work.

Frequently asked questions