Prompt Caching in Production

How enterprises reduce LLM API costs

Based on July 2026 production data. Anthropic's silent updates may change actual behaviour.

The Root of the Problem

Many enterprises that have adopted LLM APIs find their monthly bills climbing steadily with no obvious optimisation lever. The problem often sits in an overlooked place: every conversation turn, the system re-sends the exact same system prompt to the model, and every token is billed at full price.

In a production conversational system, system prompts routinely exceed four thousand tokens. Every conversation, every turn, those four thousand tokens are transmitted and billed again. This is not an architecture defect — it is a billing blind spot most teams have not noticed.

Anthropic's Prompt Caching feature addresses exactly this. However, how it actually works diverges from intuition in several critical ways. Mis-step any one of them and the cache silently stops working — the bill stays the same and the developer has no idea.

How Billing Works

Prompt Caching writes a stable token sequence to cache on first transmission. Subsequent turns read from cache instead of re-processing. The billing formula:

Total cost = Cache Read × 0.1 + Cache Write × 1.25 + uncached input + output

Cache Read is priced at 10% of the normal input rate. Cache Write is 125%, a one-time cost. The first turn writes to cache; from the second turn onward, reads begin and cost drops sharply.

From production data: turn two writes 3,157 tokens, turn three reads the same 3,157 tokens — per-turn cost drops approximately 47%. The longer the conversation, the greater the cumulative saving. Production measurements show roughly 54% savings on short conversations and 80–90% on long ones.

Models have minimum token thresholds: Claude Sonnet requires 1,024 tokens, Claude Haiku requires 2,048. A system prompt that is too short will not trigger caching — another common misconception. For a comparison of model API pricing, see 2026 LLM API pricing comparison.

What Belongs in Cache

Cache-appropriate content is stable content — the static portions of a system prompt that do not change between conversations: role definitions, response format instructions, fixed background knowledge. Consistent structure, long-term stability, highest cache hit rate.

Cache-inappropriate content is dynamic content: RAG memory blocks whose retrieval results change with each query, and the final message of a conversation history that advances each turn. This content changes every round — even with a cache marker set, the cache key is different every time, meaning every turn writes and nothing ever reads.

The critical architectural principle: stable content and dynamic content must be separated. If RAG retrieval results are mixed into the system prompt, the system prompt changes every turn and the cache never hits. This is the most common — and hardest to detect — error in production systems.

Failure Modes Encountered in Production

Caching itself is not complex, but several traps are worth knowing in advance.

Dynamic content mixed into the system prompt. As above — the most direct path to a zero hit rate. The fix is an explicit architectural separation of stablePart (with cache_control set) and dynamicPart (passed separately).

Cache marker placed on the last message of the conversation. The cache boundary advances each turn, meaning every turn is a write and nothing is ever a read. The cache marker belongs on the stable system prompt block, not a moving message boundary.

Streaming added without re-validating cache. After a streaming deployment, caching can silently stop working. Without actively tracking cache_creation and cache_read values in responses, three turns all showing cache_creation: 0 is how it gets discovered.

TTL format error. The correct format is an integer in seconds (ttl: 300), not a string (ttl: "1h"). A format error does not return a 400. The API accepts the request, the response looks normal, the cache silently does nothing. This requires deliberately monitoring billing logs to catch.

Anthropic has also updated cache behaviour without formal announcement: in March 2026, TTL was shortened from 1 hour to 5 minutes, reducing hit rates on long conversations; in July 2026, explicit TTL changed from optional to required — unset values silently fail. Neither change was communicated via error codes or documentation updates — they showed up as changed billing numbers. Regularly auditing cache hit data is a basic production monitoring requirement.

Conclusion

Prompt Caching logic is not complicated, but its effectiveness depends entirely on implementation details: whether stable and dynamic content are separated, whether cache markers are positioned correctly, whether TTL format matches the current API specification. Any one of these wrong and caching silently fails — the bill stays unchanged, and nothing in the logs indicates a problem.

For teams already using LLM APIs in production, the first useful step is to track cache_creation_input_tokens and cache_read_input_tokens on every turn. If cache_creation consistently exceeds cache_read, the architecture almost certainly has one of the issues above. Identify it, fix the separation logic, and the savings are usually immediate.