Your Prompt Cache Hit Rate Collapses Exactly When You Need It

The beautiful cost model that lied

Someone built it in a spreadsheet, and it showed the new feature costing about a tenth of the naive estimate, because the system prompt and the retrieved documents were identical across requests and the provider gives 90% off cached input. Cell B14 said the margin was healthy. The feature shipped, and for three weeks the spreadsheet was right.

Then a campaign drove a traffic spike, and cost per request did not fall with scale the way the model promised, it rose. The cache hit rate, which had hovered around 85% in the calm, fell to single digits during the busiest hour of the busiest day, the hour the spreadsheet had quietly assumed was the cheapest. The model did not have a bug. It had an assumption: that cache hits are free and reliable. They are neither.

What prompt caching actually charges

Start with the mechanics, because the pricing structure is where the danger lives. With Anthropic, the cache is not free in either direction. A cache write costs 1.25x the base input rate for the default five-minute cache and 2x for the extended one-hour cache; a cache read costs 0.1x, the famous 90% off. For Sonnet at a $3/MTok base, that is $3.75/MTok to write the 5-minute cache, $6.00/MTok for the 1-hour, and $0.30/MTok to read (Anthropic prompt caching docs). The TTL refreshes on each read, but the floor is short.

OpenAI does it differently: caching is automatic for prompts over 1,024 tokens, there is no write premium, and cached input is discounted, the published figure is a 50% reduction on the cached portion, up to ~80% lower latency, applied with no code change (OpenAI: Prompt Caching in the API). But it is best-effort: routed by prefix, not guaranteed, and entries clear after a few minutes of inactivity. Two providers, two pricing shapes, one shared truth, the discount only exists when a read lands on a live cache entry. Every part of that sentence is a place the model can break.

The write premium is a tax you pay to lose

Here is the trap built into Anthropic-style caching that the spreadsheet rarely models. A write costs 1.25x; a read saves 0.9. A single read after a write nets you ahead, easy, if the read happens. But under a workload where each entry is written and then expires before a second request arrives, you pay the 1.25x write premium and never collect the 0.1x read. You have built a system that is more expensive than no caching at all, on purpose, in pursuit of savings.

Why the hit rate collapses under load

You would think more traffic means more hits, more requests sharing a prefix, more reads per write. Sometimes. But several forces push the other way, and each strengthens under load.

TTL versus arrival rate. A five-minute TTL means an entry survives only if a matching request arrives within five minutes of the last. Steady moderate load is fine, but traffic arrives in bursts separated by quiet gaps; if your inter-arrival time for a prefix exceeds the TTL even occasionally, those entries expire and the next request re-pays the write. And the floor moved: in March 2026 Anthropic silently changed the default cache TTL from one hour to five minutes, so workloads tuned for the longer window saw quota and cost inflation overnight until they explicitly set the 1-hour duration (claude-code issue #46829).

Prefix fragmentation. Caching keys on an exact prefix match. The instant anything varies near the front, a per-user name, a timestamp, a session ID, a reordered tool list, a different document order from your retriever, you have a different prefix and a guaranteed miss. Under load you often have more variation, not less: more users, more sessions, more personalization, more retrieval diversity. Every variation is a fresh write at a premium.

Provider-side eviction and routing. Best-effort caches route requests to backends and only hit if you land on one holding your entry. Under heavy load the provider spreads you across more capacity, lowering the odds any single backend has your prefix warm. The cache becomes least reliable precisely when the fleet is busiest, when your bill is largest.

Cold starts. Every deploy that changes your system prompt invalidates every cached prefix at once. Every autoscale event starts workers cold. A spike that triggers a scale-up arrives at the worst moment: maximum volume meeting a freshly cold cache.

The number that actually drives your bill

The hit rate is not a property of your cache. It is a property of your traffic shape interacting with the TTL. The same code, same prompts, same provider will show 85% under one pattern and 10% under another. So your cost model cannot use a single hit-rate constant, it has to use a distribution. The effective input cost is a blend:

effective_input_cost =
    hit_rate          * 0.1  * base
  + (miss_then_write) * 1.25 * base
  + (miss_no_write)   * 1.0  * base

If your real peak hit rate is 15% rather than the 85% you modeled, and the misses pay the write premium, your effective input cost during peak can be several times your steady-state cost, and several times what the spreadsheet promised. The blended cost is highest exactly when volume is highest, so the two multiply. That is how a cost-saving feature produces its largest bill on its busiest day. Real reports run both directions: teams cutting LLM spend 59% with disciplined caching, and teams whose savings evaporated when TTL and prefix discipline slipped (ProjectDiscovery: cutting LLM costs 59% with prompt caching).

The fix: design for the cache, then measure the real hit rate

Stabilize prefixes ruthlessly. Everything cacheable goes at the front, byte-for-byte identical: system prompt, tool definitions, few-shot examples, static policy. Everything variable goes at the back. Pin the order of retrieved documents so the same set yields the same prefix. Strip incidental variation, a stray timestamp in a system prompt has been the difference between an 80% and a 5% hit rate in systems we have seen.

Choose TTL from your real inter-arrival time. If requests sharing a prefix routinely arrive more than five minutes apart, the default betrays you. The one-hour cache costs 2x to write but survives the gaps; whether it pays off is arithmetic against your actual traffic. Measure the distribution of inter-arrival times per prefix, its tail is your miss rate. And after March 2026, set the TTL explicitly; do not inherit the default.

Keep hot prefixes warm deliberately. For a small number of extremely common prefixes, a cheap synthetic heartbeat every few minutes keeps the entry alive across quiet gaps so the next burst hits warm. This only makes sense for genuinely hot prefixes; warming a long tail just burns money.

Model the blend, not the best case. Replace the single hit-rate cell with peak and trough scenarios and ask the only question that matters: what does this cost on the worst hour of the worst day, hit rate at its floor and volume at its ceiling? If the answer is still profitable, ship. If it is underwater, you have not built a cost optimization, you have built a bet on a number you do not control.

Test the cache cold, not warm

Every team that gets burned tested it warm. They ran the same prompt twice in a dev console, watched the second call come back cheap, saw the cache-read tokens in the usage report, and concluded caching works. It does, warm, single-tenant, no concurrency, no TTL pressure. None of which describes production.

The hit rate is a load-dependent property, so it can only be validated under load. The test that matters replays realistic traffic, real arrival timing, real prefix diversity, real concurrency, real bursts with quiet gaps, and measures the hit rate that results, not the one you can produce by hand. Build the assertion into your load tests: drive production-shaped traffic and assert that effective cost per request stays under your margin floor at peak. Assert the hit rate during a burst, not during a warm-up. Re-run after every prompt change, because a one-line edit to a system prompt invalidates every cache entry on earth and can quietly halve your hit rate without changing a single correctness test.

# Assert the blended cost under burst, not the warm single-call cost
def assert_cache_economics(samples, base_rate, margin_floor):
    reads  = sum(s.cache_read_tokens     for s in samples)
    writes = sum(s.cache_write_tokens    for s in samples)
    fresh  = sum(s.uncached_input_tokens for s in samples)
    cost = (reads*0.1 + writes*1.25 + fresh*1.0) * base_rate / 1e6
    hit  = reads / max(reads + writes + fresh, 1)
    assert cost / len(samples) < margin_floor, \
        f"effective cost ${cost/len(samples):.4f}/req exceeds floor (hit {hit:.0%})"

The discount is a loan, not a gift

Prompt caching is one of the best cost levers in the LLM stack, and the 90% read discount is real and worth chasing. But it is structured like a loan: you pay the write premium up front and collect the discount only if the reads show up before the entry expires. Whether the loan pays off depends entirely on your traffic shape, your prefix discipline, and your TTL, three things the spreadsheet treated as constants and reality treats as variables. Build the model on the floor of your hit rate, not the ceiling. Test it cold and under load. Then the discount is yours. Assume it, and it will desert you on the one day the bill is big enough to matter.