The routing cost nobody puts on the invoice: losing your prompt cache

Most writing about LLM routing treats "cost per token" as the whole cost story: model A charges more than model B, route to B when you can, report the difference as savings. That's true as far as it goes, but it skips a mechanism that can quietly erase a chunk of those savings, or turn them negative, and it's specific enough that it deserves its own explanation rather than a footnote.

What provider-side prompt caching actually does

Several providers now cache parts of a prompt on their end, most commonly a system prompt or a long, repeated context prefix, so that subsequent calls reusing that same prefix against the same model are billed at a steep discount on the cached portion. The mechanism varies by provider (some cache automatically above a length threshold, others require you to mark cache breakpoints explicitly), but the shape is consistent: send the same prefix to the same model repeatedly, and the provider charges you much less for it after the first call.

This matters enormously for the common pattern of a long, stable system prompt or a large document held in context across many requests. Without caching, you pay full input-token price on that prefix every single call. With caching, you pay it once and then a fraction of that on every subsequent call that reuses it, often for a limited window of time before the cache entry expires.

The failure mode routing introduces

Here's the part that's easy to miss: the cache is keyed to the provider and, in practice, to the specific model you called. If your last ten calls all went to the same model with the same system prompt, they built up a warm cache and you were paying the discounted rate. Route the eleventh call to a different, cheaper model because a router decided that request didn't need your usual model, and you lose the cache hit entirely. That call pays full price on the prefix, on the new model, in addition to whatever the new model itself costs.

Depending on how large that cached prefix is relative to the rest of the request, this can matter a lot. A system prompt or context block that's the bulk of your token volume, cached at a steep discount on the model you normally use, can make "route to a cheaper model this one time" a net loss once you account for the cache miss, even though the cheaper model's sticker price per token is lower. The router did exactly what it was told to do and the bill went the wrong direction anyway.

This is not a hypothetical concern invented for this post. It's a known, documented failure mode, and it has been a real factor in at least one publicized routing project's savings claims not holding up under scrutiny once someone looked past the headline percentage. A savings figure calculated purely from list-price-per-token deltas, with no accounting for cache state, isn't wrong in the sense of being fabricated. It's wrong in the sense of measuring a smaller cost than the one that actually landed on the invoice.

Why this is genuinely hard to fix

The honest reason most routers don't account for this well is that provider-side cache state isn't something the router can see. Whether a given prefix is currently warm in a given provider's cache, and how much of a discount it would earn on the next call, is provider-internal state that isn't exposed through the API in a way a router can query before deciding where to send a request. You can infer likely cache warmth from your own request pattern (same prompt, same model, recently), but that's an estimate of the provider's internal state, not a measurement of it, and it degrades further the moment multiple routing decisions are interacting with the same cache concurrently.

Building a good estimate of this is a legitimate, hard engineering problem, and it's on our list. It is not solved today.

What llm11 does about it right now, which is to say: not much, and we say so

We're not going to pretend this is handled. llm11's receipt does not currently measure or estimate the cost of a lost provider-side cache hit when a request routes to a different model than a preceding one. Rather than silently leaving that cost out of the savings figure and letting the number look better than it is, or worse, guessing at a value with no real basis and presenting it as measured, the receipt reports provider cache effects as unmeasured. The savings figure you see reflects the arithmetic we can actually stand behind: real per-request cost at provider list price, against a real baseline, for the tokens that were actually generated and verified. It does not include a cache-loss adjustment in either direction, because we don't have a trustworthy number for one yet.

If your workload leans heavily on a large, stable system prompt or long context that you'd normally expect to stay warm in a provider's cache, that's worth factoring in yourself when you decide how aggressively to route those specific requests to a different model. A router, any router, that doesn't mention this tradeoff at all isn't being more accurate than we are. It's just not telling you about a cost it isn't measuring either.

This is meant as a credibility post more than a product pitch: routing is a genuinely good idea for a lot of traffic, but "we saved you X%" is only a number worth trusting once you know what it did and didn't count. Prompt cache loss is one of the things most routers, including this one today, don't count. Better to say that plainly than to let a clean-looking percentage imply otherwise.