Published list prices are the most quoted number in AI, and the least useful. When a team sits down to do an honest AI inference pricing comparison, the sticker on the provider's pricing page rarely matches the invoice at the end of the month. The gap between the two is where most of the money hides — and understanding it is the difference between a budget you can defend and one that quietly doubles.
## List prices are the sticker, not the street
A per-million-token figure on a landing page assumes a tidy world: steady traffic, one model, no retries, no spikes. Real workloads are lumpy. You pay for peak provisioning you use for minutes a day, for retries on rate limits, and for the largest model in the catalogue because switching later means re-integrating. The cost per token AI teams actually pay is a blend of the rate card and all the operational friction around it. The rate card is the floor, not the number.
## Where the money actually goes
The single biggest hidden cost is idle capacity. Independent measurements keep landing in the same uncomfortable range: Cast AI has reported average GPU utilisation around 5%, and ClearML has put the figure near 7%. Whichever number you trust, the story is the same — the accelerators you are paying for sit idle the overwhelming majority of the time, and that waste is priced into what you are charged.
Three structural choices drive it. First, single-buyer routing: capacity reserved for one workload cannot be shared, so troughs are dead weight. Second, over-provisioned latency tiers — paying for the fastest path on requests that never needed it. Third, model over-selection, where every call goes to a frontier model regardless of whether a smaller one would have answered identically. None of these show up on the pricing page. All of them show up on the bill.
## What an orchestration layer changes
An orchestration layer sits between your application and the model providers and attacks exactly those inefficiencies. It classifies each request by cost ceiling and latency tolerance, then routes to the cheapest model that satisfies the constraint rather than the default. It batches compatible requests to lift utilisation off that 5–7% floor, and it reuses prefix caches so repeated context is not paid for twice.
The point of recovering that efficiency is to pass it through rather than keep it as margin. On Graphene the recovered headroom becomes a flat discount: every routed token is billed at 20% under each provider's published list price, and the rate applied to a request is shown next to its token counts so the discount is verifiable per call rather than promised in the abstract. It is a pass-through, not a volume tier you have to negotiate into.
## How to evaluate any inference provider
You do not have to take any vendor's word for this, Graphene included. Four tests separate a real cost story from a marketing one.
**Portability.** Can you move on and off with a one-line base-URL change, or is there an SDK to rewrite? If leaving is expensive, the quoted price is not the real price.
**Routed-rate transparency.** Does every request expose the model it hit and the exact rate charged? If the rate is only visible in aggregate at month-end, you cannot audit it.
**Budget enforcement.** Can you cap spend per workspace at the API layer, so a runaway job is held rather than billed? Alerts after the fact are not enforcement.
**Residency options.** If your workload is regulated, can routing be constrained to a region and can that constraint be verified from an audit trail?
Run those four checks against any provider — the answers tell you far more than the per-million-token figure ever will.
## The bottom line
List prices describe a world without idle GPUs, retries, or over-provisioning. Your invoice describes the real one. The way to close the gap is not a sharper spreadsheet but an architecture that recovers the waste and passes it through — with the routed rate and the budget both visible while it happens.
If you want to see the numbers for your own traffic, the [pricing page](/pricing) shows the flat 20%-under-list model, and the [Quickstart in the docs](/docs) walks through the one-line migration. It is the same base-URL change to leave, so there is nothing to lock in by trying it.