How to Compare AI Gateways Yourself — Without Trusting Anyone's Comparison Chart
Every gateway comparison chart on the internet is an ad. Here is the checklist and the free calculator to run the comparison yourself: pricing transparency, per-request ledgers, terms, and failover — criteria, not marketing.
Search for "LLM gateway comparison" and you will find neatly designed tables that all conclude the same thing: whichever gateway built the table wins. The tables are ads with columns. This short guide is the opposite: a checklist for running the comparison yourself, on your own workload, with tools you can verify.
Disclosure up front: I operate Teapot Router, an API gateway. The criteria below are ones I built my own service against — they are deliberately hard, and any gateway that passes them is genuinely competitive, including ones that are not mine.
Why most comparison charts are useless
A comparison chart made by vendor X has three structural problems: it picks the axes it wins on, it prices workloads nobody actually runs, and it goes stale the day after publication. Rates change monthly; a static table cannot. If a chart does not link to a machine-readable price list you can check yourself, it is marketing, not comparison.
The five criteria that actually differ
These are the dimensions where gateways genuinely diverge — and where a bad choice costs real money:
- Pricing transparency: is there a public, machine-readable price list (JSON endpoint, no login)? If you cannot script the price check, you cannot audit the invoice.
- Per-request ledger: cost, tokens, and latency per call, exportable. A monthly total is an autopsy; a per-request ledger is a health check.
- Terms in writing: what happens to unused credit (expiry?), how refunds are computed, minimum top-up. "It just sits there forever" should be a policy page, not a vibe.
- Protocol surface: one key covering the protocols your tools actually speak (Anthropic for Claude Code, OpenAI-compatible for the rest) — or separate keys and bills for each.
- Failover behavior: what happens when an upstream degrades — automatic switch, or your pager?
Run the comparison on YOUR workload
Generic benchmarks measure nobody's workload. Yours is a shape: hours per day, task intensity, cache behavior. Our free calculator prices that shape honestly — assumptions printed on the page, rates read live from the public pricing endpoint: https://teapotrouter.com/en/claude-code-cost-calculator
Then do the same arithmetic for any other gateway you are evaluating: their price list times your token shape. If their price list is not public, that itself is your answer.
The one-week trial that beats any table
Before committing: run one real project through the gateway for a week, then open the usage log and ask three questions. Did every line's cost match the published rate? Did the ledger explain every surprise? Would you bet your next incident on this status page? A week of real data outscores a hundred comparison cells.
Comparison charts are ads; criteria are portable. Take the five above, price your own workload shape, and make every gateway — including mine — prove it with a public price list and a per-request ledger.
FAQ
Why do gateway comparison sites all rank the same products?
Most are built by the products they rank — the table is the ad. Trust comparisons that publish their data sources and let you re-run the math on your own workload.
What is the fastest way to compare gateway pricing?
Use a calculator that prices your usage shape — hours per day and task intensity — against each gateway's public per-token rates, cache pricing included. Rates that are not public cannot be compared at all.
What should a gateway's terms page tell me?
Whether unused credit expires, how refunds are calculated, and the minimum top-up. These three terms decide whether a low rate is actually cheap.