AI Cost CalculatorThere's no single "cheapest" provider
Each provider sells a range of models, from budget to top-end, so the only fair comparison is like-for-like on your own mix of text in and text out. A tool that writes long answers cares most about the price of replies; a search app that feeds in lots of text cares about cheap input and good caching. The same model can win one and lose the other, which is why no single ranking ever settles it.
Roughly: the budget models (GPT-4o mini, 4.1 nano, Claude Haiku, Gemini Flash) all handle sorting, fact-pulling, tagging and simple chat for cents, so pick on quality and speed, not price. The middle tier (GPT-4.1 mini, Claude Sonnet, Gemini Flash) gives noticeably better reasoning for a little more, and suits most real features. The top models (GPT-4o and the o-series, Claude Opus, Gemini Pro) cost the most, especially for replies, so save them for when a better answer clearly pays for itself. Caching and batch can shuffle all of this around - a model with a higher sticker price but strong caching often wins on a context-heavy app. Pop your usage into the API cost calculator and it ranks every model on what it'd actually cost you, using the prices from the data sources page.
Start with a small test set
Gather real examples from the feature you're building - routine ones, long ones, vague instructions, and cases your current setup gets wrong. Decide how you'll judge an answer before you run anything: is it accurate, in the right format, complete, the right tone, fast enough?
Twenty well-chosen examples teach you more than hundreds of made-up ones, and they take the emotion out of it. A familiar brand gets no points for being familiar, and a cheap model gets no points if you have to fix every answer it gives.
Keep the test fair
Use the same instructions and the same source material across all three. Record what each one actually used, rather than assuming they count text the same way. Run each a few times so you spot the wobbly answers and the occasional slow one.
If one provider needs a noticeably different prompt to do well, let it have one. The fair comparison is the best version you'd actually ship - not a rule that every model gets word-for-word identical text.
Compare cost per good answer
Price per million tokens is an ingredient, not the answer. Take your total test cost and divide it by the number of answers that were actually good enough to use, and include the retries. A model that gets it right 95% of the time can easily beat a cheaper one that you have to fix one time in five.
Choosing a provider is about more than price
Look past the answer itself: rate limits, which regions you can use, data controls, support, how predictable the output format is, batch and caching options, and how cleanly it fits your stack. A small price gap may not be worth running a second integration. A real reliability gap might be.
Using two providers can add resilience and bargaining power, but it also doubles the testing and plumbing. When a product is young, start with one - unless a customer requirement or the need for a backup forces your hand.
Come back to the decision later
Models and prices change fast. Keep your test set and re-run it when a promising model shows up, a price moves, or your workload shifts. Don't wire your product to a marketing word like "premium" - choose on what you've measured.
Use the API cost calculator after the quality test, with each model's real usage and your expected volume. Quality plus cost-per-good-answer gives you a decision you can actually explain to someone.