AI Cost CalculatorThe four things you pay for
RAG spends money in four places, and they're nowhere near equal. Reading your documents in happens once and is cheap, so even a big pile of documents usually costs a few dollars. Searching each incoming question is cheaper still, since questions are short. The database that stores and searches everything stays modest too - often somewhere between free and about $70 a month. And then there's writing the answer, which dwarfs all the rest.
Why such a gap? Searching is roughly a hundred times cheaper per word than writing. Every question feeds some snippets in and gets an answer back, and both of those are charged at the expensive writing rate. Pull in eight big snippets when three would have answered the question, and you've blown up the expensive part for nothing.
How to keep it cheap
Almost all the savings are in two places: what you pull in, and what you write. Pull in fewer, tighter snippets, and add a reranker so the few you keep are the right ones. Ask for shorter answers, and send routine questions to a cheaper model with your standing instructions and popular snippets cached. And only re-read documents that have actually changed, rather than rebuilding everything on a timer. The RAG cost calculator turns your own numbers into a monthly figure.
Reading documents in is a process, not a button
The reading-in itself is cheap. The work around it matters more: pulling text out of files, stripping out duplicate menus and boilerplate, splitting documents into snippets, tagging them, and deciding what happens when a document changes. A sloppy setup can read the same footer in thousands of times and then serve it up for every question.
The fix is to only re-read what changed. Keep a note of what each document looked like last time, so you never pay to read the same thing twice.
Snippet size changes both quality and cost
Tiny snippets are precise but can lose the context needed to answer. Big snippets keep the context but shove extra text into every answer. There's no one right size - a product manual, a legal policy, a chat log and a code repository all split up differently.
Start by splitting on sections that make sense to a human reader, then test on real questions. Look at what got pulled in, not just whether the final answer sounded confident. If the right passage keeps showing up sixth in the list, better ranking will help more than just pulling in ten passages every time.
Tagging can save more than a cheaper search model
Tags like product, language, version or customer narrow the search before it even starts. That makes results more relevant and lets you send fewer snippets to the AI. It also stops an answer mixing up two product versions, or showing one customer another customer's material.
A reranker is worth testing, not assuming
A reranker is an extra step that re-sorts the snippets to put the best ones first. It earns its keep when it lets you send three great snippets instead of eight so-so ones. On a tiny, tidy knowledge base it may add complexity without changing the answer. Test both ways on the same questions and compare cost and quality.
What a real month looks like
Picture a knowledge base that changes a little each week and answers 50,000 questions a month. Reading it all in is paid once. The weekly top-ups are small. The searches are short. So the recurring bill is almost entirely those 50,000 answers, each carrying its instructions and snippets. That's exactly why the first thing to tighten is what you pull in and how long the answers run - not chasing fractions of a cent in the reading-in job.
What to keep an eye on
- how often the right snippet is in the top few results;
- how much text you pull in per question on average;
- how often answers are good enough to use, and how often they make things up;
- cost per answer, and cost per good answer;
- how fresh the documents are, and whether any updates failed.
RAG gets cheaper as it sends less junk and produces more answers people can trust. Cost and quality usually pull in the same direction here.