AI Cost Glossary

AI pricing loves to wrap simple ideas in jargon. This glossary explains the terms you'll see in the calculators in plain English - and, more usefully, why each one moves your bill up or down.

Token

A token is a chunk of text read or produced by a model. In ordinary English, one token is often around four characters or three quarters of a word, but that is only a rule of thumb. Code, numbers, punctuation, and many non-English languages tokenize differently.

Providers usually quote text-model prices per million tokens. Exact counts come from the model's tokenizer or API usage response.

Input vs output tokens

Input tokens are sent to the model: instructions, the user's message, conversation history, retrieved documents, examples, and tool definitions. Output tokens are generated by the model. They have separate prices, and output is often the more expensive side.

A workload that classifies text may be input-heavy. A report writer may be output-heavy. That difference can change which model is economical.

Context window

The context window is the amount of input and output a model can handle in one request. It is a capacity limit, not a target. Filling a large window on every call can be expensive and may make it harder for the model to focus on the relevant material.

System prompt

The system prompt contains high-level instructions about role, behavior, format, and constraints. It is commonly sent with every request, so a long system prompt becomes a recurring input cost. Stable system prompts are often good candidates for prompt caching.

Prompt caching

Prompt caching lets a provider reuse work for a repeated prompt prefix. Eligible cached-input tokens may be billed at a lower rate than ordinary input. The prefix normally has to stay identical, and provider rules differ on length and retention.

Read the full prompt caching guide before assuming every repeated idea will produce a cache hit.

Batch API

Batch processing accepts work that does not need an immediate answer and completes it asynchronously. Providers may offer lower rates in exchange for a longer completion window. Evaluations, overnight enrichment, back-office processing, and bulk classification are common uses.

Embedding

An embedding is a numeric representation of text, images, or other content. Similar items are placed near one another in vector space, which makes semantic search possible. Embedding a document is usually much cheaper than asking a generation model to write an answer.

Vector database

A vector database stores embeddings and searches for nearby vectors. In a RAG system, it retrieves passages related to the user's question. Cost may be based on storage, queries, replicas, or underlying infrastructure.

RAG (retrieval-augmented generation)

Retrieval-augmented generation searches a knowledge base and sends relevant passages to a generation model as context. RAG can ground an answer in private or current material, but every retrieved passage adds input tokens. The RAG cost calculator separates indexing, retrieval, storage, and generation.

Chunk

A chunk is a section of a source document stored and retrieved as a unit. Smaller chunks can be precise but lose surrounding meaning. Larger chunks preserve context but add more tokens. Good chunking follows the natural structure of the content and is tested on real questions.

Reranker

A reranker reviews an initial set of search results and orders them by relevance to the question. It adds a small processing step, but it can reduce generation cost if it allows the application to send fewer, better passages to the main model.

Agent

An agent uses a model to complete a multi-step task. It may plan, call tools, inspect results, retry, and produce a final answer. One agent task can contain many model calls, which is why cost should be measured per successful task rather than per user message.

Tool call

A tool call asks application code to perform an action such as search, database lookup, calculation, or file read. The tool itself may have a cost, and the agent often makes another model call after reading the result.

Retry rate

Retry rate is the share of calls or steps repeated after a timeout, invalid format, tool error, or unsatisfactory result. Retries increase cost directly. Clear schemas and useful error messages often reduce them.

Failure-adjusted cost

Failure-adjusted cost divides total spend by the number of successful outcomes. If ten tasks cost $1 in total but only eight succeed, the nominal cost is ten cents per attempt while the delivered cost is 12.5 cents per successful task.

Cache hit rate

Cache hit rate is the share of eligible requests or tokens served from cache. A high number is useful only when the cached prefix is large enough to matter. Measure cached tokens and money saved, not just the percentage of requests with any hit.

Latency and time to first token

Latency is how long the user waits for a response. Time to first token measures how quickly streaming begins; total generation time measures when the response finishes. A cheaper model or batch route may be unsuitable if the product requires an immediate answer.

Throughput

Throughput is the amount of work completed over time, often expressed as tokens per second or requests per minute. For self-hosting, throughput depends on hardware, model size, context length, batching, quantization, and concurrency.

Quantization

Quantization stores or computes model weights at lower numerical precision. It can reduce memory use and increase serving speed, sometimes with a quality tradeoff. The effect must be benchmarked on the intended model and workload.

Utilization

Utilization describes how much of paid computing capacity is doing useful work. A rented GPU costs money while idle, so higher steady utilization lowers effective cost per token. Keeping spare capacity for traffic spikes lowers utilization but may be necessary for reliability.

Self-hosting

Self-hosting means operating an open-weight model on infrastructure you control or rent. You pay for capacity, storage, engineering, and idle time rather than a simple per-token API rate. It becomes attractive with steady, high-utilization volume, or with strict requirements for control over data handling. See API vs self-hosted.

Gross margin

Gross margin is the share of revenue left after direct costs of delivering the service. For an AI SaaS product, direct costs may include model usage, hosting, payment fees, and usage-related support. Averages can hide heavy users, so margin should also be reviewed by plan and customer cohort.

Deflection rate

In customer support, deflection rate is the share of contacts resolved without a human agent. A conversation should count as deflected only when the issue is actually resolved. Blocking access to a human is not the same as successful automation.

Safety margin

A safety margin adds room for normal uncertainty such as busier traffic and imperfect averages. It should not hide a known cost. Taxes, platform fees, or required review belong in their own line when known.

Still unsure which term drives your estimate? Open the relevant calculator and change one assumption at a time. The result will show which concepts matter for that particular workload.