Guides / rag context cost
RAG context cost, where retrieval settings become a bill
RAG context cost is the share of your bill created by retrieved chunks rather than by the question a user asked. Top-k multiplied by chunk size sets input tokens per call, so retrieval settings are a pricing decision made in a config file. Culpa, a local-first LLM cost, margin, and forecast ledger, prices each call and maps the input back to the feature that assembled it.
Why this happens
Retrieval parameters get tuned for recall, because recall is what the evaluation measures. Nobody in that loop is looking at what each setting costs, and the relationship is direct: top-k times chunk size is a token count you pay on every single call. Raising top-k from five to twelve more than doubles retrieved context and barely moves most quality metrics, because the last several chunks are the least relevant ones by construction. That's the cheapest saving in a typical stack and it lives in a config file nobody prices.
What this usually looks like
- Retrieved context is the overwhelming majority of your input tokens and nobody has stated the share.
- Top-k was set once during a quality push and has never been revisited.
- A re-ranking pass runs over retrieved chunks and its own cost is untracked.
- Chunk size was chosen for embedding quality with no reference to what it costs at inference.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Mistakes that cost the most
| Mistake | Why it hurts | Do instead |
|---|---|---|
| Tuning top-k for recall alone. | Every extra chunk bills on every call forever, while the marginal chunk contributes least to the answer. | Plot answer quality against top-k and find where the curve flattens. Pay for that point, not beyond it. |
| Retrieving before deciding whether retrieval is needed. | Many queries need no context at all, so you pay full retrieval cost to answer a greeting. | Gate retrieval on a cheap classifier, and measure what share of calls skip it. |
| Ignoring chunk size once embeddings are built. | Chunk size multiplies top-k, so a large chunk quietly doubles your context bill at the same k. | Treat chunk size and top-k as one budget, measured in tokens per call, and set that budget deliberately. |
Run this check tonight
- Multiply top-k by chunk size. That's your retrieval input cost per call, before the user typed anything.
- Divide it by total input tokens per call. Most teams find retrieval is the overwhelming majority.
- Re-run your evaluation at half your current top-k and compare quality honestly.
- Count what share of calls trigger retrieval that produced nothing useful.
- Price the re-ranking pass separately. It's a model call too, and it usually goes uncounted.
Top-k twelve against top-k five, same corpus
Illustrative example
A retrieval feature on Gemini 3.5 Flash at real rates of $0.0015 per 1k input and $0.009 output. Chunks are 800 tokens, the question and system prompt total 700 tokens, and answers run 350 tokens. Call volume is modelled.
Seven chunks that contribute least to the answer cost $21,000 a month. Retrieval settings move more money than prompt wording ever will, and they're tuned by people measuring recall rather than spend.
Every number, with its confidence and source
| Figure | What it means | Confidence | Source |
|---|---|---|---|
| $0.0102 to $0.0186 | modelled cost per call at top-k 5 against top-k 12 on Gemini 3.5 Flash | estimated | Both endpoints at real Gemini 3.5 Flash rates per 1k tokens from the price book, effective 2026-07-02. A range because chunk size and answer length are modelled. |
| 93% | share of input tokens that are retrieved context at top-k 12 with 800-token chunks | calculated | 9,600 retrieved tokens divided by 10,300 total input tokens, from the teardown arithmetic. |
What a generic answer can’t know
A provider sees a long prompt and prices it. Which part was retrieved context, how many chunks were used, and whether the answer needed them are facts about your retrieval code. Culpa records them alongside the token counts on your infrastructure, keeps your prompts and responses there, and counts the calls to run your plan.
Questions founders ask next
How much does RAG add to cost per call?
Top-k multiplied by chunk size, priced at your input rate, on every call that triggers retrieval. On a typical setup that's the large majority of input tokens, which makes retrieval settings the dominant term in your input bill.
Does lowering top-k hurt answer quality?
Less than most teams expect, because the lowest-ranked chunks are the least relevant by construction. Plot quality against top-k on your own evaluation set and find where the curve flattens, then pay for that point rather than for a round number.
Should retrieved context be cached?
Only where it repeats across calls. Retrieved chunks usually differ per query, so caching helps far less here than it does for a stable system prompt, and on Anthropic a cache that rebuilds each call costs more than not caching.
On your infrastructure
Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.
How Culpa works
Find the culprit. Not just the total.
Your dashboard shows what you spent. It stops short of who spent it. Culpa shows the conversation, the user and the feature behind it.
Your prompts stay local.
Culpa runs on your own infrastructure. What you send to a model reaches us at no point.
Every dollar has a name.
Follow any charge to the conversation, the user, the feature and the customer behind it.
See the bill before it lands.
Cost your next feature before you ship it. You get the likely bill and the worst case, at best, median, p90 and p99.
Three steps to your first answer.
Change one base URL.
Or drop in the Python or TypeScript library.
Find your most expensive conversation.
In the first session, not the first week.
Cost your next feature before you ship it.
Why the bill went up
Example dashboardCalls traced
418,209
across 3 projects
Spend this week
$378.41
+ $182 vs last week
Failed calls
312
74% retried, and you paid for all of them
+ $182 this week traced to one culprit
Spend over 14 days
Most expensive users
Next week forecast
Graded against reality. Accuracy shown as results land.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Keep reading
Sources: Gemini API pricing. Last reviewed 2026-08-01. Plain text version.