# RAG context cost, where retrieval settings become a bill > RAG context cost is top-k multiplied by chunk size on every call. Retrieval settings set your input bill more than your prompt does. URL: https://getculpa.com/rag-context-cost Last reviewed: 2026-08-01 ## Answer RAG context cost is the share of your bill created by retrieved chunks rather than by the question a user asked. Top-k multiplied by chunk size sets input tokens per call, so retrieval settings are a pricing decision made in a config file. Culpa, a local-first LLM cost, margin, and forecast ledger, prices each call and maps the input back to the feature that assembled it. ## Why this happens Retrieval parameters get tuned for recall, because recall is what the evaluation measures. Nobody in that loop is looking at what each setting costs, and the relationship is direct: top-k times chunk size is a token count you pay on every single call. Raising top-k from five to twelve more than doubles retrieved context and barely moves most quality metrics, because the last several chunks are the least relevant ones by construction. That's the cheapest saving in a typical stack and it lives in a config file nobody prices. ## What this usually looks like - Retrieved context is the overwhelming majority of your input tokens and nobody has stated the share. - Top-k was set once during a quality push and has never been revisited. - A re-ranking pass runs over retrieved chunks and its own cost is untracked. - Chunk size was chosen for embedding quality with no reference to what it costs at inference. ## Common mistakes - Tuning top-k for recall alone. Why it hurts: Every extra chunk bills on every call forever, while the marginal chunk contributes least to the answer. Do instead: Plot answer quality against top-k and find where the curve flattens. Pay for that point, not beyond it. - Retrieving before deciding whether retrieval is needed. Why it hurts: Many queries need no context at all, so you pay full retrieval cost to answer a greeting. Do instead: Gate retrieval on a cheap classifier, and measure what share of calls skip it. - Ignoring chunk size once embeddings are built. Why it hurts: Chunk size multiplies top-k, so a large chunk quietly doubles your context bill at the same k. Do instead: Treat chunk size and top-k as one budget, measured in tokens per call, and set that budget deliberately. ## Self-check - Multiply top-k by chunk size. That's your retrieval input cost per call, before the user typed anything. - Divide it by total input tokens per call. Most teams find retrieval is the overwhelming majority. - Re-run your evaluation at half your current top-k and compare quality honestly. - Count what share of calls trigger retrieval that produced nothing useful. - Price the re-ranking pass separately. It's a model call too, and it usually goes uncounted. ## Top-k twelve against top-k five, same corpus (illustrative) A retrieval feature on Gemini 3.5 Flash at real rates of $0.0015 per 1k input and $0.009 output. Chunks are 800 tokens, the question and system prompt total 700 tokens, and answers run 350 tokens. Call volume is modelled. Top-k 12: (12 x 800) + 700 = 10,300 input, so (10.3 x $0.0015) + (0.35 x $0.009) = $0.01545 + $0.00315 = $0.0186 Top-k 5: (5 x 800) + 700 = 4,700 input, so (4.7 x $0.0015) + (0.35 x $0.009) = $0.00705 + $0.00315 = $0.0102 Retrieved context is 9,600 of 10,300 input tokens at top-k 12, which is 93% At 2.5 million calls a month: $46,500 against $25,500 The saving is $21,000 a month, from one number in a config file Seven chunks that contribute least to the answer cost $21,000 a month. Retrieval settings move more money than prompt wording ever will, and they're tuned by people measuring recall rather than spend. ## Cost figures Every figure carries its confidence and its source. No figure on this site is provider-reported. - $0.0102 to $0.0186 — modelled cost per call at top-k 5 against top-k 12 on Gemini 3.5 Flash [estimated] Source: Both endpoints at real Gemini 3.5 Flash rates per 1k tokens from the price book, effective 2026-07-02. A range because chunk size and answer length are modelled. - 93% — share of input tokens that are retrieved context at top-k 12 with 800-token chunks [calculated] Source: 9,600 retrieved tokens divided by 10,300 total input tokens, from the teardown arithmetic. ## FAQ Q: How much does RAG add to cost per call? A: Top-k multiplied by chunk size, priced at your input rate, on every call that triggers retrieval. On a typical setup that's the large majority of input tokens, which makes retrieval settings the dominant term in your input bill. Q: Does lowering top-k hurt answer quality? A: Less than most teams expect, because the lowest-ranked chunks are the least relevant by construction. Plot quality against top-k on your own evaluation set and find where the curve flattens, then pay for that point rather than for a round number. Q: Should retrieved context be cached? A: Only where it repeats across calls. Retrieved chunks usually differ per query, so caching helps far less here than it does for a stable system prompt, and on Anthropic a cache that rebuilds each call costs more than not caching. ## Sources - Gemini API pricing: https://ai.google.dev/gemini-api/docs/pricing Run the free Cost Leak Scan: https://app.getculpa.com/scan?source=pseo&slug=rag-context-cost&cluster=problem Machine-readable index of every guide: https://getculpa.com/api/pages Human-readable index of every guide: https://getculpa.com/guides Site overview: https://app.getculpa.com/llms.txt Privacy: Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.