Guides / context window cost optimization
Context window cost optimization
Gemini charges $1.25 per million input on prompts up to 200k tokens and $2.50 above it, and the output rate moves with the prompt too, from $10.00 to $15.00. So one token past the boundary reprices the entire call rather than the excess. Culpa, a local-first LLM cost, margin, and forecast ledger, measures how many of your prompts sit just over it.
Why this happens
Almost every piece of advice about prompt size assumes a slope, where trimming tokens saves money in proportion. Tiered context pricing is a step. Gemini 2.5 Pro and 3.1 Pro both publish two rate cards split at 200k tokens, and crossing that line doubles the input rate on the whole prompt. The part that surprises people is the output column, which also moves, so a long prompt raises the price of tokens the prompt never touched. A retrieval change that adds one percent of context can add most of a bill, and an average prompt size tells you nothing about it, because the cost depends on the share of calls sitting the wrong side of one number.
What this usually looks like
- Spend jumped after a retrieval change that added a small amount of context.
- Your average prompt size looks fine and your bill doesn't.
- Nobody can say what share of your calls exceed 200,000 tokens.
- Output cost rose while your responses stayed the same length.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Mistakes that cost the most
| Mistake | Why it hurts | Do instead |
|---|---|---|
| Treating prompt size as a smooth cost curve. | The boundary is a step, so the last token before it costs far more than the first. | Find the boundary for your model and measure the distribution of prompts around it. |
| Reasoning about average prompt size. | An average sitting under the boundary hides a tail that crosses it on every expensive call. | Work in percentiles. The share of calls above the line is the number that bills you. |
| Assuming only input reprices. | Gemini's output rate is set by prompt size, so long prompts tax your responses as well. | Price both columns at both tiers before deciding a context change is cheap. |
Run this check tonight
- Find whether your model publishes a context tier and what token boundary splits it.
- Plot your prompt sizes as a distribution rather than an average.
- Establish what share of your calls land above the boundary.
- Price a month at both tiers and treat the gap as your exposure to context growth.
- Re-check after every retrieval or memory change, because those move prompt size directly.
One percent more context, ninety-eight percent more cost
Illustrative example
Gemini 2.5 Pro, whose published rates split at 200,000 tokens: $1.25 per million input and $10.00 output at or below the boundary, $2.50 and $15.00 above it, read from Gemini's pricing page 2026-08-02. Culpa's price book carries only the lower tier. 100,000 calls a month each returning 2,000 output tokens. Volumes are modelled.
A single token doubles the input cost of a call, and the output rate rises by half on tokens the prompt never touched. This is the one place where trimming context is worth vastly more than it looks.
Every number, with its confidence and source
| Figure | What it means | Confidence | Source |
|---|---|---|---|
| 98% | modelled monthly cost increase from raising prompts one percent across the 200k boundary | calculated | $53,250 divided by $26,875, from the teardown arithmetic at Gemini 2.5 Pro's two published tiers. |
| $26,875 to $53,250 | modelled monthly cost of the same workload either side of the boundary | estimated | Both endpoints from the teardown arithmetic at rates read off Gemini's pricing page 2026-08-02. The above-200k tier isn't in Culpa's price book. A range because the call volumes are modelled. |
What a generic answer can’t know
The boundary is published and your distribution around it isn't, because that needs every prompt measured rather than averaged. No provider reports what share of your calls crossed the line, and an invoice shows the total either way. Culpa measures the calls on your infrastructure, keeps your prompts and responses there, and counts the calls to run your plan.
Questions founders ask next
Does the higher rate apply to the whole prompt or only the excess?
The whole prompt. Gemini publishes the rate as a price for prompts above 200k tokens rather than as a surcharge on the tokens past it, so a 201,000-token prompt prices all 201,000 at the higher rate.
Which models have a context tier?
On Gemini it's a Pro-model property, checked 2026-08-02: 2.5 Pro and 3.1 Pro split at 200k while 2.5 Flash, 3.5 Flash and 3.1 Flash-Lite publish one rate. OpenAI's GPT-5.6 family publishes short and long context columns without stating the boundary in the table.
Why did my output cost rise when responses stayed the same length?
Because the output rate is selected by prompt size on a tiered model. Crossing the boundary lifts Gemini 2.5 Pro's output from $10.00 to $15.00 per million, so identical responses bill 50% more on the strength of a longer prompt.
On your infrastructure
Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.
How Culpa works
Find the culprit. Not just the total.
Your dashboard shows what you spent. It stops short of who spent it. Culpa shows the conversation, the user and the feature behind it.
Your prompts stay local.
Culpa runs on your own infrastructure. What you send to a model reaches us at no point.
Every dollar has a name.
Follow any charge to the conversation, the user, the feature and the customer behind it.
See the bill before it lands.
Cost your next feature before you ship it. You get the likely bill and the worst case, at best, median, p90 and p99.
Three steps to your first answer.
Change one base URL.
Or drop in the Python or TypeScript library.
Find your most expensive conversation.
In the first session, not the first week.
Cost your next feature before you ship it.
Why the bill went up
Example dashboardCalls traced
418,209
across 3 projects
Spend this week
$378.41
+ $182 vs last week
Failed calls
312
74% retried, and you paid for all of them
+ $182 this week traced to one culprit
Spend over 14 days
Most expensive users
Next week forecast
Graded against reality. Accuracy shown as results land.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Keep reading
Sources: Gemini API pricing, OpenAI API pricing. Last reviewed 2026-08-02. Plain text version.