Guides / batch processing cost optimization
Batch processing cost optimization
OpenAI, Anthropic and Gemini each bill batched requests at half their standard rates in exchange for a turnaround window rather than an immediate answer. Caching stacks on top of that, though hits inside a batch are best-effort. Culpa, a local-first LLM cost, margin, and forecast ledger, measures which of your traffic could wait and what the cached share actually was.
Why this happens
Half off is the largest single discount any of these vendors publishes and it asks for one thing, which is patience. OpenAI quotes a 24-hour turnaround and expires anything that misses it. Anthropic reports most batches finishing inside an hour. So the eligibility test isn't whether a job is asynchronous, it's whether a result that arrives tomorrow is still worth having, and whether the work can be resubmitted if a batch expires. The second half of the problem is that caching and batching interact. Anthropic documents that the two discounts can stack, that cache hits inside a batch are provided on a best-effort basis, and that batches routinely run past five minutes, so batch traffic wants the one-hour cache. A cache that misses still charges the write, which is how a second discount turns into a surcharge.
What this usually looks like
- Evals, backfills and report generation run on the synchronous API out of habit.
- Nobody has listed which of your jobs would still be useful tomorrow.
- Batch was switched on with caching and the saving came in below half.
- Cache write volume on batch traffic is high and the cached-read share is low.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Mistakes that cost the most
| Mistake | Why it hurts | Do instead |
|---|---|---|
| Treating batch eligibility as a question about async code. | Plenty of async work still needs an answer in seconds, and plenty of sync work doesn't. | Ask whether tomorrow's answer is still worth having. That's the actual test. |
| Assuming caching and batching simply add up. | Hits inside a batch are best-effort, and a missed cache still bills the write premium. | Use the one-hour cache for batch traffic and measure the cached share before trusting it. |
| Ignoring what an expired batch costs. | OpenAI expires batches that miss the 24-hour window, so the work has to be run again. | Size batches to finish well inside the window and treat a re-run as part of the price. |
Run this check tonight
- List every job whose result would still be useful in 24 hours.
- Total their spend. Half of that number is the ceiling on what batching can save.
- Check whether your batch traffic uses the one-hour cache rather than the five-minute one.
- Measure the cached-read share on batch calls and compare it against the write count.
- Confirm no batch is sized close enough to the window to risk expiring.
One eval job, four prices, and a second discount that can invert
Illustrative example
A monthly eval job on Claude Sonnet 5 at real rates of $2.00 per million input and $10.00 output, effective 2026-07-02. It sends 200M input tokens, of which 150M is a repeated prefix, and 20M output. Anthropic charges all batch usage at 50% of standard. Volumes and the cache hit outcome are modelled.
Batching halves the bill and holds. Adding caching on top ranges from cutting it by another 42% to pushing it 50% back above plain batch, decided by hit behaviour the vendor describes as best-effort. Measuring the cached share is the only way to know which happened.
Every number, with its confidence and source
| Figure | What it means | Confidence | Source |
|---|---|---|---|
| 50% | published batch discount against standard rates at OpenAI, Anthropic and Gemini | provider-reported | Anthropic states all batch usage is charged at 50% of standard API prices. OpenAI's pricing table carries a batch row at exactly half of each standard row and its Batch API guide quotes 50% lower costs with a 24-hour turnaround. Gemini lists a 50% cost reduction on its paid tier. All checked 2026-08-02. |
| $174.50 to $600 | modelled monthly cost of one eval job, standard rates down to batch with cache hits landing | estimated | Both endpoints from the teardown arithmetic at real Claude Sonnet 5 rates from the price book, effective 2026-07-02. A range because the volumes and the cache hit outcome are modelled. |
What a generic answer can’t know
Which of your jobs can wait is a product judgement no vendor can make, and what share of your batched calls hit cache is a measurement no invoice reports. Both live in your own traffic. Culpa measures them on your infrastructure, keeps your prompts and responses there, and counts the calls to run your plan.
Questions founders ask next
How much does batch processing save?
Half. OpenAI, Anthropic and Gemini all publish batch usage at 50% of standard rates, checked 2026-08-02. OpenAI's rate card carries a batch column at exactly half of each standard row, and Anthropic states that all batch usage is charged at 50% of standard API prices.
Do prompt caching and batching stack?
Anthropic documents that the discounts can stack, with the caveat that cache hits inside a batch are best-effort because requests are processed asynchronously and concurrently. A missed hit still bills the write, so the second discount can't be assumed and can cost you.
Which cache duration should batch traffic use?
The one-hour one. Anthropic's own guidance is that batches routinely take longer than five minutes to process, so the shorter entry expires before the rest of the batch reaches it. That also raises the break-even from a second call to a third.
On your infrastructure
Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.
How Culpa works
Find the culprit. Not just the total.
Your dashboard shows what you spent. It stops short of who spent it. Culpa shows the conversation, the user and the feature behind it.
Your prompts stay local.
Culpa runs on your own infrastructure. What you send to a model reaches us at no point.
Every dollar has a name.
Follow any charge to the conversation, the user, the feature and the customer behind it.
See the bill before it lands.
Cost your next feature before you ship it. You get the likely bill and the worst case, at best, median, p90 and p99.
Three steps to your first answer.
Change one base URL.
Or drop in the Python or TypeScript library.
Find your most expensive conversation.
In the first session, not the first week.
Cost your next feature before you ship it.
Why the bill went up
Example dashboardCalls traced
418,209
across 3 projects
Spend this week
$378.41
+ $182 vs last week
Failed calls
312
74% retried, and you paid for all of them
+ $182 this week traced to one culprit
Spend over 14 days
Most expensive users
Next week forecast
Graded against reality. Accuracy shown as results land.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Keep reading
Sources: Anthropic Message Batches, OpenAI Batch API guide, Anthropic pricing. Last reviewed 2026-08-02. Plain text version.