Guides / llm eval job cost
LLM eval job cost, the spend with no customer attached
Evaluation runs bill at production rates while serving no customer, and they scale with test cases times prompt variants times a judge call. That product grows fast. Culpa, a local-first LLM cost, margin, and forecast ledger, tags eval traffic as its own feature, so it stops hiding inside your unattributed spend.
Why this happens
Eval spend hides for a structural reason. It carries no user identifier and no customer, so any attribution keyed on those drops it into an unlabelled bucket. Meanwhile the arithmetic is multiplicative. A hundred test cases across four prompt variants is four hundred calls, and if each is graded by a model judge that's eight hundred. Run it on every commit and the eval suite can outspend the product it tests.
What this usually looks like
- A large share of your spend carries no user or customer attribution at all.
- Cost spikes on weekday afternoons and vanishes at weekends, which tracks engineering activity rather than users.
- Spend correlates with your commit frequency more closely than with your traffic.
- Nobody can state what a single full eval run costs, because nobody has ever priced one.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Mistakes that cost the most
| Mistake | Why it hurts | Do instead |
|---|---|---|
| Running the full eval suite on every commit. | Cost scales with commit frequency, which is the one number engineering is actively trying to increase. | Run a fast subset per commit and the full suite on merge or nightly, and price both. |
| Grading with a frontier model by default. | The judge often costs more than the thing being judged, and doubles every case in the suite. | Grade with the cheapest model that agrees with your human labels, and check that agreement periodically. |
| Leaving eval traffic untagged. | It lands in unattributed spend, so it never appears in any breakdown anyone reviews. | Tag eval runs as their own feature, with the suite name, so the line item is visible and ownable. |
Run this check tonight
- Price one full eval run: cases times variants times calls per case, times your cost per call.
- Multiply that by how often the suite runs in a month. Compare the result to your production spend.
- Check what share of your spend has no user identifier attached. Eval traffic usually lives there.
- Compare your judge model's cost to the model under test. Often the judge is the larger half.
- Look at how many test cases have never changed a decision. Those are pure cost with no signal.
One eval suite, priced honestly
Illustrative example
200 test cases across 3 prompt variants, each graded by a model judge. The model under test is GPT-5.4 at $0.0025 and $0.015 per 1k. The judge is GPT-5.5 at $0.005 and $0.030 per 1k. Each test call is 1,500 input and 400 output tokens, and each judge call is 2,000 input and 150 output.
$582 a month for a suite nobody has on a dashboard, and the grader costs more than the model being graded. Moving the judge to a cheaper model is the single biggest lever here.
Every number, with its confidence and source
| Figure | What it means | Confidence | Source |
|---|---|---|---|
| $14.55 | modelled cost of one full eval run across 200 cases and 3 variants with a model judge | calculated | 600 test calls at $0.00975 plus 600 judge calls at $0.0145, using real GPT-5.4 and GPT-5.5 rates per 1k tokens from the price book, effective 2026-07-02. Case counts and token sizes are modelled. |
| 60% | share of the modelled eval run cost taken by the judge rather than the model under test | calculated | $8.70 judge subtotal divided by the $14.55 run total, from the teardown arithmetic. |
What a generic answer can’t know
A provider can't distinguish an eval call from a production one. Both are requests from your key. Separating them needs a tag applied when the call is made, which means the decision has to happen in your own stack. Culpa captures it on your infrastructure, keeps your prompts and responses there, and counts the calls to run your plan.
Questions founders ask next
Why does eval spend never show up in cost reviews?
Because it carries no user or customer identifier, so any breakdown keyed on those puts it in an unlabelled bucket. It isn't hidden deliberately, it simply has no owner in the attribution scheme, and unowned spend doesn't get reviewed.
Is a model judge worth its cost?
Only if it agrees with your human labels. Check that agreement on a sample before assuming a frontier judge is necessary, because a cheaper judge that agrees just as often halves the largest line in most eval suites.
How often should a full eval suite run?
Often enough to catch a regression before it ships, rarely enough that cost tracks releases rather than commits. A fast subset per commit plus the full suite on merge is the usual balance, and both should be priced.
On your infrastructure
Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Keep reading
Sources: OpenAI API pricing. Last reviewed 2026-08-01. Plain text version.