# LLM eval job cost, the spend with no customer attached > Evaluation runs bill like production but serve no customer. Why eval spend hides, how to price a run, and how to cut it without losing coverage. URL: https://getculpa.com/llm-eval-job-cost Last reviewed: 2026-08-01 ## Answer Evaluation runs bill at production rates while serving no customer, and they scale with test cases times prompt variants times a judge call. That product grows fast. Culpa, a local-first LLM cost, margin, and forecast ledger, tags eval traffic as its own feature, so it stops hiding inside your unattributed spend. ## Why this happens Eval spend hides for a structural reason. It carries no user identifier and no customer, so any attribution keyed on those drops it into an unlabelled bucket. Meanwhile the arithmetic is multiplicative. A hundred test cases across four prompt variants is four hundred calls, and if each is graded by a model judge that's eight hundred. Run it on every commit and the eval suite can outspend the product it tests. ## What this usually looks like - A large share of your spend carries no user or customer attribution at all. - Cost spikes on weekday afternoons and vanishes at weekends, which tracks engineering activity rather than users. - Spend correlates with your commit frequency more closely than with your traffic. - Nobody can state what a single full eval run costs, because nobody has ever priced one. ## Common mistakes - Running the full eval suite on every commit. Why it hurts: Cost scales with commit frequency, which is the one number engineering is actively trying to increase. Do instead: Run a fast subset per commit and the full suite on merge or nightly, and price both. - Grading with a frontier model by default. Why it hurts: The judge often costs more than the thing being judged, and doubles every case in the suite. Do instead: Grade with the cheapest model that agrees with your human labels, and check that agreement periodically. - Leaving eval traffic untagged. Why it hurts: It lands in unattributed spend, so it never appears in any breakdown anyone reviews. Do instead: Tag eval runs as their own feature, with the suite name, so the line item is visible and ownable. ## Self-check - Price one full eval run: cases times variants times calls per case, times your cost per call. - Multiply that by how often the suite runs in a month. Compare the result to your production spend. - Check what share of your spend has no user identifier attached. Eval traffic usually lives there. - Compare your judge model's cost to the model under test. Often the judge is the larger half. - Look at how many test cases have never changed a decision. Those are pure cost with no signal. ## One eval suite, priced honestly (illustrative) 200 test cases across 3 prompt variants, each graded by a model judge. The model under test is GPT-5.4 at $0.0025 and $0.015 per 1k. The judge is GPT-5.5 at $0.005 and $0.030 per 1k. Each test call is 1,500 input and 400 output tokens, and each judge call is 2,000 input and 150 output. Test calls: 200 x 3 = 600. Each = (1.5 x $0.0025) + (0.4 x $0.015) = $0.00375 + $0.006 = $0.00975 Test subtotal = 600 x $0.00975 = $5.85 Judge calls: 600. Each = (2 x $0.005) + (0.15 x $0.030) = $0.010 + $0.0045 = $0.0145 Judge subtotal = 600 x $0.0145 = $8.70, which is more than the thing it grades One run = $14.55. On 40 merges a month = $582, and the judge is 60% of it $582 a month for a suite nobody has on a dashboard, and the grader costs more than the model being graded. Moving the judge to a cheaper model is the single biggest lever here. ## Cost figures Every figure carries its confidence and its source. No figure on this site is provider-reported. - $14.55 — modelled cost of one full eval run across 200 cases and 3 variants with a model judge [calculated] Source: 600 test calls at $0.00975 plus 600 judge calls at $0.0145, using real GPT-5.4 and GPT-5.5 rates per 1k tokens from the price book, effective 2026-07-02. Case counts and token sizes are modelled. - 60% — share of the modelled eval run cost taken by the judge rather than the model under test [calculated] Source: $8.70 judge subtotal divided by the $14.55 run total, from the teardown arithmetic. ## FAQ Q: Why does eval spend never show up in cost reviews? A: Because it carries no user or customer identifier, so any breakdown keyed on those puts it in an unlabelled bucket. It isn't hidden deliberately, it simply has no owner in the attribution scheme, and unowned spend doesn't get reviewed. Q: Is a model judge worth its cost? A: Only if it agrees with your human labels. Check that agreement on a sample before assuming a frontier judge is necessary, because a cheaper judge that agrees just as often halves the largest line in most eval suites. Q: How often should a full eval suite run? A: Often enough to catch a regression before it ships, rarely enough that cost tracks releases rather than commits. A fast subset per commit plus the full suite on merge is the usual balance, and both should be priced. ## Sources - OpenAI API pricing: https://developers.openai.com/api/docs/pricing Run the free Cost Leak Scan: https://app.getculpa.com/scan?source=pseo&slug=llm-eval-job-cost&cluster=problem Machine-readable index of every guide: https://getculpa.com/api/pages Human-readable index of every guide: https://getculpa.com/guides Site overview: https://app.getculpa.com/llms.txt Privacy: Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.