Guides / llm a/b test cost
Costing the arms of an LLM A/B test
An LLM A/B test compares variants on quality, and each variant also carries its own cost per call. Judging on quality alone picks a winner without pricing it. Culpa, a local-first LLM cost, margin, and forecast ledger, prices every call and tags it with the variant, so an experiment reports cost per arm beside the score.
Why this happens
Every arm of a prompt or model test has a unit cost, and those costs can differ by multiples rather than percentages: a longer prompt, a reasoning model, an extra retrieval pass or a self-critique step all buy quality with money. Plenty of tooling shows you this. Braintrust puts estimated cost beside duration and token counts in an experiment, specifically so you can weigh the trade when you switch models, and its logs carry the same instrumentation into production. So the gap isn't coverage, and anyone telling you otherwise hasn't read the docs. It sits in two narrower places. The first is the word estimated, which is the honest label for a figure derived from a rate table nobody versions by date, and rates move: Claude Sonnet 5 leaves introductory pricing on 2026-09-01. The second is that cost alone can't rank arms, because the arm that costs triple may also serve the customers who pay you most. A team ships the variant that won by four points and meets the real number at the end of the month, by which point reverting means arguing against a metric that improved. What closes it: the variant on every call, priced from a dated book, set against what those customers actually pay.
What this usually looks like
- Your experiment scorecard has quality columns and no cost column.
- A winning variant shipped and the bill moved the following month.
- Nobody can say what the losing arm would have cost.
- Experiments run on production traffic and their spend is invisible inside the total.
- You can rank variants by score and not by cost per point of score.
Free, no card, no account
Run the free Cost Leak Scan
It shows your most expensive conversation before you install anything.
Mistakes that cost the most
| Mistake | Why it hurts | Do instead |
|---|---|---|
| Choosing a variant on quality alone. | It picks a winner without pricing it, and the price can be a multiple rather than a margin. | Report cost per call beside score per arm, and decide on both. |
| Not tagging experiment traffic. | Arm-level spend disappears into the total, so neither arm can be priced afterwards. | Record the experiment and variant on every call the test produces. |
| Comparing arms on total spend. | Arms rarely get equal traffic, so totals compare allocation rather than the variants. | Compare cost per call, which is independent of how the split was weighted. |
| Ignoring what the experiment itself costs. | Running two arms in production means paying for both, plus any judge model scoring them. | Budget the experiment as a line, including evaluation calls, before launching it. |
Run this check tonight
- Open your most recent experiment result and look for a cost column.
- Work out cost per call for the arm you shipped and the arm you rejected.
- Divide the cost difference by the quality difference, and decide whether you'd pay that.
- Check whether experiment traffic is distinguishable from normal traffic in your ledger.
The winner, priced
Illustrative example
Two modelled prompt variants, each priced at 3,000,000 calls a month, which is the volume the winner serves once it ships rather than the split during the test. Rates are Claude Haiku 4.5 at a real $1.00 and $5.00 per million from the price book effective 2026-07-02. Arm A uses 2,000 input and 300 output tokens. Arm B adds a self-critique pass, taking it to 7,000 input and 1,200 output. Quality scores and token shapes are modelled.
If arm B won by four points of quality, that's $7,125.00 a month per point. It might be worth it. The problem is that the scorecard which chose it never showed the number, so nobody made that call deliberately.
Every number, with its confidence and source
| Figure | What it means | Confidence | Source |
|---|---|---|---|
| $28,500.00 per month | modelled cost difference between two experiment arms at 3 million calls a month | calculated | Arm A at 2,000 input and 300 output tokens costs $0.0035 per call, arm B at 7,000 input and 1,200 output costs $0.0130, both on Claude Haiku 4.5 at real rates of $1.00 and $5.00 per million from the price book effective 2026-07-02. At 3,000,000 calls that's $10,500.00 against $39,000.00, a difference of $28,500.00 and a ratio of 3.71. Token shapes and call volume are modelled. |
What a generic answer can’t know
Cost per arm is available, and Braintrust reports it across both experiments and logs. What no cost tool holds is the other half of the ratio, because what a customer pays you lives in your billing system and never appears in a provider invoice or a trace. Culpa carries whatever identifiers you attach, including the experiment and the variant, prices each call from a versioned price book in exact decimal rather than estimating it, and takes your revenue alongside. So the experiment stops reporting cost per variant and starts reporting margin per variant, which is the version of the question a business can act on, and the arm that costs triple sometimes wins it.
Questions founders ask next
Why does an A/B test need a cost column?
Because arms can differ in unit cost by multiples. In the modelled example a self-critique variant cost 3.71 times the baseline per call, which is $28,500.00 a month at 3 million calls. Braintrust does put estimated cost in its experiment view, so the column exists. The harder half is ranking arms on margin rather than cost, which needs what those customers pay you.
How do I price an experiment arm?
Tag every call with the experiment and variant, price each call from a rate book, then compare cost per call rather than total spend per arm. Totals reflect how you split traffic, so they compare your allocation rather than the variants themselves.
What number should decide the experiment?
Cost per unit of quality gained, alongside the raw score. Dividing the cost difference by the score difference turns the decision into a trade a business can reason about, rather than a metric that improved and a bill that moved a month later.
Does running the experiment itself cost anything?
Yes, and it's routinely left out. You pay for both arms in production, and for any LLM-as-judge calls scoring them. The judge traffic scales with the number of evaluators rather than with users, so it belongs in the experiment budget as its own line.
On your infrastructure
Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.
How Culpa works
Find the culprit. Not just the total.
Your dashboard shows what you spent. It stops short of who spent it. Culpa shows the conversation, the user and the feature behind it.
Your prompts stay local.
Culpa runs on your own infrastructure. What you send to a model reaches us at no point.
Every dollar has a name.
Follow any charge to the conversation, the user, the feature and the customer behind it.
See the bill before it lands.
Cost your next feature before you ship it. You get the likely bill and the worst case, at best, median, p90 and p99.
Three steps to your first answer.
Change one base URL.
Or drop in the Python or TypeScript library.
Find your most expensive conversation.
In the first session, not the first week.
Cost your next feature before you ship it.
Why the bill went up
Example dashboardCalls traced
418,209
across 3 projects
Spend this week
$378.41
+ $182 vs last week
Failed calls
312
74% retried, and you paid for all of them
+ $182 this week traced to one culprit
Spend over 14 days
Most expensive users
Next week forecast
Graded against reality. Accuracy shown as results land.
Keep reading
Sources: Anthropic pricing, Braintrust, interpreting eval results, Braintrust, logging production traffic. Last reviewed 2026-08-03, rates effective 2026-07-02. Plain text version.