# Costing the arms of an LLM A/B test > Experiment arms differ in unit cost by multiples, not percentages. Tools that show it call it estimated. Margin per variant needs the revenue side too. URL: https://getculpa.com/llm-ab-test-cost Last reviewed: 2026-08-03 Rates effective: 2026-07-02 ## Answer An LLM A/B test compares variants on quality, and each variant also carries its own cost per call. Judging on quality alone picks a winner without pricing it. Culpa, a local-first LLM cost, margin, and forecast ledger, prices every call and tags it with the variant, so an experiment reports cost per arm beside the score. ## Why this happens Every arm of a prompt or model test has a unit cost, and those costs can differ by multiples rather than percentages: a longer prompt, a reasoning model, an extra retrieval pass or a self-critique step all buy quality with money. Plenty of tooling shows you this. Braintrust puts estimated cost beside duration and token counts in an experiment, specifically so you can weigh the trade when you switch models, and its logs carry the same instrumentation into production. So the gap isn't coverage, and anyone telling you otherwise hasn't read the docs. It sits in two narrower places. The first is the word estimated, which is the honest label for a figure derived from a rate table nobody versions by date, and rates move: Claude Sonnet 5 leaves introductory pricing on 2026-09-01. The second is that cost alone can't rank arms, because the arm that costs triple may also serve the customers who pay you most. A team ships the variant that won by four points and meets the real number at the end of the month, by which point reverting means arguing against a metric that improved. What closes it: the variant on every call, priced from a dated book, set against what those customers actually pay. ## What this usually looks like - Your experiment scorecard has quality columns and no cost column. - A winning variant shipped and the bill moved the following month. - Nobody can say what the losing arm would have cost. - Experiments run on production traffic and their spend is invisible inside the total. - You can rank variants by score and not by cost per point of score. ## Common mistakes - Choosing a variant on quality alone. Why it hurts: It picks a winner without pricing it, and the price can be a multiple rather than a margin. Do instead: Report cost per call beside score per arm, and decide on both. - Not tagging experiment traffic. Why it hurts: Arm-level spend disappears into the total, so neither arm can be priced afterwards. Do instead: Record the experiment and variant on every call the test produces. - Comparing arms on total spend. Why it hurts: Arms rarely get equal traffic, so totals compare allocation rather than the variants. Do instead: Compare cost per call, which is independent of how the split was weighted. - Ignoring what the experiment itself costs. Why it hurts: Running two arms in production means paying for both, plus any judge model scoring them. Do instead: Budget the experiment as a line, including evaluation calls, before launching it. ## Self-check - Open your most recent experiment result and look for a cost column. - Work out cost per call for the arm you shipped and the arm you rejected. - Divide the cost difference by the quality difference, and decide whether you'd pay that. - Check whether experiment traffic is distinguishable from normal traffic in your ledger. ## The winner, priced (illustrative) Two modelled prompt variants, each priced at 3,000,000 calls a month, which is the volume the winner serves once it ships rather than the split during the test. Rates are Claude Haiku 4.5 at a real $1.00 and $5.00 per million from the price book effective 2026-07-02. Arm A uses 2,000 input and 300 output tokens. Arm B adds a self-critique pass, taking it to 7,000 input and 1,200 output. Quality scores and token shapes are modelled. arm A per call: 2,000 x $1.00/M + 300 x $5.00/M = $0.0020 + $0.0015 = $0.0035 arm B per call: 7,000 x $1.00/M + 1,200 x $5.00/M = $0.0070 + $0.0060 = $0.0130 at 3,000,000 calls: arm A = $10,500.00, arm B = $39,000.00 arm B costs 3.71 times arm A, a difference of $28,500.00 a month If arm B won by four points of quality, that's $7,125.00 a month per point. It might be worth it. The problem is that the scorecard which chose it never showed the number, so nobody made that call deliberately. ## Cost figures Every figure carries its confidence and its source. No figure on this page is provider-reported. - $28,500.00 per month, modelled cost difference between two experiment arms at 3 million calls a month [calculated] Source: Arm A at 2,000 input and 300 output tokens costs $0.0035 per call, arm B at 7,000 input and 1,200 output costs $0.0130, both on Claude Haiku 4.5 at real rates of $1.00 and $5.00 per million from the price book effective 2026-07-02. At 3,000,000 calls that's $10,500.00 against $39,000.00, a difference of $28,500.00 and a ratio of 3.71. Token shapes and call volume are modelled. ## FAQ Q: Why does an A/B test need a cost column? A: Because arms can differ in unit cost by multiples. In the modelled example a self-critique variant cost 3.71 times the baseline per call, which is $28,500.00 a month at 3 million calls. Braintrust does put estimated cost in its experiment view, so the column exists. The harder half is ranking arms on margin rather than cost, which needs what those customers pay you. Q: How do I price an experiment arm? A: Tag every call with the experiment and variant, price each call from a rate book, then compare cost per call rather than total spend per arm. Totals reflect how you split traffic, so they compare your allocation rather than the variants themselves. Q: What number should decide the experiment? A: Cost per unit of quality gained, alongside the raw score. Dividing the cost difference by the score difference turns the decision into a trade a business can reason about, rather than a metric that improved and a bill that moved a month later. Q: Does running the experiment itself cost anything? A: Yes, and it's routinely left out. You pay for both arms in production, and for any LLM-as-judge calls scoring them. The judge traffic scales with the number of evaluators rather than with users, so it belongs in the experiment budget as its own line. ## Sources - Anthropic pricing: https://platform.claude.com/docs/en/docs/about-claude/pricing - Braintrust, interpreting eval results: https://www.braintrust.dev/docs/evaluate/interpret-results - Braintrust, logging production traffic: https://www.braintrust.dev/docs/observe Run the free Cost Leak Scan: https://app.getculpa.com/scan?source=pseo&slug=llm-ab-test-cost&cluster=problem Machine-readable index of every guide: https://getculpa.com/api/pages Human-readable index of every guide: https://getculpa.com/guides Site overview: https://app.getculpa.com/llms.txt Privacy: Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.