# Why the pilot said it was cheap and production disagreed

> A demo's cost per call is the figure everyone extrapolates and the one that changes most. A modelled forecast lands 78% under at production scale.

URL: https://getculpa.com/poc-to-production-cost
Last reviewed: 2026-08-05
Rates effective: 2026-07-02

## Answer
A proof of concept measures cost per call under conditions production never repeats: short conversations, no retries, no tail users. Multiplying that figure by real volume under-forecasts, often badly. Culpa, a local-first LLM cost, margin, and forecast ledger, prices your real traffic so the forecast comes from production rather than from a demo.

## Why this happens
The pilot works, the bill is trivial, and somebody divides it by the request count to get a cost per call. That number then gets multiplied by the production forecast, and it's the one input in the whole exercise that doesn't hold. A demo runs short conversations, because nobody demos a forty-turn session. It has no retry rate, because it ran on a good afternoon. It has no tail of heavy users, because it had six users and they were all colleagues. Each of those pushes real cost per call up, and they compound rather than average out. There's a second, opposite effect that confuses the argument: some savings only become available at scale. Caching a large system prompt loses money on a pilot and pays handsomely in production, so the pilot is simultaneously over-optimistic about volume-driven cost and over-pessimistic about what optimisation can recover. The only way through is to measure the shape rather than the total, which means knowing your real conversation lengths, retry rate and usage distribution before committing to a number.

## What this usually looks like
- Your production forecast is the pilot's cost per call times expected volume.
- The pilot ran conversations far shorter than real ones.
- Nobody knows your retry rate because the pilot never hit one.
- Cost per call has risen every month since launch and nobody expected it to move.
- The business case was signed off on a number measured over a few hundred requests.

## Common mistakes
- Extrapolating the pilot's cost per call. Why it hurts: It's the one figure the pilot measures least reliably, and it moves upward with scale. Do instead: Forecast from the shape: conversation length, retry rate and the usage distribution.
- Demoing short conversations. Why it hurts: Input grows with turn number, so real sessions cost more per call than demo ones. Do instead: Run the pilot at realistic conversation lengths, even if it's slower to demo.
- Assuming usage is evenly distributed. Why it hurts: A small tail of heavy users can dominate the bill, and a pilot rarely contains one. Do instead: Model a skewed distribution explicitly, and price the top decile separately.
- Writing off caching because the pilot showed no benefit. Why it hurts: A cache write costs more than base input, so caching genuinely loses money at pilot volumes. Do instead: Re-test caching at production call rates rather than concluding from the pilot.

## Self-check
- Compare your pilot's average conversation length to a real session today.
- Find your actual retry rate, and check whether the pilot's forecast included one.
- Look at usage by user and see how much of it your top decile accounts for.
- Re-run the caching decision at production call volume rather than pilot volume.

## The pilot's figure against the production one (illustrative)
A modelled pilot at 3,000 input and 500 output tokens per call on Claude Sonnet 5, at its real introductory rates of $2.00 and $10.00 per million from the price book effective 2026-07-02. Production runs the same feature at 6,000 input, because conversations accumulate history, with a 15% retry rate. Volume is 5,000,000 calls a month. Rates are published, every volume and shape below is modelled.
  pilot per call: 3,000 x $2.00/M + 500 x $10.00/M = $0.0060 + $0.0050 = $0.0110
  the pilot's own bill over 1,000 calls: $11.00, which reads as free
  forecast at 5,000,000 calls using that figure: $55,000.00 a month
  production per call: (6,000 x $2.00/M + 500 x $10.00/M) x 1.15 = $0.0170 x 1.15 = $0.01955
  actual at 5,000,000 calls: $97,750.00, which is $42,750.00 and 78% above the forecast
Nothing in that gap is a price change or a mistake in the arithmetic. The pilot measured a real cost per call for the conditions it ran under, and production ran under different ones. The forecast was wrong the moment it treated a measurement as a constant.

## Cost figures
Every figure carries its confidence and its source. No figure on this page is provider-reported.
- $97,750.00 against $55,000.00, modelled monthly cost at production shape against the same volume forecast from a pilot's cost per call [calculated] Source: A modelled 3,000 input and 500 output tokens per call on Claude Sonnet 5 at real introductory rates of $2.00 and $10.00 per million from the price book effective 2026-07-02 gives $0.0110, so 5,000,000 calls forecasts $55,000.00. At a modelled production shape of 6,000 input, the same 500 output and a 15% retry rate, per-call cost is $0.0170 x 1.15 = $0.01955, so the same volume costs $97,750.00, which is $42,750.00 and 78% higher. Rates are published, every volume and shape is modelled.

## FAQ
Q: Why did production cost more than the pilot predicted?
A: Usually conversation length and retries. A pilot runs short sessions with no retry rate, and both push real cost per call up. In the modelled example a pilot figure of $0.0110 became $0.01955 in production, so a $55,000.00 forecast landed at $97,750.00.

Q: How should I forecast from a pilot?
A: Forecast the shape rather than the total. Take your real conversation length, an assumed retry rate and a skewed usage distribution, price those, and give a range. The pilot's cost per call is the least transferable number it produces.

Q: Does anything get cheaper at scale?
A: Yes, and that's what makes this confusing. Caching a large prompt loses money at pilot volumes, because a cache write costs more than base input and a pilot doesn't read it enough times to break even. The same cache pays clearly in production, so a pilot can be pessimistic and optimistic at once.

Q: What should the pilot actually measure?
A: Token shapes, not dollars. How long conversations really run, what share of calls retry, and how skewed usage is across users. Those three transfer to production. The total bill doesn't, and it's the thing everybody writes down.

## Sources
- Anthropic pricing: https://platform.claude.com/docs/en/docs/about-claude/pricing

Run the free Cost Leak Scan: https://app.getculpa.com/scan?source=pseo&slug=poc-to-production-cost&cluster=problem

Machine-readable index of every guide: https://getculpa.com/api/pages
Human-readable index of every guide: https://getculpa.com/guides
Site overview: https://app.getculpa.com/llms.txt

Privacy: Culpa runs on your infrastructure. Your prompts and responses never leave it. Culpa counts calls to run your plan, and it fails open, so if it ever breaks your app keeps running.