Why the pilot said it was cheap and production disagreed

The pilot works, the bill is trivial, and somebody divides it by the request count to get a cost per call. That number then gets multiplied by the production forecast, and it's the one input in the whole exercise that doesn't hold. A demo runs short conversations, because nobody demos a forty-turn session. It has no retry rate, because it ran on a good afternoon. It has no tail of heavy users, because it had six users and they were all colleagues. Each of those pushes real cost per call up, and they compound rather than average out. There's a second, opposite effect that confuses the argument: some savings only become available at scale. Caching a large system prompt loses money on a pilot and pays handsomely in production, so the pilot is simultaneously over-optimistic about volume-driven cost and over-pessimistic about what optimisation can recover. The only way through is to measure the shape rather than the total, which means knowing your real conversation lengths, retry rate and usage distribution before committing to a number.

• Your production forecast is the pilot's cost per call times expected volume.
• The pilot ran conversations far shorter than real ones.
• Nobody knows your retry rate because the pilot never hit one.
• Cost per call has risen every month since launch and nobody expected it to move.
• The business case was signed off on a number measured over a few hundred requests.

1. Compare your pilot's average conversation length to a real session today.
2. Find your actual retry rate, and check whether the pilot's forecast included one.
3. Look at usage by user and see how much of it your top decile accounts for.
4. Re-run the caching decision at production call volume rather than pilot volume.

Illustrative example

A modelled pilot at 3,000 input and 500 output tokens per call on Claude Sonnet 5, at its real introductory rates of $2.00 and $10.00 per million from the price book effective 2026-07-02. Production runs the same feature at 6,000 input, because conversations accumulate history, with a 15% retry rate. Volume is 5,000,000 calls a month. Rates are published, every volume and shape below is modelled.

Nothing in that gap is a price change or a mistake in the arithmetic. The pilot measured a real cost per call for the conditions it ran under, and production ran under different ones. The forecast was wrong the moment it treated a measurement as a constant.

Conversation length, retry rate and the shape of your usage distribution are all properties of your own traffic, and none of them exist until you have traffic. That's the genuine difficulty here rather than an oversight: a pilot can't measure what a pilot doesn't have. What it can do is record the shape it does see, so the forecast is built from something rather than assumed. Culpa prices every call from a versioned price book in exact decimal and keeps the token shapes alongside the costs, so cost per call is a series you can watch climb rather than a constant you set once. Every forecast is stored and scored against what actually happened, which is what turns the second forecast into something better than the first.

• Power User AI Cost: Why Your Heaviest Account Is Heavy
• What is a blended rate
• What is LLM cost attribution
• AI Cost in Investor Diligence: They Ask for Concentration