Tactical · 2026-07-07 · 5 min read

The token bill nobody forecasts.

The demo cost you thirty cents. The production system costs forty thousand dollars a month, and the finance team wants to know why. The gap is made of calls nobody put in the spreadsheet.

TL;DR

Your token bill is dominated by the calls that aren't in the demo: retries, multi-step agent loops, re-ranking, eval runs, and context that grows silently.
Cost is an architecture property, not a line item you negotiate after the fact.
Three levers move the bill more than model choice: context discipline, caching, and cascade routing.

Here's a number that surprises people: in most production LLM systems we're asked to cost-optimize, the single user-facing answer the customer sees accounts for well under half the tokens spent producing it. The rest goes to work the user never sees and the original budget never counted.

A team prices its pilot by taking the token count of one clean request-response and multiplying by expected volume. That estimate is off by a factor of three to ten, every time, because a production system does far more per user action than a demo does. If you want a forecast that survives contact with the invoice, you have to count the invisible calls.

Where the tokens actually go.

These are the line items that don't show up in the demo and dominate the bill in production:

Retries

Timeouts, malformed outputs, and validation failures each re-run the call. A 5% retry rate on a multi-step task compounds fast.

Agent loops

One user request becomes eight model calls as the agent plans, retrieves, checks, and acts. The demo showed you one.

Grounding

Re-ranking, summarization, and citation passes are model calls too, often on large retrieved payloads.

Eval runs

Every deploy runs the eval suite. LLM-as-judge scoring across hundreds of cases is a recurring, forecastable cost few people forecast.

There's a fifth, quieter one: context creep. A conversation that carries its full history forward grows its per-turn token count every single turn. By turn ten, each response costs several times what turn one did, and nobody notices until the monthly bill does.

Lever 1: context discipline.

The cheapest token is the one you don't send. Most systems we audit are shipping far more context than the task needs: whole documents where four sentences would do, full histories where a summary would do. Tightening what goes into the window is usually the single biggest cost win available, and it improves answer quality at the same time because the model isn't diluting attention across noise.

This is the same discipline that makes a system accurate. Cost and quality are aligned here, which is rare and worth exploiting.

Lever 2: caching.

A surprising fraction of what you send is identical across requests: the system prompt, the policy set, the tool definitions, the few-shot examples. Sending that prefix fresh every call is pure waste. Prompt caching lets you pay full price once and a fraction thereafter for the stable prefix, and for high-volume systems the savings are not marginal: they're the difference between viable and not.

Structure the request so the stable content sits at the front and the variable content at the back. It's a small architectural decision with an outsized effect on the bill.

Lever 3: cascade routing.

Not every request needs your most capable model. A large share of production traffic is routine (classification, extraction, simple lookups) that a smaller, cheaper, faster model handles perfectly. Route those to the cheap model and reserve the frontier model for the requests that actually need its reasoning.

The trick is the router: a cheap first pass that decides whether a request is routine or hard, and escalates only when needed. Done well, cascade routing cuts cost dramatically with no visible quality loss, because the hard requests still get the good model. They're just a minority of the traffic.

Forecast it before you ship it.

The right time to model your token economics is before launch, not after the first invoice. Instrument the pilot to count every model call per user action: not the clean path, but the real path with retries and loops and grounding. Multiply that by real volume. The number will be uncomfortable, and it's far better to be uncomfortable in a planning meeting than in a budget review.

Model choice matters, but it's rarely the biggest lever. The biggest levers are architectural: what you send, what you cache, and what you route where. Teams that treat cost as an engineering property from day one ship systems that scale. Teams that treat it as something to optimize later ship systems that get quietly capped when finance notices.

BizzSoftware designs, builds, secures, and runs the internal applications your teams work in every day, with AI features built in. About us →