Controlling LLM Costs: Route by Task, Cap by Day

LLM (large language model) costs need to be measured per task as well as in the monthly bill. Small per-request charges can add up when traffic increases. It usually isn't one expensive feature that does it. A system may send every request, trivial or complex, to the same expensive model because nobody revisited the configuration after launch.

Cost control on an LLM system isn't a finance problem you solve after the fact. It's an architecture decision you make before the first request goes out.

Route by task, not by default

Not every LLM call in a system needs the same model. Think of it like a clinic: a minor complaint gets handled by a nurse, a complex case goes up to the specialist. A production system works the same way — several kinds of calls per user turn, each with different requirements.

• Judging and scoring — is this passage relevant, does this quote actually appear in the source, does this classification hold up — is a narrow, structured task. The model reads a short input and returns a small, constrained output: a label, a confidence number, a yes/no. A cheaper model is worth testing on this bounded task, but classification and source judgments can still be wrong. Compare against labelled examples, especially high-impact decisions.
• Enrichment and extraction — pulling structured fields out of unstructured text, drafting a short research plan — is similarly bounded. A cheaper model, a focused prompt, and low temperature (an output-variation setting) can be starting points for testing. Low temperature does not ensure correctness.
• Final answer generation — the response the user actually reads — is where you want your strongest model, because this is where output quality actually shows up.

In a regulatory-classification assistant I built, the relevance-judging pass that checks whether a proposed code actually fits a business runs on a separate, cheaper model than the one that drafts the final explanation. The judge fires many times per turn — once per candidate, sometimes dozens per conversation — while the final answer fires once.

Routing the high-volume, narrow-task calls to the cheap model, and reserving the expensive one for the single generation step that needs it, cut the effective cost per turn by a wide margin. In that system, separating models reduced costs. This does not guarantee unchanged quality for every task: measure judge errors and final-answer errors after a model change.

Daily budget caps, not just per-request limits

Per-request rate limiting stops abuse from one client hammering your endpoint. It does nothing to stop a normal day of legitimate traffic from quietly exceeding what you're willing to spend. A daily cost cap — a hard ceiling on how many paid model calls the system will make per day, enforced per user and globally — is the backstop a rate limiter can't provide.

The practical version weights the cap by request size instead of counting every request as equal: a longer prompt costs more "units" than a short one. That workload quota is separate from a spending ceiling: reserve the estimated monetary cost, limit output tokens, and reconcile actual usage after each call. Clipping the units charged for a large request does not cap its provider bill.

Once a user or the system as a whole hits the ceiling, the system degrades gracefully — it falls back to a non-LLM answer or an honest "at capacity" message — rather than erroring out or, worse, continuing to spend past the limit you set.

Per-turn deadlines protect the budget from a single slow call

A daily cap controls total spend. A per-turn deadline controls how much any single conversation can cost in time and, indirectly, in money. Set a wall-clock deadline for the whole turn, and cap every individual model call inside it at whatever time remains.

When the deadline is reached, don't make one more call hoping it finishes in time. Stop and return the fallback immediately. A slow model call that blows through a deadline anyway is worse than no call at all, because you paid for a response the user never got in time to use.

This also protects against a specific failure: an agentic loop that keeps calling tools and re-prompting the model round after round. Without a hard round cap and a deadline, a difficult query can spiral into a dozen expensive calls chasing an answer that was never going to resolve cleanly. The deadline forces an honest stop instead.

Measure cost per answered question, not just total spend

Total monthly spend alone does not explain which tasks drive the cost. What you want is cost per answered question, broken down by which path answered it — how many turns were handled by the expensive model, how many by the cheap model, how many fell back to a deterministic non-LLM answer, how many hit the budget cap and got an honest decline.

That breakdown tells you whether your routing is actually working, or whether everything is quietly still hitting the expensive path regardless of what you configured.

Why one-model-for-everything burns money

The instinct to route everything through your best model comes from a reasonable place: nobody wants to be the one who shipped a worse answer to save a few cents. But the math doesn't work like "a few cents."

An agentic system can make five, ten, or more model calls in a single user turn: search, judge, judge again, compose, verify. If every one of those calls uses your top-tier model's pricing, your cost per user question multiplies by the number of internal calls, not by the number of questions asked.

Route by what the task actually needs. Cap what you're willing to spend per day. Set a clock on every turn. That's the difference between an LLM feature that scales and one that gets pulled the first time someone reads the invoice.

For the retrieval work that feeds these model calls, see RAG explained for business systems, and for how this budget discipline connects to answer trustworthiness, see guardrails against hallucination.