Controlling LLM Costs: Route by Task, Cap by Day

Controlling LLM Costs: Route by Task, Cap by Day

Most teams' first LLM bill teaches the same lesson: token cost feels free per request, then becomes the largest new line item within a quarter. It usually isn't one expensive feature that does it. It's a system quietly sending every request, trivial or not, to the same top-tier model — because that's the default the SDK ships with and nobody revisited it after launch.

Cost control on an LLM system isn't a finance problem you solve after the fact. It's an architecture decision you make before the first request goes out.

Route by task, not by default

Not every LLM call in a system needs the same model. Think of it like a clinic: a minor complaint gets handled by a nurse, a complex case goes up to the specialist. A production system works the same way — several kinds of calls per user turn, each with different requirements.

• Judging and scoring — is this passage relevant, does this quote actually appear in the source, does this classification hold up — is a narrow, structured task. The model reads a short input and returns a small, constrained output: a label, a confidence number, a yes/no. A fast, cheap model handles this reliably, because the task doesn't need the broad reasoning a flagship model is priced for.
• Enrichment and extraction — pulling structured fields out of unstructured text, drafting a short research plan — is similarly bounded. Cheap model, tight prompt, low temperature.
• Final answer generation — the response the user actually reads — is where you want your strongest model, because this is where output quality actually shows up.

In a regulatory-classification assistant I built, the relevance-judging pass that checks whether a proposed code actually fits a business runs on a separate, cheaper model than the one that drafts the final explanation. The judge fires many times per turn — once per candidate, sometimes dozens per conversation — while the final answer fires once.

Routing the high-volume, narrow-task calls to the cheap model, and reserving the expensive one for the single generation step that needs it, cut the effective cost per turn by a wide margin. Answer quality didn't drop, because the judging task never needed flagship reasoning in the first place.

Daily budget caps, not just per-request limits

Per-request rate limiting stops abuse from one client hammering your endpoint. It does nothing to stop a normal day of legitimate traffic from quietly exceeding what you're willing to spend. A daily cost cap — a hard ceiling on how many paid model calls the system will make per day, enforced per user and globally — is the backstop a rate limiter can't provide.

The practical version weights the cap by request size instead of counting every request as equal: a longer prompt costs more "units" than a short one, but each request's contribution is capped so a handful of oversized requests can't silently drain the whole day's budget.

Once a user or the system as a whole hits the ceiling, the system degrades gracefully — it falls back to a non-LLM answer or an honest "at capacity" message — rather than erroring out or, worse, continuing to spend past the limit you set.

Per-turn deadlines protect the budget from a single slow call

A daily cap controls total spend. A per-turn deadline controls how much any single conversation can cost in time and, indirectly, in money. Set a wall-clock deadline for the whole turn, and cap every individual model call inside it at whatever time remains.

When the deadline is reached, don't make one more call hoping it finishes in time. Stop and return the fallback immediately. A slow model call that blows through a deadline anyway is worse than no call at all, because you paid for a response the user never got in time to use.

This also protects against a specific failure: an agentic loop that keeps calling tools and re-prompting the model round after round. Without a hard round cap and a deadline, a difficult query can spiral into a dozen expensive calls chasing an answer that was never going to resolve cleanly. The deadline forces an honest stop instead.

Measure cost per answered question, not just total spend

Total monthly spend tells you almost nothing actionable. What you want is cost per answered question, broken down by which path answered it — how many turns were handled by the expensive model, how many by the cheap model, how many fell back to a deterministic non-LLM answer, how many hit the budget cap and got an honest decline.

That breakdown tells you whether your routing is actually working, or whether everything is quietly still hitting the expensive path regardless of what you configured.

Why one-model-for-everything burns money

The instinct to route everything through your best model comes from a reasonable place: nobody wants to be the one who shipped a worse answer to save a few cents. But the math doesn't work like "a few cents."

An agentic system can make five, ten, or more model calls in a single user turn: search, judge, judge again, compose, verify. If every one of those calls uses your top-tier model's pricing, your cost per user question multiplies by the number of internal calls, not by the number of questions asked.

Route by what the task actually needs. Cap what you're willing to spend per day. Set a clock on every turn. That's the difference between an LLM feature that scales and one that gets pulled the first time someone reads the invoice.

For the retrieval work that feeds these model calls, see RAG explained for business systems, and for how this budget discipline connects to answer trustworthiness, see guardrails against hallucination.