How to save AI costs with LLM routing

August 9, 2026

Most production workloads mix simple extraction, retrieval-backed questions, short transformations, tool calls, and difficult reasoning. Sending all of them to the strongest model pays the highest rate for requests with very different requirements.

LLM routing sends each request to the least expensive eligible model that meets the workflow's quality, latency, data-handling, and tool-use requirements. Keep the stronger model available for hard cases and explicit escalation. Define a successful result before optimizing the route.

Why one model creates a cost ceiling

Request count alone says little about an LLM bill. A useful starting model is:

request cost = input tokens × input rate + output tokens × output rate + tool charges + retries + routing and infrastructure overhead

Current provider rate cards separate input and output pricing, and some also distinguish cached input, cache writes, batch processing, or tool use. See the current OpenAI API pricing, Anthropic pricing, and Gemini API pricing for examples. Model names and rates change, so keep production routing rates in configuration, outside application code.

If one expensive model handles every request, you leave three savings opportunities unused:

• easy requests do not need frontier-level reasoning;
• repeated context may not need to be processed at the full input rate; and
• asynchronous work may not need the same service tier as an interactive response.

Routing addresses the first opportunity directly. Caching, shorter prompts, shorter outputs, and batch processing address the others. They work together, but they solve different problems.

Consider an illustrative calculation. Suppose 100,000 requests each contain 1,500 input tokens and produce 300 output tokens. Assume a stronger model costs $5 per million input tokens and $30 per million output tokens. Sending every request to it costs about $1,650 in model-token charges. Now assume a cheaper model costs $0.20 per million input tokens and $1.20 per million output tokens, and an evaluated router sends 80% of requests there while keeping 20% on the stronger model. The same workload costs about $383 before router overhead, retries, cache effects, and other charges. The large assumed rate difference drives the apparent reduction. Your result will depend on your rate card, traffic mix, output length, and quality failures.

Savings depend on the share of work that can safely move to a lower-cost route. A router that sends 99% of traffic to the expensive model may still improve reliability or latency, but it will not produce a large token-cost reduction.

Route by the work

Start by classifying the work your application already does. Avoid labels such as “easy” and “hard” unless you define them with observable signals. A first policy might look like this:

Work type	Starting route	Escalate when
Extraction, classification, and short rewriting	A small, fast model with a strict output schema	The schema is invalid, confidence is low, or the item is out of distribution
Retrieval-backed product questions	A lower-cost general model with the relevant context	Retrieved evidence is missing, contradictory, or the response needs synthesis across sources
Multi-step planning and tool selection	A middle tier with the tools it is allowed to call	The plan is ambiguous, a tool call fails, or the action has a higher business risk
Code, long-form reasoning, or novel workflows	A stronger model	Only after the task meets the stronger model's quality requirement

The names of the models are less important than the contract for each route. For example, “classification” should include the allowed labels, the acceptable error types, and what happens when the model is uncertain. “Product action” should include the permitted tools, user permissions, tenant boundary, and whether a human or application confirmation is required.

Eligibility comes before price. A model that fails the required schema, tool, context-window, or data-policy requirement will create retry and correction costs regardless of its token rate.

Amazon Bedrock documents one managed version of this pattern. Its intelligent prompt routing analyzes a request, predicts response quality across models in a family, and uses a configured quality difference against a fallback model when deciding whether to switch. AWS also recommends monitoring quality and cost and reviewing performance as models change. The design is a useful architecture reference; your workload still needs its own economic test. See AWS's intelligent prompt routing documentation.

Build the router in a measurable sequence

1. Establish a single-model baseline

Before adding routing, log the current path. At minimum, record:

• workflow or task class;
• model and provider;
• input and output token counts;
• time to first token and total latency;
• tool calls and retries;
• structured-output or application-level success;
• human correction, user re-ask, or fallback events; and
• cost using the rate card active at request time.

Do not use average cost per request as the only baseline. A small number of long agent runs can dominate spend, while a route that looks cheap may generate more corrections. Cost per successful task is a more useful operating measure:

cost per successful task = total model, tool, retry, router, and infrastructure spend ÷ successful tasks

2. Define quality before choosing a cheaper model

Create an evaluation set from real, representative requests. Include routine cases, edge cases, long context, malformed inputs, permission-sensitive requests, and cases where the right answer is to ask for clarification or refuse an action.

For each workflow, define an evaluation metric: exact-match accuracy for extraction, groundedness for retrieval, a valid tool-call sequence for an agent, or a human-approved result for a high-consequence action. The metric needs to make the route decision visible.

OpenAI's model selection guidance follows the same order: set an accuracy target, build an evaluation dataset, then compare smaller models for cost and latency while preserving the target. A router is the runtime expression of that process. It should not be used to avoid doing the evaluation.

3. Create an eligibility filter

For each request, filter the candidate models using hard constraints before applying a cost preference. Useful constraints include:

• required context length and modalities;
• structured-output or tool-calling support;
• provider, region, or retention requirements;
• tenant and permission rules;
• latency budget;
• maximum output length; and
• whether the request may be retried on another provider.

The filter should be deterministic wherever possible. A policy that can explain why a model was ineligible is easier to audit than a score that only says the model “felt” unsuitable.

4. Choose the least-cost eligible route

Once the candidate set is safe and capable, choose the least-cost model that cleared the quality target for that task class. Start with explicit rules. For example:

1. Send short, low-risk, schema-constrained work to the small model.
2. Send normal retrieval and product questions to the lower-cost general model.
3. Escalate when the classifier, validator, tool plan, or evaluation signal indicates that the first route is not enough.
4. Keep a strong fallback for provider failures and requests that cross the workflow's difficulty or risk boundary.

The first router does not need to be a separate LLM. Request metadata, endpoint, tool set, context size, account tier, and workflow state often provide enough signal to make an initial route. If you add a classifier model later, include its tokens and latency in the cost model. A router that costs nearly as much as the requests it redirects is not optimizing the right layer.

5. Add fallbacks without hiding their cost

A fallback buys reliability by accepting the risk of a second bill and a longer response after the primary call has consumed tokens. Record the original route, failure reason, fallback route, total tokens, and final outcome.

Order fallbacks by eligibility and reliability, then price. A lower-cost fallback that cannot produce a valid tool call is worse than a more expensive model that completes the task on the first retry. Libraries and gateways can implement model fallbacks, but the application still needs a policy for which failures are retryable and which should stop and ask for help. For an example of SDK-level fallback behavior, see LiteLLM's reliability documentation.

6. Roll out gradually and compare successful work

Run the proposed routes in shadow mode or on a controlled slice of traffic. Compare the routed path with the baseline on the same task categories. Watch for changes in:

• successful completion rate;
• correction and re-ask rate;
• output length and tool-call count;
• p50 and p95 latency;
• fallback rate;
• cost per request; and
• cost per successful task.

Route share alone cannot prove savings. Pair “80% of requests use the small model” with the completion rate, required quality threshold, and total cost for those requests.

Use caching and batch processing where they fit

Repeated context creates token spend even when the router chooses the right model. Put stable system instructions, tool definitions, and reusable background context in a cacheable prefix when the provider supports it. OpenAI documents cached input pricing and prefix-based cache routing in its prompt caching guide. Anthropic also documents cache reads, cache writes, time-to-live options, and the requirement for matching prompt segments in its prompt caching guide.

Caching does not make a poor route good. A large cached prompt can still be attached to an unnecessarily strong model, and a cache miss can change the economics. Track cache hits and misses by route.

Use an asynchronous tier for work without an immediate-response requirement. OpenAI's Batch API documentation currently describes a 50% lower cost than synchronous APIs and a stated 24-hour turnaround window. That offer and constraint are specific to the provider. Route interactive tasks for responsiveness and batch-suitable work for cost, then measure the end-to-end workflow as well as the model call.

Common mistakes that erase the savings

Making the router another expensive agent

A model that reads the entire conversation, reasons about every possible provider, and then chooses a model can consume enough tokens to defeat the savings. Begin with deterministic signals and a small candidate set. Add model-based routing only when the measured quality gain exceeds its cost and latency.

Routing on prompt length alone

Token count belongs beside task type, tool permissions, risk, and evaluation results. A short request can trigger an expensive failure through an irreversible action, while a smaller model may handle a long repetitive classification task well.

Sending sensitive work to the cheapest provider

Provider eligibility, data handling, regional requirements, and tenant boundaries belong in the filter. Price should be applied only after those constraints are satisfied. Document the allowed providers for each data class and make an ineligible route impossible to select.

Treating every failure as a retry

Transient rate limits, timeouts, invalid schemas, policy blocks, and incorrect answers require different responses. A retry may clear a timeout; it cannot repair an invalid task definition or permission violation. Classify failures so the router knows when to retry, escalate, or stop.

Hardcoding model names into business logic

Model names, capabilities, tokenizers, and prices change. Keep a route policy separate from the provider adapter. Store model capabilities, current rates, and availability as versioned configuration, and record the configuration version with each request. This makes a cost change explainable when a provider updates a rate card or retires a model.

Where Ginger Labs fits

An embedded SaaS agent is a natural place to apply this policy because the same user-facing copilot may answer a product question, retrieve records, plan a multi-step operation, or perform an authorized action. Those jobs do not need identical model capacity or latency.

The Ginger Labs embedded AI agent or copilot can live inside a customer's SaaS or web application, answer questions, and perform product work across the product's schemas, stages, records, and data. Its SDK includes retrieval, evaluations, self-learning loops, and observability. In this architecture, the route policy sits beside the agent request layer, where the application can see the workflow type, available tools, tenant context, and evaluation result. It can send routine work to an eligible lower-cost route and reserve a stronger route for difficult or higher-risk work.

The customer still owns the product API and data model, domain rules, user permissions, tenant boundaries, allowed actions, customer-facing experience, and definition of a correct result. Ginger Labs does not replace those decisions, and routing does not remove them. If a workflow cannot state what “correct” means, it is not ready for cost optimization by model selection.

Build or buy the routing layer

Build a focused router when you have a small number of workflows, a limited model set, and rules that your team can test directly. You keep control of eligibility, evaluation, pricing configuration, and telemetry. The trade-off is operational ownership: provider adapters, retries, rate limits, audit data, and model changes become your responsibility.

Use a gateway or managed routing service when multi-provider failover, centralized spend controls, and shared observability justify giving up direct ownership of every integration. Confirm how it handles prompt data, tenant isolation, tool calls, rate limits, fallback billing, and model capability changes. An abstraction can reduce integration work while adding its own cost, latency, and dependency.

For either approach, keep the application-level quality contract in your code and tests. No routing service can infer the business cost of a wrong answer, an unauthorized action, or a customer having to repeat the same request unless you measure those outcomes.

The next step

Choose one high-volume workflow and measure it for a week or another representative operating window. Capture its current model-token cost, success rate, latency, retries, and correction rate. Build a small evaluation set, identify one cheaper eligible model and one escalation model, then route a controlled share of traffic. Keep the route only if cost per successful task falls without violating the workflow's quality, latency, or data constraints.

LLM routing saves money by reserving expensive models for work that earns their cost. Measure savings per successful task, keep failed routes in the bill, and revise the policy as models and workloads change.

Sources

The following primary sources were checked on August 9, 2026. Provider prices, model availability, and feature behavior can change.

• OpenAI API pricing
• OpenAI model selection guidance
• OpenAI prompt caching
• OpenAI Batch API
• Anthropic API pricing
• Anthropic prompt caching
• Google Gemini API pricing
• Amazon Bedrock intelligent prompt routing
• LiteLLM reliability and fallbacks