LLM Routing: Send Each Task to the Cheapest Model That Passes

Model prices differ by more than an order of magnitude between the small and large tiers of the same provider. A system that runs every call on the largest model is paying premium rates for work a small model would do just as well, such as classification, extraction, formatting and short replies.

Routing fixes that. It is one of the simplest high-return decisions in AI system design.

Three kinds of routing

1. Routing by step

Decided at design time. Each step in a workflow is assigned a model. Intake conversation: small model. Classification: small model. Planning a complex build: large model. Final copy: large model. Checks: code. This alone usually removes most of the waste, and it needs no extra infrastructure.

2. Routing by request

Decided at run time. A cheap classifier reads each incoming request and labels it simple or complex, or by type. Simple requests go to a small model, complex ones to a large model. The classifier can be a small model, an embedding similarity check, or plain rules on length and keywords.

3. Routing by fallback

Try the small model first. Run a code check on its output. If it fails, retry on the large model. This works well when most requests are easy and failures are easy to detect, such as structured extraction with a schema.

How to decide which model a step needs

1. Build a test set of 30 to 100 real examples for the step.
2. Run the smallest candidate model on all of them.
3. Score the outputs with code checks and, where needed, a review against a rubric.
4. Move up a tier only if the pass rate is below your bar.

Teams often skip this and pick the large model "to be safe". The test set usually shows the small model passes most steps.

An example

In the page builder I run, a small, fast model handles the intake chat with each customer, asking questions until the brief is complete. Only then does a large model take the brief and build the page. The intake can run for many turns, and none of those turns are billed at large model prices.

Routing pitfalls

• A router that costs more than it saves. Do not use a large model to decide whether to use a large model.
• Prompts written for one model. Test each prompt on the model that will run it.
• No logging of the route. Record which model handled each call so you can compare cost and quality.

Routing pairs naturally with prompt caching: route to the right model, then make each call's stable prefix cacheable.

A worked example: routing a lead pipeline

Three of four steps moved down a tier. Only the step that genuinely needed the large model kept it. Quality on the test set held, and the cost per lead dropped sharply because most calls in the pipeline were the cheap ones.

Routing and caching together

Each model has its own cache, so routing the same prompt between models at random reduces cache hits. Route by step where you can, so each step consistently uses one model and its stable prefix stays warm.

Routing across providers

Some systems route across providers as well as tiers, for price, availability or capability. Gateways that expose many models behind one API make this easier. Keep the same discipline: a test set per step, logged routes, and a fallback provider for outages.

How I use it

Every system on my systems page routes by step. The page builder runs its intake conversation on a small model and its build on a large one; the voice and chat bot uses fast models for live turns. It is one of the first changes I make when reviewing someone's AI costs.

Semantic routing

Semantic routing compares an incoming request with example requests for each route using embeddings, and picks the closest route. It is fast and cheap, and works well when routes are distinct, such as billing versus technical support. Combine it with a fallback: if no route is close enough, send the request to a general handler or a small model classifier.

A routing checklist

• Every step has a named default model.
• Every step has a test set and a pass bar.
• Escalation to a larger model is triggered by a check, not by guesswork.
• The router itself is cheaper than the savings it creates.
• The chosen model is logged on every call.
• A fallback provider exists for outages on critical paths.

Frequently asked questions

What is LLM routing?

LLM routing sends each request or workflow step to a chosen language model based on what the task needs, typically using smaller, cheaper models for simple tasks and larger models only where they are required.

How does LLM routing reduce cost?

Small models cost far less per token than large ones. Routing classification, extraction and short replies to small models, and keeping large models for planning and complex writing, can cut model spend substantially without lowering quality.

How do I know which model a task needs?

Build a test set of real examples for the task, run the smallest candidate model, score the results with code checks or a rubric, and only move to a larger model if the pass rate is too low.

What is an LLM router?

An LLM router is a component, often a small model or a set of rules, that decides which language model should handle each request or step based on its type, difficulty or required quality.

Does routing between models affect prompt caching?

Yes. Caches are per model, so randomly switching models lowers cache hits. Routing consistently by step keeps each model's cached prefix warm.

What is semantic routing?

Semantic routing uses embeddings to compare a request with example requests for each route and sends it to the closest match. It is a fast, cheap way to route requests that fall into clearly different categories.