LLM Routing Explained: Send Each Request to the Right Model
LLM routing is the practice of deciding, for every request, which model should answer it. Instead of sending all traffic to one large model, a router looks at the request (its type, its difficulty, the current load) and picks a model from a pool: a small fast one for easy work, a larger one for hard reasoning, a vision model for images, an embedding model for search.
The payoff is cost and speed. Most requests in a typical internal app are routine: short summaries, classification, extraction, rewording. A small model handles those well. Routing lets you reserve the expensive model for the requests that need it.
This guide covers the common routing strategies, how to build a simple rules-based router, when a learned router is worth it, and how to tell whether your router is helping or quietly hurting quality.
What LLM routing (model routing) actually decides
A router sits between your application and your models. Your app sends one request to one place, and the router chooses where it goes. The decision usually depends on one or more of these:
- What kind of task it is. An image needs a vision-language model. A request for vectors needs an embedding model. A code question may do better on a code-tuned model.
- How hard it is. "Translate this sentence" and "find the flaw in this 40-page contract" should not cost the same.
- What is available right now. If a model is down, overloaded or slow, the request goes elsewhere.
Model routing is not the same as load balancing. A load balancer spreads identical requests across identical copies of one model. A router picks between different models. In practice you often want both: route to the right model, then balance across its replicas.
Four routing strategies
1. Route by task or modality
This is the simplest and most reliable strategy, because the signal is explicit. A request with an image attached goes to a vision model. A request to embed documents goes to an embedding model. A request from your coding assistant goes to a code model.
Little can go wrong here. A text model cannot read an image no matter how capable it is, so there is no quality trade-off to tune.
2. Route by difficulty or cost
Here the router guesses how hard a request is and sends easy ones to a small model. The guess can come from simple signals (prompt length, which app sent it, keywords like "analyze" or "prove") or from a trained classifier.
This is where the savings are, and also where the risk is. A misrouted hard question gets a confident, wrong answer from the small model, and nothing in your logs flags it.
A variant is the cascade: try the small model first, check the answer, and escalate to a larger model only if the check fails. The FrugalGPT paper studied this approach and reports matching GPT-4 quality with up to 98% lower cost on its benchmarks. Your numbers will differ, but the pattern is sound when you have a cheap way to check an answer, such as validating JSON against a schema.
3. Fallback on failure
When a model returns an error, times out or rejects a request because it is longer than its context window, the router retries on a different model. This is about reliability rather than cost. A fallback chain might be: primary 14B model, then a second replica, then a larger model with a longer context window.
Set a timeout for each step. Without one, a fallback chain can turn one slow request into three slow requests.
4. Route by latency
Interactive features such as autocomplete or chat need a fast first token. Batch jobs such as overnight document summaries do not. Routing by latency sends interactive traffic to the fastest available model and lets background work queue on a larger one.
Building a simple rules-based router
Start with rules. They are predictable, cheap to run and easy to debug, and for many teams they are all the routing they ever need. Here is a minimal router in Python that calls OpenAI-compatible endpoints:
from openai import OpenAI
MODELS = {
"vision": OpenAI(base_url="https://llm.internal.example.com/vision/v1", api_key="..."),
"code": OpenAI(base_url="https://llm.internal.example.com/code/v1", api_key="..."),
"small": OpenAI(base_url="https://llm.internal.example.com/small/v1", api_key="..."),
"large": OpenAI(base_url="https://llm.internal.example.com/large/v1", api_key="..."),
}
HARD_HINTS = ("analyze", "compare", "step by step", "prove", "plan")
def pick_route(messages, app):
last = messages[-1]["content"]
if isinstance(last, list) and any(p.get("type") == "image_url" for p in last):
return "vision"
if app == "code-assistant":
return "code"
text = last if isinstance(last, str) else ""
if len(text) > 6000 or any(h in text.lower() for h in HARD_HINTS):
return "large"
return "small"
def chat(messages, app, fallback="large"):
route = pick_route(messages, app)
try:
return MODELS[route].chat.completions.create(
model=route, messages=messages, timeout=30
), route
except Exception:
if route == fallback:
raise
return MODELS[fallback].chat.completions.create(
model=fallback, messages=messages, timeout=60
), fallback
Three things make this usable in production. Log the chosen route with every request, so you can see the split. Make the rules configuration, not code, so you can change thresholds without a deploy. And let callers override the route for a request when they know better.
Because each model speaks the same API, adding a model to the pool is one new entry. The OpenAI-compatible API guide covers the differences between models that still need testing.
Learned routers and RouteLLM
Rules stop scaling when difficulty is the main signal, because "hard" is hard to write down. A learned router trains a small model to predict, from the prompt alone, whether a weak model's answer will be good enough.
RouteLLM is an open-source framework (Apache-2.0) for serving and evaluating routers of this kind. It routes between a strong and a weak model and ships several router types, including matrix factorization, a BERT classifier and a similarity-weighted ranking, trained on human preference data. The RouteLLM paper reports cutting costs by over 2 times in some cases without lowering response quality, and the project README cites up to 85% lower cost while keeping 95% of GPT-4's performance on MT Bench. It can also run as an OpenAI-compatible server, so it can sit in front of your existing endpoints.
Another approach is semantic routing: embed the request and compare it with example phrases for each route. Semantic Router (MIT) works this way and can run with a local embedding model, which suits private deployments.
Use a learned router when two conditions hold: you have enough traffic that the savings matter, and you can build an evaluation set to check it. Without the evaluation set, you are trading a visible cost for an invisible quality loss.
Evaluating a router and its failure modes
Treat the router like a model. Before launch:
- Collect 200 to 500 real requests, covering each app and request type.
- Run every request through both the small and the large model.
- Grade the pairs (human review, a rubric, or automated checks for structured output) and mark where the small model's answer was acceptable.
- Run the router over the same set and measure two numbers: the share of traffic it sends to the small model, and the share of those it got wrong.
A router that sends 70% of traffic to the small model with 2% bad answers is a different decision from one that sends 85% with 10% bad. The table makes the trade-off concrete for whoever owns the app.
The failure modes to watch after launch:
- Silent quality drop. Misrouted requests do not error. Track user feedback or downstream checks per route.
- Drift. A new feature changes what users ask, and the router's rules or training no longer match.
- Fallback storms. If the small model degrades, every request falls back to the large one and overloads it. Alert when the fallback rate rises.
- Added latency. A learned router adds a step to every request. Measure it; an embedding lookup is small, an LLM-based classifier is not.
You need per-route metrics to catch any of these, which is why routing and LLM observability belong together.
How a small model can take most of the traffic
The economics come from the gap between model sizes. Using the weight sizes from our guide to GPU memory for LLM inference, at 4-bit:
- 8B model: 8B x ~0.6 bytes = ~5 GB of weights
- 70B model: 70B x ~0.6 bytes = ~42 GB of weights
The small model fits on a much cheaper GPU and generates tokens several times faster on the same hardware. Now suppose your large-model pool is sized for 400 requests per minute at peak. If routing sends 75% of requests to the small model, the large pool only needs to handle:
400 x (1 - 0.75) = 100 requests per minute
That is a quarter of the large-model capacity you planned for, with the remaining 300 requests per minute served by hardware that costs far less per request. Self-hosted LLM cost walks through turning that into a hardware budget. The 75% figure is an assumption: your evaluation set tells you the real share for your traffic.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, includes smart model routing on Professional and Enterprise plans. It reads each request and sends it to the best available model for the task: vision, text, embeddings or code. That covers strategy 1 above without writing a router yourself, across models you deploy from its catalog inside your own infrastructure, each with an OpenAI-compatible endpoint.
It is not the right fit for every routing need. AI Refinery's routing is by task type. If you want difficulty-based routing between a small and a large text model, a cascade, or a learned router such as RouteLLM, you build that layer yourself and point it at the endpoints. And if a single model serves all your traffic well, you do not need routing at all yet.