Routing
This page covers routing for a router-kind Router: Routerly
selects which model to use for each request by running a configurable stack
of routing policies. Policies are applied in priority order; each policy
can score, filter, or directly pick a model from the candidate set.
A passthrough-kind Router does not use this policy layer at all. An
orchestrator-kind Router uses a restricted subset of it, scored against its
candidate Routers instead of models — see Orchestrators and Passthrough
Routers below.
Reproducible routing benchmarks — latency overhead, cost savings, and failover behaviour — are published at github.com/Inebrio/routerly-benchmark.
How Routing Works
- The Router's configured models are loaded as the candidate set.
- Policies run in the order they appear in the routing configuration.
- Each policy either filters some models out or scores them. At the end, the model with the highest combined score is selected.
- If no model passes all filters, Routerly returns a
503error with a descriptive message.
Positional Scoring
Each model's position in the routing list contributes a base score:
weight = total_models - index
So a model at position 0 gets weight = N, the one at position 1 gets weight = N-1, etc. This creates a natural preference order even when no other scoring policies are active.
Available Policies
cheapest
Selects the model with the lowest estimated cost for the current request. Estimation is based on registered pricing and the input token count. Output tokens are estimated at a configurable multiplier.
Use when: cost control is the primary concern.
health
Filters out models that have had a high error rate in the recent window, or that failed the last health check. Keeps Routerly routing away from degraded providers automatically.
Use when: you want automatic failover.
performance
Scores models by their recent p95 latency. Faster models receive higher scores.
Use when: response time matters more than cost.
capability
Filters models by required capabilities (vision, functionCalling, thinking, json). Only models that have all required capabilities remain as candidates.
Use when: the request requires a specific capability (e.g. image input).
context
Filters out models whose context window is smaller than the current request's estimated token count.
Use when: you send long documents or long conversations, and some of your models have smaller context windows.
llm
Uses a separate LLM call to decide which model to route to, based on request content. This policy is experimental and introduces an extra API call per request.
Config options:
| Option | Type | Default | Description |
|---|---|---|---|
thinking | boolean | false | If true and the routing model supports extended thinking (e.g. Claude with thinking capability), the routing call uses it for more accurate decisions. Warning: this increases routing latency significantly. |
Routing model validation and visibility. Every routing model id (router-level routingModelId, fallbackRoutingModelIds, and the llm policy routingModelId / fallbackModelIds) must exist in the model registry: creating a router or routing profile, or adding such an id on edit, returns HTTP 400 naming the unknown id (ids already stored stay editable). At request time an unusable routing model is never silent: when a fallback routing model serves, the trace records llm-policy:fallback (primary, model used, skip reasons); when every policy fails, the trace records router:policies-failed, the recap lists failedPolicies and the service logs a warning. Candidates are then ranked with neutral scores, as before.
Use when: you want dynamic model selection based on request semantics.
rate-limit
Filters out models that are currently rate-limited (i.e. received a 429 response recently). The cooldown period is configurable per model.
Use when: your usage volume can hit provider rate limits.
fairness
Distributes requests across models to balance load, or ensures that cheaper models are only used up to a configured share of traffic.
Use when: you have multiple capable models and want to spread load.
budget-remaining
Scores models by how much of their associated budget is still available. Models with more remaining budget get higher scores.
Use when: you have per-model spending limits and want Routerly to naturally prefer models with headroom.
semantic-intent
Classifies each incoming request by semantic intent using embeddings, then restricts the candidate pool to the models you have mapped to that intent.
How it works:
- You define intents — each intent has a name, a list of example phrases that represent it, and the target models that should handle requests of that type.
- When a request arrives, Routerly embeds the user message and compares it against the centroid of each intent's examples using cosine similarity.
- Based on the best match score and the gap between the top two intents, the policy produces one of three outcomes:
| Outcome | Condition | Effect |
|---|---|---|
| Confident | Top score ≥ threshold and margin ≥ ambiguity gap | Hard-filters candidates to the matched intent's model pool |
| Ambiguous | Top score ≥ threshold but gap is too small | Merges the top-2 intent pools |
| Unknown | Top score below threshold | No filtering — all candidates pass through |
Configuration:
| Option | Default | Description |
|---|---|---|
embedding_provider | (required) | openai or ollama |
embedding_model | (required) | Model ID to use for embedding (must have the embedding capability) |
absolute_threshold | 0.60 | Minimum cosine similarity score to consider a match |
ambiguity_threshold | 0.08 | Minimum margin between top-2 scores to consider a match confident |
Use when: you have distinct request categories that should always be routed to specific models (e.g. billing questions → a fine-tuned model, code requests → a coding model).
Embeddings for intent examples are computed once and cached in memory for 1 hour. Changing an intent's examples automatically invalidates the cache.
Orchestrators and Passthrough Routers
Two other Router kinds sit alongside the policy-driven router kind covered
above. Both are created the same way (routerly router create --kind ... or
the dashboard's Routers page); see Concepts: Architecture
for the full request-lifecycle detail.
Orchestrator
An orchestrator-kind Router targets other Routers, not models. Instead
of a model list and policies, it carries a priority-ordered list of candidate
Routers (--candidate <routerId>, repeatable — order of repetition sets
priority, first = highest), with optional per-candidate usage limits. A
request to an Orchestrator is scored across its candidates and ties break by
list position, then forwarded to the picked Router, which then applies its
own routing (or is itself a Passthrough or another Orchestrator).
The management API only ever returns a candidate's resolved name and its
position in the list, never its internal model list or policies — an
Orchestrator's candidates are opaque to the client the same way a
router-kind Router's provider configuration is.
An Orchestrator can also carry its own routing policies, scored against its
candidate Routers rather than models. Only the policies whose scoring is
Router-attribute-based, not model-attribute-based, are supported:
health, rate-limit, fairness,
performance, and budget-remaining.
The other 6 (cheapest, context, capability, llm, semantic-intent,
model-preference) pick among a Router's own models, which an Orchestrator
has none of.
Passthrough
A passthrough-kind Router does not select a model at all. It is reached at
/passthrough/<slug>/*, where slug is derived automatically from the
Router's name at creation (disambiguated with a numeric suffix if another
Passthrough Router already slugs to the same value) — there is no --slug
flag or manual path field; the slug never changes afterward, even if the
Router is renamed. Because a Passthrough is reached by slug and never by
name, its name only needs to be unique among other Passthroughs — it can
share a name with an existing Router or Orchestrator. Router and Orchestrator
names remain unique among themselves. Its models list always carries a
fixed pass-through entry, and two things do not apply to a request that
resolves to it, unlike every other Router kind:
- No Routerly authentication. The path carries no Router token; the
client's own
Authorization/x-api-keyheader is forwarded unchanged to the realapi.openai.comorapi.anthropic.com. - No budgets or usage limits. Cost is unknown for traffic Routerly never priced, so budget and limit configuration do not apply to this entry.
Guardrails and PII policies configured on the Router still run regardless.
A Passthrough Router's models list can also carry real target models
alongside the fixed pass-through entry. A request resolving to one of those
is scored, authenticated with a Routerly-issued token, and budgeted/metered
exactly like a router-kind Router's model — both exemptions above apply
only to the pass-through entry itself. Which of the two a request gets by
default depends on the pass-through entry's position in the list: see
Concepts: Architecture
for the full ordering rule and the fallback's forwarded-credential caveat.
Routing Profiles
A routing profile bundles the policy layer described above with two further settings into one reusable, named unit:
- a selector: how the final model is picked among the ranked candidates
- a fallback strategy: what happens when the picked model fails
A Router either keeps its own inline policies (configured on its Routing tab), or is assigned a shared profile instead. Routing is one of three profile kinds, alongside optimizer and security profiles; see Dashboard: Profiles and API: Profiles for how to manage and assign them.
Built-in Profiles
Routerly ships 4 built-in routing profiles, read-only and always available. Each can be cloned into an editable, user-owned copy.
| Profile | Policies | Selector | Fallback | Optimizes for |
|---|---|---|---|---|
| Auto | health, performance, cheapest, capability | argmax | next-best | A general-purpose mix of speed, cost, and reliability |
| Cheap | cheapest, budget-remaining, health | cheapest | next-best | Lowest cost per request |
| Fast | performance, health | lowest-latency | retry-after-cooldown | Lowest response time |
| Coding | capability, model-preference, performance, health | argmax | next-best | Capable, developer-preferred models for code tasks |
Two earlier presets, balanced and offline, are no longer offered. Routers
still pointing at balanced are migrated to the byte-identical auto;
Routers on offline keep resolving it unchanged, but it cannot be picked or
cloned any more.
Selectors
The selector picks one model from the candidates the policy layer scored (and did not filter out):
| Selector | Behaviour |
|---|---|
argmax | Picks the highest-scored candidate. Ties within a small tolerance are broken by weighted-random among the tied group. |
weighted-random | Picks one candidate at random, with probability proportional to its score. |
round-robin | Cycles through candidates in a deterministic order, one per request, evenly distributing load regardless of score. |
cheapest | Picks the lowest-cost candidate; ties break by higher score. |
lowest-latency | Picks the candidate with the lowest recent observed latency; candidates with no latency data sort last. |
The selector is not editable from the dashboard or the CLI: a profile created
there uses argmax. The built-in profiles keep the selectors listed above,
and the API accepts any of them.
Fallback Strategies
| Strategy | Behaviour |
|---|---|
next-best | On failure, try the next-highest-ranked remaining candidate. |
retry-after-cooldown | On failure, retry the same model after a cooldown period instead of moving to the next candidate. |
abort | On failure, stop immediately with no retry. |
The fallback strategy is a field stored on the profile and settable through the API, but it does not yet change what actually happens when a request fails at runtime: the reverse-proxy retry loop does not read it yet. Treat it as configuration staged for a future release, not a currently active behaviour. That is why neither the dashboard nor the CLI exposes it.
Configuring Routing
Dashboard (recommended)
- Open the Router → Routing tab
- Drag a policy from the left panel into the active list
- Configure the policy's parameters in the settings panel on the right
- Drag to reorder — policies at the top have higher priority
- Add target models below the policies
CLI
# Add a target model to a Router, with a hint for the routing model
routerly router model add "My App" gpt-5-mini --prompt "Short factual answers"
# List the target models of a Router
routerly router model list "My App"
# Remove a model
routerly router model remove "My App" gpt-5-mini
Budgets are not set here: they live on the model (routerly model edit <id> --monthly-budget) or on a Router token. See Budgets and Limits.
Example: Cost-first with Health Failover
This configuration tries the cheapest available healthy model:
Policies (in order):
1. health — remove unhealthy models
2. cheapest — prefer lowest cost
Models (in priority order):
1. gpt-5-nano
2. gpt-5-mini
3. gpt-5
If gpt-5-nano is unhealthy, health removes it from candidates, and cheapest picks gpt-5-mini.
Example: Capability Routing
Route vision requests to a capable model while serving text-only requests with a cheaper model:
Policies:
1. capability — requires: vision (if the request includes an image)
Models:
1. gpt-4.1 (has vision)
2. gpt-5-nano (no vision)
Text-only requests → both are candidates → positional scoring picks gpt-4.1. Vision requests → gpt-5-nano is filtered out → gpt-4.1 is used. If no vision model is available, Routerly returns 503.