Skip to main content
Version: 0.6.0

Routing

This page covers routing for a router-kind Router: Routerly selects which model to use for each request by running a configurable stack of routing policies. Policies are applied in priority order; each policy can score, filter, or directly pick a model from the candidate set.

A passthrough-kind Router does not use this policy layer at all. An orchestrator-kind Router uses a restricted subset of it, scored against its candidate Routers instead of models — see Orchestrators and Passthrough Routers below.

Benchmarks

Reproducible routing benchmarks — latency overhead, cost savings, and failover behaviour — are published at github.com/Inebrio/routerly-benchmark.


How Routing Works​

  1. The Router's configured models are loaded as the candidate set.
  2. Policies run in the order they appear in the routing configuration.
  3. Each policy either filters some models out or scores them. At the end, the model with the highest combined score is selected.
  4. If no model passes all filters, Routerly returns a 503 error with a descriptive message.

Positional Scoring​

Each model's position in the routing list contributes a base score:

weight = total_models - index

So a model at position 0 gets weight = N, the one at position 1 gets weight = N-1, etc. This creates a natural preference order even when no other scoring policies are active.


Available Policies​

cheapest​

Selects the model with the lowest estimated cost for the current request. Estimation is based on registered pricing and the input token count. Output tokens are estimated at a configurable multiplier.

Use when: cost control is the primary concern.

health​

Filters out models that have had a high error rate in the recent window, or that failed the last health check. Keeps Routerly routing away from degraded providers automatically.

Use when: you want automatic failover.

performance​

Scores models by their recent p95 latency. Faster models receive higher scores.

Use when: response time matters more than cost.

capability​

Filters models by required capabilities (vision, functionCalling, thinking, json). Only models that have all required capabilities remain as candidates.

Use when: the request requires a specific capability (e.g. image input).

context​

Filters out models whose context window is smaller than the current request's estimated token count.

Use when: you send long documents or long conversations, and some of your models have smaller context windows.

llm​

Uses a separate LLM call to decide which model to route to, based on request content. This policy is experimental and introduces an extra API call per request.

Config options:

OptionTypeDefaultDescription
thinkingbooleanfalseIf true and the routing model supports extended thinking (e.g. Claude with thinking capability), the routing call uses it for more accurate decisions. Warning: this increases routing latency significantly.

Routing model validation and visibility. Every routing model id (router-level routingModelId, fallbackRoutingModelIds, and the llm policy routingModelId / fallbackModelIds) must exist in the model registry: creating a router or routing profile, or adding such an id on edit, returns HTTP 400 naming the unknown id (ids already stored stay editable). At request time an unusable routing model is never silent: when a fallback routing model serves, the trace records llm-policy:fallback (primary, model used, skip reasons); when every policy fails, the trace records router:policies-failed, the recap lists failedPolicies and the service logs a warning. Candidates are then ranked with neutral scores, as before.

Use when: you want dynamic model selection based on request semantics.

rate-limit​

Filters out models that are currently rate-limited (i.e. received a 429 response recently). The cooldown period is configurable per model.

Use when: your usage volume can hit provider rate limits.

fairness​

Distributes requests across models to balance load, or ensures that cheaper models are only used up to a configured share of traffic.

Use when: you have multiple capable models and want to spread load.

budget-remaining​

Scores models by how much of their associated budget is still available. Models with more remaining budget get higher scores.

Use when: you have per-model spending limits and want Routerly to naturally prefer models with headroom.

semantic-intent​

Classifies each incoming request by semantic intent using embeddings, then restricts the candidate pool to the models you have mapped to that intent.

How it works:

  1. You define intents — each intent has a name, a list of example phrases that represent it, and the target models that should handle requests of that type.
  2. When a request arrives, Routerly embeds the user message and compares it against the centroid of each intent's examples using cosine similarity.
  3. Based on the best match score and the gap between the top two intents, the policy produces one of three outcomes:
OutcomeConditionEffect
ConfidentTop score ≥ threshold and margin ≥ ambiguity gapHard-filters candidates to the matched intent's model pool
AmbiguousTop score ≥ threshold but gap is too smallMerges the top-2 intent pools
UnknownTop score below thresholdNo filtering — all candidates pass through

Configuration:

OptionDefaultDescription
embedding_provider(required)openai or ollama
embedding_model(required)Model ID to use for embedding (must have the embedding capability)
absolute_threshold0.60Minimum cosine similarity score to consider a match
ambiguity_threshold0.08Minimum margin between top-2 scores to consider a match confident

Use when: you have distinct request categories that should always be routed to specific models (e.g. billing questions → a fine-tuned model, code requests → a coding model).

Intent centroids are cached

Embeddings for intent examples are computed once and cached in memory for 1 hour. Changing an intent's examples automatically invalidates the cache.


Orchestrators and Passthrough Routers​

Two other Router kinds sit alongside the policy-driven router kind covered above. Both are created the same way (routerly router create --kind ... or the dashboard's Routers page); see Concepts: Architecture for the full request-lifecycle detail.

Orchestrator​

An orchestrator-kind Router targets other Routers, not models. Instead of a model list and policies, it carries a priority-ordered list of candidate Routers (--candidate <routerId>, repeatable — order of repetition sets priority, first = highest), with optional per-candidate usage limits. A request to an Orchestrator is scored across its candidates and ties break by list position, then forwarded to the picked Router, which then applies its own routing (or is itself a Passthrough or another Orchestrator). The management API only ever returns a candidate's resolved name and its position in the list, never its internal model list or policies — an Orchestrator's candidates are opaque to the client the same way a router-kind Router's provider configuration is.

An Orchestrator can also carry its own routing policies, scored against its candidate Routers rather than models. Only the policies whose scoring is Router-attribute-based, not model-attribute-based, are supported: health, rate-limit, fairness, performance, and budget-remaining. The other 6 (cheapest, context, capability, llm, semantic-intent, model-preference) pick among a Router's own models, which an Orchestrator has none of.

Passthrough​

A passthrough-kind Router does not select a model at all. It is reached at /passthrough/<slug>/*, where slug is derived automatically from the Router's name at creation (disambiguated with a numeric suffix if another Passthrough Router already slugs to the same value) — there is no --slug flag or manual path field; the slug never changes afterward, even if the Router is renamed. Because a Passthrough is reached by slug and never by name, its name only needs to be unique among other Passthroughs — it can share a name with an existing Router or Orchestrator. Router and Orchestrator names remain unique among themselves. Its models list always carries a fixed pass-through entry, and two things do not apply to a request that resolves to it, unlike every other Router kind:

  • No Routerly authentication. The path carries no Router token; the client's own Authorization/x-api-key header is forwarded unchanged to the real api.openai.com or api.anthropic.com.
  • No budgets or usage limits. Cost is unknown for traffic Routerly never priced, so budget and limit configuration do not apply to this entry.

Guardrails and PII policies configured on the Router still run regardless.

A Passthrough Router's models list can also carry real target models alongside the fixed pass-through entry. A request resolving to one of those is scored, authenticated with a Routerly-issued token, and budgeted/metered exactly like a router-kind Router's model — both exemptions above apply only to the pass-through entry itself. Which of the two a request gets by default depends on the pass-through entry's position in the list: see Concepts: Architecture for the full ordering rule and the fallback's forwarded-credential caveat.


Routing Profiles​

A routing profile bundles the policy layer described above with two further settings into one reusable, named unit:

  • a selector: how the final model is picked among the ranked candidates
  • a fallback strategy: what happens when the picked model fails

A Router either keeps its own inline policies (configured on its Routing tab), or is assigned a shared profile instead. Routing is one of three profile kinds, alongside optimizer and security profiles; see Dashboard: Profiles and API: Profiles for how to manage and assign them.

Built-in Profiles​

Routerly ships 4 built-in routing profiles, read-only and always available. Each can be cloned into an editable, user-owned copy.

ProfilePoliciesSelectorFallbackOptimizes for
Autohealth, performance, cheapest, capabilityargmaxnext-bestA general-purpose mix of speed, cost, and reliability
Cheapcheapest, budget-remaining, healthcheapestnext-bestLowest cost per request
Fastperformance, healthlowest-latencyretry-after-cooldownLowest response time
Codingcapability, model-preference, performance, healthargmaxnext-bestCapable, developer-preferred models for code tasks

Two earlier presets, balanced and offline, are no longer offered. Routers still pointing at balanced are migrated to the byte-identical auto; Routers on offline keep resolving it unchanged, but it cannot be picked or cloned any more.

Selectors​

The selector picks one model from the candidates the policy layer scored (and did not filter out):

SelectorBehaviour
argmaxPicks the highest-scored candidate. Ties within a small tolerance are broken by weighted-random among the tied group.
weighted-randomPicks one candidate at random, with probability proportional to its score.
round-robinCycles through candidates in a deterministic order, one per request, evenly distributing load regardless of score.
cheapestPicks the lowest-cost candidate; ties break by higher score.
lowest-latencyPicks the candidate with the lowest recent observed latency; candidates with no latency data sort last.

The selector is not editable from the dashboard or the CLI: a profile created there uses argmax. The built-in profiles keep the selectors listed above, and the API accepts any of them.

Fallback Strategies​

StrategyBehaviour
next-bestOn failure, try the next-highest-ranked remaining candidate.
retry-after-cooldownOn failure, retry the same model after a cooldown period instead of moving to the next candidate.
abortOn failure, stop immediately with no retry.
Not yet wired into live retries

The fallback strategy is a field stored on the profile and settable through the API, but it does not yet change what actually happens when a request fails at runtime: the reverse-proxy retry loop does not read it yet. Treat it as configuration staged for a future release, not a currently active behaviour. That is why neither the dashboard nor the CLI exposes it.


Configuring Routing​

  1. Open the Router → Routing tab
  2. Drag a policy from the left panel into the active list
  3. Configure the policy's parameters in the settings panel on the right
  4. Drag to reorder — policies at the top have higher priority
  5. Add target models below the policies

CLI​

# Add a target model to a Router, with a hint for the routing model
routerly router model add "My App" gpt-5-mini --prompt "Short factual answers"

# List the target models of a Router
routerly router model list "My App"

# Remove a model
routerly router model remove "My App" gpt-5-mini

Budgets are not set here: they live on the model (routerly model edit <id> --monthly-budget) or on a Router token. See Budgets and Limits.


Example: Cost-first with Health Failover​

This configuration tries the cheapest available healthy model:

Policies (in order):
1. health — remove unhealthy models
2. cheapest — prefer lowest cost

Models (in priority order):
1. gpt-5-nano
2. gpt-5-mini
3. gpt-5

If gpt-5-nano is unhealthy, health removes it from candidates, and cheapest picks gpt-5-mini.


Example: Capability Routing​

Route vision requests to a capable model while serving text-only requests with a cheaper model:

Policies:
1. capability — requires: vision (if the request includes an image)

Models:
1. gpt-4.1 (has vision)
2. gpt-5-nano (no vision)

Text-only requests → both are candidates → positional scoring picks gpt-4.1. Vision requests → gpt-5-nano is filtered out → gpt-4.1 is used. If no vision model is available, Routerly returns 503.