Skip to main content
Version: next

Provider Adapters

Each LLM provider has a different HTTP API, authentication scheme, and wire format. Routerly bridges those differences through provider adapters — thin classes that translate a normalised internal request into the provider's specific format and translate the response back.

Adapters are selected automatically based on the provider field in a model's configuration.


Adapter Overview​

Provider IDClassProtocolNotes
openaiOpenAIAdapterOpenAI Chat CompletionsNative SDK; also handles /v1/responses
anthropicAnthropicAdapterAnthropic Messages APIFull message conversion, prompt caching
geminiGeminiAdapterOpenAI-compatible endpointUses OpenAI SDK pointed at Google's OpenAI-compatible base URL
ollamaOllamaAdapterOpenAI-compatible endpointUses OpenAI SDK pointed at local Ollama host
customCustomAdapterOpenAI-compatible endpointAny endpoint that speaks /v1/chat/completions
anthropic-webAnthropicWebAdapterUnofficial, session cookieUse at your own risk — see below
openai-webOpenAIWebAdapterUnofficial, session cookieUse at your own risk — see below

OpenAI Adapter​

Uses the official openai Node.js SDK.

Model ID resolution — If the registered model ID contains a slash (e.g. openai/gpt-4o), the adapter strips the prefix and sends only the part after the slash (gpt-4o) to the provider. This lets you namespace model IDs within Routerly without confusing OpenAI.

Endpoint override — If endpoint is set in the model config, the adapter uses it instead of https://api.openai.com/v1. This lets you point to Azure OpenAI, local OpenAI proxies, or compatible services.

Streaming — Uses the SDK's native async iterator. Chunks are forwarded to the client as Server-Sent Events (SSE) as they arrive.

// Example model config
{
"id": "gpt-5-mini",
"provider": "openai",
"apiKey": "<encrypted>",
"endpoint": null
}

Anthropic Adapter​

Uses the official @anthropic-ai/sdk Node.js SDK.

Message format conversion — The Anthropic Messages API differs from OpenAI's Chat Completions format in several ways. The adapter handles all conversions automatically:

OpenAI formatAnthropic format
messages[].role = "tool"Converted to role: "user" with a tool_result content block
messages[].role = "assistant" with tool_callsContent blocks of type tool_use
Consecutive tool result messagesMerged into a single user message (Anthropic requirement)
content as an array of content partsMapped block-by-block, preserving cache_control fields
system message in messages[]Extracted and passed as Anthropic's top-level system field
image_url content parts (data URI)Converted to Anthropic base64 image blocks
image_url content parts (URL)Converted to Anthropic URL image source

Prompt caching — The adapter preserves any cache_control fields present in message content parts, enabling Anthropic's prompt caching feature to work end-to-end.

Streaming — Uses the SDK's streaming API. SSE chunks are translated back to OpenAI-compatible format for /v1/chat/completions requests, or forwarded as-is for /v1/messages requests.

// Example model config
{
"id": "claude-haiku-4-5",
"provider": "anthropic",
"apiKey": "<encrypted>",
"endpoint": null
}

Gemini Adapter​

Uses the openai SDK pointed at Google's OpenAI-compatible base URL (https://generativelanguage.googleapis.com/v1beta/openai). No format conversion is needed — Gemini's compatibility layer handles it.

// Example model config
{
"id": "gemini-2.5-flash",
"provider": "gemini",
"apiKey": "<encrypted>"
}

Ollama Adapter​

Uses the openai SDK pointed at the Ollama host. No API key is required (Ollama has no auth by default).

Set the baseUrl field in the model config to your Ollama host:

// Example model config
{
"id": "llama3",
"provider": "ollama",
"endpoint": "http://localhost:11434/v1",
"apiKey": null
}

If you run Ollama on a different machine, change the endpoint to match. Routerly treats Ollama models as zero-cost (inputPerMillion: 0, outputPerMillion: 0) by default unless you configure explicit pricing.


Custom Adapter​

For any provider that exposes an OpenAI-compatible /v1/chat/completions endpoint.

Wire-format transparency: chatCompletion() and streamCompletion() use raw fetch (not the OpenAI SDK) to guarantee 100% wire-format transparency. Some SDKs (like OpenAI) normalize or strip non-standard response fields (e.g. DeepSeek's reasoning_content) before returning; using raw fetch ensures such fields reach the client unmodified. The messages() method uses the SDK because it converts Anthropic→OpenAI format anyway, so SDK normalization is harmless there.

Required field: endpoint must be set to the provider's base URL.

// Example model config
{
"id": "my-local-llm",
"provider": "custom",
"endpoint": "http://192.168.1.50:8080/v1",
"apiKey": "optional-key-if-required"
}

This adapter works with LM Studio, llama.cpp server, vLLM, LocalAI, DeepSeek, and any other service that implements the OpenAI /v1/chat/completions interface.


Mistral, Cohere, xAI​

These providers use the OpenAI-compatible protocol. Register them using the custom adapter with the appropriate endpoint and apiKey:

ProviderEndpoint
Mistralhttps://api.mistral.ai/v1
Coherehttps://api.cohere.com/compatibility/v1
xAI (Grok)https://api.x.ai/v1
// Example: Mistral
{
"id": "mistral-large",
"provider": "custom",
"endpoint": "https://api.mistral.ai/v1",
"apiKey": "<encrypted>"
}

Web Session Adapters (anthropic-web, openai-web)​

⚠️ Unofficial, use at your own risk. These adapters authenticate with a browser session cookie instead of an official API key, replaying the same internal endpoints the provider's own web/chat interface uses. This is not an officially supported integration path. It relies on undocumented behaviour that can change or break at any time, and using it may violate the provider's Terms of Service. Evaluate that risk yourself before using this feature.

Both adapters are always available: creating a connection with providerId: "anthropic-web" or providerId: "openai-web" needs no extra step. The dashboard form keeps the unofficial-provider warning.

Credentials — instead of an API key, the connection's encrypted credentials hold the session cookie value copied from a logged-in browser session:

Provider IDFieldWhere to find it
anthropic-webcookieEncThe sessionKey cookie (DevTools → Application → Cookies)
openai-webcookieEncThe session's bearer access token
openai-webcfClearanceEnc (optional)The cf_clearance cookie, if Cloudflare challenges the session

Credentials are encrypted at rest (encryptCredential/decryptCredential, AES-256-GCM) and decrypted only in memory, immediately before a request is built.

No refresh flow — unlike the OAuth adapters (anthropic-oauth, openai-oauth), there is no token endpoint to refresh against. A browser session cookie simply expires; when it does, the provider's own authentication error is returned to the caller unmodified.

Wire-format transparency — once the session cookie is resolved, request and response bodies are built and parsed exactly as before; only the credential lookup is affected.


Timeout Handling​

Two separate timeouts apply. The model timeout field (ms, default 60000) is the total-duration cap of one upstream call, for streaming and non-streaming requests alike; the SDK clients never retry on their own (maxRetries: 0), so the cap is not multiplied — retry and failover are Routerly's job. The router timeoutMs (default 2000 ms, 0 disables it) is a time-to-first-token budget for streaming requests only. A timed-out attempt fails over to the next candidate and is recorded as outcome: "timeout" in usage records; the health policy penalises the model accordingly. When every candidate fails before any byte is sent, streaming and non-streaming requests both get HTTP 503 (server_error).


Endpoint override for cloud providers​

azure-openai, bedrock and vertex call their cloud host by default. When the connection has an endpoint, it replaces the host (Azure: {endpoint}/openai/deployments/{deployment}, Bedrock: {endpoint}/model/{id}/converse[-stream], Vertex: {endpoint}/v1beta1/projects/…). Leave it empty for production. It exists for gateways and for the official provider mocks.