LLM router
LLM router
An LLM router decides which channel a request goes to. Tokenjoy finds the available channels for a model, splits traffic by priority and weight, and switches automatically when a channel fails. Your app keeps talking to one endpoint.
Pick a channel per model
One model can have several channels. On each request the gateway distributes traffic by priority and weight, so changing channels needs no application change.
No interruption when something breaks
- Failover: switch to another available channel as you configured
- Retries: retry on upstream errors by policy, with status and errors on record
- Rate limits: per API key request rate and burst caps
Governance beyond routing
Before routing, the gateway checks the API key, model permissions and quota. After routing, the caller, model, token usage and cost of each call are logged. Filter the call log by channel to see what each channel consumes.
How to connect
It is OpenAI- and Anthropic-compatible: set base_url to https://app.tokenjoy.me/v1 and use an API key issued by the gateway.
FAQ
What is the difference between an LLM router and an LLM gateway?
A router decides which channel handles a request. A gateway also covers authentication, access control, quota and audit. Tokenjoy provides both.
Do requests fail when a channel goes down?
The gateway switches to another available channel as you configured and retries by policy. Whether it can switch depends on how many channels the model has.