How one model name keeps answering when a provider does not
Apps call rch-advanced, not a vendor's model id. Here is what happens behind that name when the first key fails, and why the app never notices.
5 min readREM CloudHub team
AI gatewayReliability
Most teams wire an AI provider straight into their code: a key in an environment variable, a model id in a constant. It works until the provider has a bad afternoon, a key hits its rate limit, or someone wants to try another vendor. Then the change is a code change, a review and a redeploy.
REM CloudHub puts a name in between.
The name is the contract
An app calls a model name that belongs to your platform, such as rch-fast or rch-advanced. The name has a
display name, a plain description and a context window. It does not say which provider serves it, and the app does
not need to know.
curl "$RCH_AI_URL/v1/chat/completions" \
-H "Authorization: Bearer $RCH_AI_KEY" \
-d '{"model": "rch-advanced", "messages": [{"role": "user", "content": "Summarise this invoice"}]}'
Routes, in order
Behind each name sits a list of routes. A route is one provider key from the platform pool, the upstream model on that provider, and an order from 1 to 20.
| Order | Route | What happens |
|---|---|---|
| 1 | Key A, upstream model X | Tried first |
| 2 | Key B, upstream model Y | Tried when order 1 fails or is cooling down |
| 2 | Key C, upstream model Y | Shares the load with key B |
| 3 | Key D, upstream model Z | Last resort |
The gateway sends a call to the healthy routes with the lowest order. When that call fails, it retries the next order. A route that keeps failing is put in cooldown and left out until it recovers. Routes with the same order share the load.
What the app sees
One answer. The retry happens inside the gateway, so the app sees a slower response at worst, not an error.
Changing providers without a deploy
Because the routes are data, moving rch-advanced to a different provider is an edit on the routes board followed by
a sync. Unchanged routes are kept, changed ones are re-created, removed ones disappear. Every organisation that may use the
name gets the change on the next sync of its gateway team.
Health you can read
The models page shows, for each route, whether its deployment is in the gateway and the result of the latest background health check. Cooldowns are not shown yet, because the gateway engine has no admin route for them; that is on the list.
Where plans come in
An organisation's plan decides which model names it may call. A key asking for a name outside the plan is refused with a
clear error, ai_model_not_in_plan, instead of a surprise on the invoice.
Was this useful?