Skip to content

How one model name keeps answering when a provider does not

Apps call rch-advanced, not a vendor's model id. Here is what happens behind that name when the first key fails, and why the app never notices.

5 min readREM CloudHub team

AI gatewayReliability

Most teams wire an AI provider straight into their code: a key in an environment variable, a model id in a constant. It works until the provider has a bad afternoon, a key hits its rate limit, or someone wants to try another vendor. Then the change is a code change, a review and a redeploy.

REM CloudHub puts a name in between.

The name is the contract

An app calls a model name that belongs to your platform, such as rch-fast or rch-advanced. The name has a display name, a plain description and a context window. It does not say which provider serves it, and the app does not need to know.

curl "$RCH_AI_URL/v1/chat/completions" \
  -H "Authorization: Bearer $RCH_AI_KEY" \
  -d '{"model": "rch-advanced", "messages": [{"role": "user", "content": "Summarise this invoice"}]}'

Routes, in order

Behind each name sits a list of routes. A route is one provider key from the platform pool, the upstream model on that provider, and an order from 1 to 20.

Order Route What happens
1 Key A, upstream model X Tried first
2 Key B, upstream model Y Tried when order 1 fails or is cooling down
2 Key C, upstream model Y Shares the load with key B
3 Key D, upstream model Z Last resort

The gateway sends a call to the healthy routes with the lowest order. When that call fails, it retries the next order. A route that keeps failing is put in cooldown and left out until it recovers. Routes with the same order share the load.

What the app sees

One answer. The retry happens inside the gateway, so the app sees a slower response at worst, not an error.

Changing providers without a deploy

Because the routes are data, moving rch-advanced to a different provider is an edit on the routes board followed by a sync. Unchanged routes are kept, changed ones are re-created, removed ones disappear. Every organisation that may use the name gets the change on the next sync of its gateway team.

Health you can read

The models page shows, for each route, whether its deployment is in the gateway and the result of the latest background health check. Cooldowns are not shown yet, because the gateway engine has no admin route for them; that is on the list.

Where plans come in

An organisation's plan decides which model names it may call. A key asking for a name outside the plan is refused with a clear error, ai_model_not_in_plan, instead of a surprise on the invoice.

Was this useful?

Subscribe to the newsletter

New posts and product news from REM CloudHub, a few times a month. Nothing else.