Provider Failover

Most models in the Gloo AI catalog can be served from more than one provider. When the provider currently serving a model is failing, Gloo sends your request to another provider that serves the same model. This is provider failover, and it runs on every chat and responses endpoint with no configuration. Failover changes who serves your request, never what you get: the model in the response is the model you asked for. (Model deprecation is separate from failover: a deprecated model ID with a replacement_model is served by that replacement, and the response’s model names the replacement. See Model Lifecycle.) It is separate from auto-routing and model_family, which choose a model for you on the guarded Completions endpoint.

When a request fails over

Failover handles failures that belong to the provider, not to your request:

• Provider errors and outages. A timeout, a lost connection, a provider server error, an empty response, or the provider refusing the request because it is over capacity or rate-limited. Gloo retries the request on another provider serving the same model.
• A provider that can’t take the request. When a provider can’t take a request for reasons on its side, including as a stream opens, Gloo sends the request to another provider that serves the same model. If no provider can take it, you receive a 503 with retryable: true: 2002 / PROVIDER_UNAVAILABLE when the providers are unavailable (retry with backoff), or 3003 / SERVICE_UNAVAILABLE with a Retry-After header when the problem is on Gloo’s side (wait, then retry). See Every provider unavailable.
• A provider that keeps failing. Gloo tracks each provider’s health per model. When a provider starts failing for a model, new requests for that model go to a healthy provider instead of the failing one, which returns to service automatically once it recovers.
• A stream that fails before the answer starts. If a streamed request fails before any text reaches you, Gloo retries it, on another provider when needed, and the answer starts from the beginning. You see the stream start later, not an error.
• A stream cut off mid-answer. For models that support it, another provider continues the answer from where the stream stopped, in the same response. For models that do not, the stream ends with an error event instead; see Handling Streaming Failures for how to continue it from your application.

Failover works within a time budget for each request. If no provider answers within it, you receive the error instead of waiting indefinitely. A request that uses the image_generation tool goes only to a provider that can run the tool. When none can, you get a 503 with 2002 / PROVIDER_UNAVAILABLE and retryable: true, rather than an answer from a provider that cannot draw images. See Image generation errors.

When a request does not fail over

Some failures would fail the same way on every provider, so Gloo returns them straight away instead of repeating the request elsewhere:

Reasoning models and the output limit

On reasoning models the output limit (max_tokens on Chat Completions, max_output_tokens on Responses) holds the thinking as well as the answer, so a limit that’s too small for the effort can be used up before the model writes any text. See Set your output limit for reasoning models for how Gloo applies the limit and how to size it. The provider answered correctly and would answer the same way again, so this is not retried or failed over. Gloo returns the provider’s answer:

• Chat Completions, streaming and non-streaming: the response ends with finish_reason: "length" and no text, and usage reports the tokens the model used.
• Responses (non-streaming, OpenAI reasoning models): HTTP 200 with status: "incomplete" and incomplete_details: {"reason": "max_output_tokens"}, plus a suggestion field that names the tokens used and the limit you sent.

Raise the limit or lower the reasoning effort, leaving generous headroom above the length of the answer you expect; the sizing table gives starting points.

Which endpoints fail over

What you observe

• The same model. The response carries the model you requested, whichever provider served it. A deprecated model ID that has a replacement is the exception: the response names the replacement that served it.
• Extra latency. A request that failed over waited for the failed attempt first.
• A colder prompt cache. Prompt caches are held by each provider, so a request served by a different provider does not read the cache built on the original one. Expect fewer cached tokens on requests that failed over, and on requests routed around an unhealthy provider, until the cache warms on the new one. See Prompt Caching.
• Stream headers from the first attempt. A stream’s response headers, such as X-Ignored-Params, are sent before the first chunk, so they describe the request as it started and aren’t updated if it then fails over.
• No charge for the failed attempt. An attempt that failed and was replaced by another provider is not billed.
• An error only when failover could not help. Errors you receive carry the standard error object: fault says whether the problem was your request, the provider, or Gloo, and retryable says whether the same request may succeed if you send it again.

What your application should do

1. Rely on failover for provider trouble. You don’t need to switch models or add a second provider yourself to ride out an outage.
2. Still retry retryable errors. When an error has retryable: true, retry: after the Retry-After delay when the response has one, otherwise with exponential backoff. See Error Handling and Retries. A later request benefits from Gloo having already routed around the failing provider. Do not retry errors with retryable: false unchanged, including 3001 / INTERNAL_ERROR.
3. Handle interrupted streams. On models where Gloo cannot continue a cut-off stream, the stream ends with an error event after partial output (a response.failed event on Direct Responses). A provider failure mid-stream arrives as a provider error code (for example 2014 / PROVIDER_STREAM_ERROR) with retryable: true; 3001 / INTERNAL_ERROR is an error on Gloo’s side with retryable: false. Follow Handling Streaming Failures.
4. Size output limits for reasoning models. Treat finish_reason: "length" or status: "incomplete" with no text as a sign to raise max_tokens or max_output_tokens, or to lower the reasoning effort, not as an outage.
5. Log trace_id. Include it, with the X-Request-Id response header on the direct endpoints, when you contact support about a failed request.

Next steps

For what each endpoint family runs around your request, see Endpoint Types. For how caching works on each provider, see Prompt Caching.