> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Lesson 9: Production-Ready

> Keep a Gloo app running with real users: handle rate limits and retries, understand provider failover and routing, and stay ahead of model deprecations.

## When Real Users Arrive

In development, every request works. In production, with real traffic, some won't. A provider has a bad minute. A request is too long. Your organization hits its spending limit. A model you depend on is retired.

None of this is unusual, and none of it has to break your app. This lesson covers three things that keep it running: handling errors and retries, knowing what failover does for you, and keeping up with model changes.

***

## Core Concepts

### Read the Error, Then Decide

Most errors from Gloo's AI endpoints carry an `error` object. Two of its fields tell you what to do:

* **`retryable`** says whether the same request may succeed if you send it again.
* **`fault`** says whose problem it was: your request (`client`), the model provider (`provider`) or Gloo (`internal`).

Some errors, such as content-policy refusals and validation errors, carry a `detail` object instead. On the direct endpoints, every error also has an `X-Gloo-Error-Code` header to branch on and a short `message` you can show your users, and most carry a `developer_message` for your logs. Don't match on the message text: its wording can change, and the codes don't.

The rules:

* **`retryable: false`**: don't send the same request again. Fix it, or show the user a message.
* **`retryable: true` with a `Retry-After` header**: wait that many seconds, then retry.
* **`retryable: true` without `Retry-After`**: retry with **exponential backoff** (wait longer after each failure) plus a little random **jitter**, and give up after a few tries.

### Not Every 429 Is a Rate Limit

A `429 Too Many Requests` can mean different things:

| Error | What it means | Retry? |
| :- | :- | :- |
| `2004 / RATE_LIMIT` (`rate_limit`) | The model's provider is limiting requests for now. Not your quota. | Yes, after `Retry-After`. In a stream that already started, there's no header: check `error.retryable` and back off |
| `1005 / SPENDING_LIMIT_EXCEEDED` | Your organization reached its spending limit | No, until the limit is raised |
| `1009 / MONTHLY_SPEND_LIMIT_REACHED` | Your organization reached its monthly API spend limit | No, until the month resets or support raises it |
| An endpoint-specific request quota | Some endpoints enforce their own quotas | Honor `Retry-After` when present |

Don't assume a `429` includes headers like `X-RateLimit-Remaining`. They aren't part of a universal Gloo contract.

### Failover Works for You

Most models on Gloo can be served by more than one provider. When the provider serving a model is failing, Gloo sends your request to another provider that serves **the same model**. This is **provider failover**, and it runs on every chat and Responses endpoint with no setup.

What you notice:

* **The same model.** The response names the model you asked for.
* **A little extra latency** on a request that failed over.
* **Fewer cached tokens** until the cache warms on the new provider.
* **No charge** for the failed attempt.

Failover doesn't cover every case. [Anthropic Messages](https://docs.gloo.com/api-guides/anthropic-messages.md) is served directly by Anthropic with no failover, and [Embeddings](https://docs.gloo.com/api-guides/embeddings.md) returns a retryable `503` instead. A model with only one provider has nowhere to fail over to. When no provider can take the request, you get a `503` with `retryable: true`. And errors that would fail the same way everywhere, such as a request that's too long, come straight back.

### Routing: Who Picks the Model

**Failover** changes who serves a model. **Routing** changes which model you get. On the guarded [Completions](https://docs.gloo.com/api-guides/completions-v2.md) endpoint you can opt in to routing:

* **AI Core Select** (`model_family`): name a provider family and Gloo picks a model within it.
* **AI Core** (`auto_routing: true`): Gloo analyzes each query and chooses a model for it.

Both can give you a different model from one call to the next. For production, tool calling and anything you need to reproduce, use **AI Select**: pin a `model`. It is our recommendation for most applications.

### Models Come and Go

Providers release new models and retire old ones. The model catalog, [`GET /platform/v2/models`](https://docs.gloo.com/api-reference/get-models/v2), is where you see both. For each model, three fields tell you about retirement:

* **`is_deprecated`**: `true` once the model is scheduled for retirement
* **`replacement_model`**: the model requests are routed to
* **`deprecation_note`**: a plain-language explanation, including any price change

For generation models, when `replacement_model` is set, requests to a deprecated ID go to that replacement, so your app keeps working while you migrate. The response's `model` names the replacement that served it. A replacement's price is normally capped at 1.3 times the old rate on input or output, and any exception is stated in `deprecation_note`.

A deprecated model is intended to stay listed for at least a week before it can be removed. Treat that as best effort: a provider can withdraw a model faster.

Embeddings are different. Vectors from different models don't mix, so changing an embedding model means re-embedding your stored content.

***

## Get Started

```bash theme={null}
pip install openai requests
```

### A Retry Wrapper

This helper retries only what the API says is retryable. It waits at least as long as `Retry-After` asks, and stops after a few attempts or when the total wait would exceed a budget. The OpenAI SDK retries some errors on its own by default, including ones marked not retryable, such as a spending limit. This example turns that off with `max_retries=0` so only these rules apply. Do the same in production.

The limits below are the budget for background work. When someone is waiting on the result, set `MAX_ATTEMPTS = 3` (two retries) and `MAX_TOTAL_WAIT` to a few seconds, so a long `Retry-After` fails fast instead of keeping them waiting.

```python Python theme={null}
import os
import random
import time
import openai
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GLOO_API_KEY"],
    base_url="https://platform.ai.gloo.com/ai/v2/direct",
    max_retries=0,
)

MAX_ATTEMPTS = 5  # at most 5 requests in total, including the first
MAX_TOTAL_WAIT = 60  # seconds this caller is willing to wait in all


def is_retryable(err):
    body = err.body if isinstance(err.body, dict) else {}
    error = body.get("error") if isinstance(body.get("error"), dict) else body
    return error.get("retryable") is True


def create_with_retries(**kwargs):
    waited = 0
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            return client.responses.create(**kwargs)
        except openai.APIStatusError as err:
            if not is_retryable(err) or attempt == MAX_ATTEMPTS:
                print("Request failed:", err.response.headers.get("X-Gloo-Error-Code"),
                      "request id:", err.response.headers.get("X-Request-Id"))
                raise
            retry_after = err.response.headers.get("Retry-After")
            if retry_after:
                wait = float(retry_after)
            else:
                wait = min(2 ** (attempt - 1), 30) + random.random()
            if waited + wait > MAX_TOTAL_WAIT:
                raise  # too long to wait now: defer the work instead
            time.sleep(wait)
            waited += wait


response = create_with_retries(
    model="gloo-anthropic-claude-sonnet-5.5",
    input="Give me one tip for welcoming first-time visitors.",
)
print(response.output_text)
```

For a live stream, use a smaller budget: two retries at most before any text appears, and a delay of a couple of seconds at most. [Handling Streaming Failures](https://docs.gloo.com/best-practices/completions-streaming-failures.md#retry-budgets-and-backoff) has recommended limits.

### Check Your Models for Deprecations

Run this on a schedule, such as daily, against the model IDs your app uses:

```python Python theme={null}
import requests

MY_MODELS = {"gloo-anthropic-claude-sonnet-5.5", "gloo-openai-gpt-5-mini"}

catalog = requests.get("https://platform.ai.gloo.com/platform/v2/models").json()["data"]
listed = {m["id"]: m for m in catalog}

for model_id in sorted(MY_MODELS):
    model = listed.get(model_id)
    if model is None:
        print(f"{model_id}: no longer in the catalog. Move to a listed model.")
    elif model.get("is_deprecated"):
        print(f"{model_id}: deprecated. Replacement: {model.get('replacement_model')}")
        print("  Note:", model.get("deprecation_note"))
```

When a model you use is flagged, read the note, test the replacement with your own prompts, and update your `model` ID.

***

## Good to Know

* **Log the request ID.** Keep `X-Request-Id` on the direct endpoints, and the `trace_id` in error bodies. Support can find the exact request with it.
* **`3001 / INTERNAL_ERROR` isn't retryable.** Don't resend it unchanged. Contact support with the request ID if it keeps happening.
* **A non-streaming request can time out.** `3007 / GATEWAY_DEADLINE` means it ran too long. Stream it or lower the output limit instead of retrying.
* **An empty answer may be the output limit.** If a reasoning model stops with `finish_reason: "length"` or `status: "incomplete"` and no text, it spent the limit thinking. Raise the limit; it isn't an outage.
* **Don't build on continued routing.** An ID you can still see in the catalog is supported. One that has been removed isn't, even if requests still go through for a while.

***

## Key Takeaway

Let the error tell you what to do: retry only when `retryable` is `true`, wait for `Retry-After` when it's there, and back off with jitter when it isn't. Rely on failover for provider trouble, pin your model in production, and watch `is_deprecated` in the catalog so model changes never surprise you.

> [!TIP]
>
> **Try it on Gloo:** [Provider Failover](https://docs.gloo.com/api-guides/provider-failover.md) lists what fails over and what your app should still handle, and [Model Lifecycle](https://docs.gloo.com/api-guides/model-lifecycle.md) shows how to detect deprecations from the API.

***

## Learn more

* [Rate Limits](https://docs.gloo.com/api-reference/general/limits.md): every kind of `429` and a retry example
* [Errors](https://docs.gloo.com/api-reference/general/errors.md): the error object, every code and the direct endpoint error contract
* [Provider Failover](https://docs.gloo.com/api-guides/provider-failover.md): when requests fail over and when they don't
* [Model Lifecycle](https://docs.gloo.com/api-guides/model-lifecycle.md): new models, deprecations and replacements
* [Error Handling and Retries](https://docs.gloo.com/best-practices/error-handling-and-retries.md): which errors to retry, backoff with jitter and retry budgets in production
* [Handling Streaming Failures](https://docs.gloo.com/best-practices/completions-streaming-failures.md): retry budgets and continuing an interrupted stream

***

## Conclusion: You're Ready to Build

You've completed AI 104: Building with Gloo.

You started by streaming answers so your app feels fast, and kept conversations going by sending the history and trimming it to fit. You got JSON back that your code can trust, let the model call your own functions, saw how a coding agent runs that loop to finish a whole task, and worked with images in both directions. You pointed people to the resources that will help them next, learned what each request costs and how to lower it, and finished with how to handle errors and model changes in production.

Put together, that's a real application. Pick one small thing your users need, build it with what you've learned here, and grow it from there.

To keep going:

* [Tutorials](https://docs.gloo.com/tutorials/overview.md): step-by-step projects in several languages
* [Best Practices](https://docs.gloo.com/best-practices/overview.md): patterns for ingestion, streaming failures and grounded apps
* [API Reference](https://docs.gloo.com/api-reference/general/overview.md): every endpoint and field


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.