Lesson 9: Production-Ready

When Real Users Arrive

In development, every request works. In production, with real traffic, some won’t. A provider has a bad minute. A request is too long. Your organization hits its spending limit. A model you depend on is retired. None of this is unusual, and none of it has to break your app. This lesson covers three things that keep it running: handling errors and retries, knowing what failover does for you, and keeping up with model changes.

Core Concepts

Read the Error, Then Decide

Most errors from Gloo’s AI endpoints carry an error object. Two of its fields tell you what to do:

retryable says whether the same request may succeed if you send it again.
fault says whose problem it was: your request (client), the model provider (provider) or Gloo (internal).

Some errors, such as content-policy refusals and validation errors, carry a detail object instead. On the direct endpoints, every error also has an X-Gloo-Error-Code header to branch on and a short message you can show your users, and most carry a developer_message for your logs. Don’t match on the message text: its wording can change, and the codes don’t. The rules:

retryable: false: don’t send the same request again. Fix it, or show the user a message.
retryable: true with a Retry-After header: wait that many seconds, then retry.
retryable: true without Retry-After: retry with exponential backoff (wait longer after each failure) plus a little random jitter, and give up after a few tries.

Not Every 429 Is a Rate Limit

A 429 Too Many Requests can mean different things:

Error	What it means	Retry?
`2004 / RATE_LIMIT` (`rate_limit`)	The model’s provider is limiting requests for now. Not your quota.	Yes, after `Retry-After`. In a stream that already started, there’s no header: check `error.retryable` and back off
`1005 / SPENDING_LIMIT_EXCEEDED`	Your organization reached its spending limit	No, until the limit is raised
`1009 / MONTHLY_SPEND_LIMIT_REACHED`	Your organization reached its monthly API spend limit	No, until the month resets or support raises it
An endpoint-specific request quota	Some endpoints enforce their own quotas	Honor `Retry-After` when present

Don’t assume a 429 includes headers like X-RateLimit-Remaining. They aren’t part of a universal Gloo contract.

Failover Works for You

Most models on Gloo can be served by more than one provider. When the provider serving a model is failing, Gloo sends your request to another provider that serves the same model. This is provider failover, and it runs on every chat and Responses endpoint with no setup. What you notice:

The same model. The response names the model you asked for.
A little extra latency on a request that failed over.
Fewer cached tokens until the cache warms on the new provider.
No charge for the failed attempt.

Failover doesn’t cover every case. Anthropic Messages is served directly by Anthropic with no failover, and Embeddings returns a retryable 503 instead. A model with only one provider has nowhere to fail over to. When no provider can take the request, you get a 503 with retryable: true. And errors that would fail the same way everywhere, such as a request that’s too long, come straight back.

Routing: Who Picks the Model

Failover changes who serves a model. Routing changes which model you get. On the guarded Completions endpoint you can opt in to routing:

AI Core Select (model_family): name a provider family and Gloo picks a model within it.
AI Core (auto_routing: true): Gloo analyzes each query and chooses a model for it.

Both can give you a different model from one call to the next. For production, tool calling and anything you need to reproduce, use AI Select: pin a model. It is our recommendation for most applications.

Models Come and Go

Providers release new models and retire old ones. The model catalog, GET /platform/v2/models, is where you see both. For each model, three fields tell you about retirement:

is_deprecated: true once the model is scheduled for retirement
replacement_model: the model requests are routed to
deprecation_note: a plain-language explanation, including any price change

For generation models, when replacement_model is set, requests to a deprecated ID go to that replacement, so your app keeps working while you migrate. The response’s model names the replacement that served it. A replacement’s price is normally capped at 1.3 times the old rate on input or output, and any exception is stated in deprecation_note. A deprecated model is intended to stay listed for at least a week before it can be removed. Treat that as best effort: a provider can withdraw a model faster. Embeddings are different. Vectors from different models don’t mix, so changing an embedding model means re-embedding your stored content.

Get Started

pip install openai requests

A Retry Wrapper

This helper retries only what the API says is retryable. It waits at least as long as Retry-After asks, and stops after a few attempts or when the total wait would exceed a budget. The OpenAI SDK retries some errors on its own by default, including ones marked not retryable, such as a spending limit. This example turns that off with max_retries=0 so only these rules apply. Do the same in production. The limits below are the budget for background work. When someone is waiting on the result, set MAX_ATTEMPTS = 3 (two retries) and MAX_TOTAL_WAIT to a few seconds, so a long Retry-After fails fast instead of keeping them waiting.

Python

import os
import random
import time
import openai
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GLOO_API_KEY"],
    base_url="https://platform.ai.gloo.com/ai/v2/direct",
    max_retries=0,
)

MAX_ATTEMPTS = 5  # at most 5 requests in total, including the first
MAX_TOTAL_WAIT = 60  # seconds this caller is willing to wait in all


def is_retryable(err):
    body = err.body if isinstance(err.body, dict) else {}
    error = body.get("error") if isinstance(body.get("error"), dict) else body
    return error.get("retryable") is True


def create_with_retries(**kwargs):
    waited = 0
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            return client.responses.create(**kwargs)
        except openai.APIStatusError as err:
            if not is_retryable(err) or attempt == MAX_ATTEMPTS:
                print("Request failed:", err.response.headers.get("X-Gloo-Error-Code"),
                      "request id:", err.response.headers.get("X-Request-Id"))
                raise
            retry_after = err.response.headers.get("Retry-After")
            if retry_after:
                wait = float(retry_after)
            else:
                wait = min(2 ** (attempt - 1), 30) + random.random()
            if waited + wait > MAX_TOTAL_WAIT:
                raise  # too long to wait now: defer the work instead
            time.sleep(wait)
            waited += wait


response = create_with_retries(
    model="gloo-anthropic-claude-sonnet-5.5",
    input="Give me one tip for welcoming first-time visitors.",
)
print(response.output_text)

For a live stream, use a smaller budget: two retries at most before any text appears, and a delay of a couple of seconds at most. Handling Streaming Failures has recommended limits.

Check Your Models for Deprecations

Run this on a schedule, such as daily, against the model IDs your app uses:

Python

import requests

MY_MODELS = {"gloo-anthropic-claude-sonnet-5.5", "gloo-openai-gpt-5-mini"}

catalog = requests.get("https://platform.ai.gloo.com/platform/v2/models").json()["data"]
listed = {m["id"]: m for m in catalog}

for model_id in sorted(MY_MODELS):
    model = listed.get(model_id)
    if model is None:
        print(f"{model_id}: no longer in the catalog. Move to a listed model.")
    elif model.get("is_deprecated"):
        print(f"{model_id}: deprecated. Replacement: {model.get('replacement_model')}")
        print("  Note:", model.get("deprecation_note"))

When a model you use is flagged, read the note, test the replacement with your own prompts, and update your model ID.

Good to Know

Log the request ID. Keep X-Request-Id on the direct endpoints, and the trace_id in error bodies. Support can find the exact request with it.
3001 / INTERNAL_ERROR isn’t retryable. Don’t resend it unchanged. Contact support with the request ID if it keeps happening.
A non-streaming request can time out. 3007 / GATEWAY_DEADLINE means it ran too long. Stream it or lower the output limit instead of retrying.
An empty answer may be the output limit. If a reasoning model stops with finish_reason: "length" or status: "incomplete" and no text, it spent the limit thinking. Raise the limit; it isn’t an outage.
Don’t build on continued routing. An ID you can still see in the catalog is supported. One that has been removed isn’t, even if requests still go through for a while.

Key Takeaway

Let the error tell you what to do: retry only when retryable is true, wait for Retry-After when it’s there, and back off with jitter when it isn’t. Rely on failover for provider trouble, pin your model in production, and watch is_deprecated in the catalog so model changes never surprise you.

Try it on Gloo: Provider Failover lists what fails over and what your app should still handle, and Model Lifecycle shows how to detect deprecations from the API.

Learn more

Rate Limits: every kind of 429 and a retry example
Errors: the error object, every code and the direct endpoint error contract
Provider Failover: when requests fail over and when they don’t
Model Lifecycle: new models, deprecations and replacements
Error Handling and Retries: which errors to retry, backoff with jitter and retry budgets in production
Handling Streaming Failures: retry budgets and continuing an interrupted stream

Conclusion: You’re Ready to Build

You’ve completed AI 104: Building with Gloo. You started by streaming answers so your app feels fast, and kept conversations going by sending the history and trimming it to fit. You got JSON back that your code can trust, let the model call your own functions, saw how a coding agent runs that loop to finish a whole task, and worked with images in both directions. You pointed people to the resources that will help them next, learned what each request costs and how to lower it, and finished with how to handle errors and model changes in production. Put together, that’s a real application. Pick one small thing your users need, build it with what you’ve learned here, and grow it from there. To keep going:

Tutorials: step-by-step projects in several languages
Best Practices: patterns for ingestion, streaming failures and grounded apps
API Reference: every endpoint and field

AI 101: Fundamentals

AI 102: Prompting

AI 103: RAG

AI 104: Building with Gloo

Lesson 9: Production-Ready

When Real Users Arrive

Core Concepts

Read the Error, Then Decide

Not Every 429 Is a Rate Limit

Failover Works for You

Routing: Who Picks the Model

Models Come and Go

Get Started

A Retry Wrapper

Check Your Models for Deprecations

Good to Know

Key Takeaway

Learn more

Conclusion: You’re Ready to Build

​When Real Users Arrive

​Core Concepts

​Read the Error, Then Decide

​Not Every 429 Is a Rate Limit

​Failover Works for You

​Routing: Who Picks the Model

​Models Come and Go

​Get Started

​A Retry Wrapper

​Check Your Models for Deprecations

​Good to Know

​Key Takeaway

​Learn more

​Conclusion: You’re Ready to Build

When Real Users Arrive

Core Concepts

Read the Error, Then Decide

Not Every 429 Is a Rate Limit

Failover Works for You

Routing: Who Picks the Model

Models Come and Go

Get Started

A Retry Wrapper

Check Your Models for Deprecations

Good to Know

Key Takeaway

Learn more

Conclusion: You’re Ready to Build