> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a Resilient Client

> Wrap Gloo AI requests in a small client that classifies errors, retries with backoff and jitter, honors Retry-After, works with provider failover, and recovers interrupted streams.

## Overview

Requests to a model can fail for reasons that have nothing to do with your code, at the [inference](https://docs.gloo.com/ai-learning-center/gloo-ai-101/prompting-and-generation.md#inference) step itself: a provider is briefly overloaded, a connection drops, a stream is cut off halfway through an answer. Gloo AI already absorbs much of this for you with [provider failover](https://docs.gloo.com/api-guides/provider-failover.md). What reaches your application is a smaller set of errors, and every one of them tells you what to do next.

In this tutorial you'll build a small client wrapper that reads those signals and acts on them: retry when the API says a retry can work, wait as long as the API asks, stop when retrying would only fail again, and keep a streamed answer on screen when the stream breaks.

## What You'll Build

By the end of this tutorial you'll have a client module with:

* **An error classifier** that turns any Gloo AI error response into one of four decisions: success, retry, fix the request, or stop
* **Backoff with jitter** that honors `Retry-After` and gives up within a fixed budget
* **A request wrapper** that retries non-streaming calls and logs what support needs
* **Failover awareness**: what to expect in a response that was served after a provider failed, and what not to build yourself
* **Stream recovery** that retries before any text is shown and continues the answer once after text is shown

The examples are in Python and TypeScript, and run against the Chat Completions shape used by [Completions V2](https://docs.gloo.com/api-guides/completions-v2.md) and [Direct Completions](https://docs.gloo.com/api-guides/direct-completions.md).

## Prerequisites

Before starting, ensure you have:

* A Gloo AI Studio account
* Your API key from the [API Keys page](https://docs.gloo.com/studio/manage-api-credentials.md), exported as `GLOO_API_KEY`
* **Authentication setup**: complete the [Authentication Tutorial](https://docs.gloo.com/tutorials/authentication.md) first
* Python 3.10+ with `requests`, or Node.js 18+ (the TypeScript examples use the built-in `fetch`)
* Familiarity with [streaming](https://docs.gloo.com/ai-learning-center/gloo-ai-104/streaming-responses.md). If you haven't streamed a completion before, start with [Streaming AI Responses in Real Time](https://docs.gloo.com/tutorials/completions-streaming.md).

> [!NOTE]
>
> This tutorial is the hands-on build. For the policy behind each choice, read the best practices alongside it: [Error Handling and Retries](https://docs.gloo.com/best-practices/error-handling-and-retries.md) and [Handling Streaming Failures](https://docs.gloo.com/best-practices/completions-streaming-failures.md).

***

## Step 1: Learn What an Error Tells You

Every Gloo AI error carries a structured `error` object. Branch on its fields, never on the wording of the message:

```json theme={null}
{
  "error": {
    "message": "The provider did not respond in time. Please try again.",
    "type": "timeout_error",
    "code": 2001,
    "name": "PROVIDER_TIMEOUT",
    "category": "provider_error",
    "description": "The upstream model provider did not respond within the allowed time window.",
    "fault": "provider",
    "retryable": true,
    "trace_id": "trace-id"
  }
}
```

| Field | Use it for |
| :- | :- |
| `retryable` | **The retry decision.** Whether sending the same request again may succeed. |
| `fault` | Who was responsible: `client`, `provider` or `internal`. |
| `code` / `name` | What happened, for logs, metrics and alerts. |
| `message` | A message you can show your users. |
| `trace_id` | The ID to give support. |

The [direct endpoints](https://docs.gloo.com/api-guides/endpoint-types.md) add a few more signals:

| Signal | Where | What it holds |
| :- | :- | :- |
| `X-Gloo-Error-Code` | Header, on every error response | A stable lowercase code such as `provider_unavailable` or `rate_limit`. Branch on it. |
| `X-Request-Id` | Header, on every response | The request's ID. Log it and include it when you [contact support](https://docs.gloo.com/getting-started/support.md#contact-support). |
| `Retry-After` | Header, when Gloo knows how long to wait | Seconds to wait before retrying. |
| `error.developer_message` | Body | What happened and what to change. Log it; don't show it to users. |
| `error.retry_after` | Body | The same wait, in seconds. |

Some refusals use a `detail` field instead of an `error` object. A content-policy refusal carries a `detail` object with a string `code` and `retryable: false`; an authentication failure on the guarded endpoints carries `detail` as a plain string. Your classifier needs to handle all three shapes.

The errors you'll see most fall into four groups:

| Decision | Examples | What the client does |
| :- | :- | :- |
| **Retry after `Retry-After`** | `429 / 2004 RATE_LIMIT` (the provider is limiting requests for this model, not your quota), `503 / 3003 SERVICE_UNAVAILABLE` (usually 30 seconds) | Wait at least the given number of seconds, then retry |
| **Retry with backoff** | `503 / 2002 PROVIDER_UNAVAILABLE`, `504 / 2001 PROVIDER_TIMEOUT`, `500 / 2013 PROVIDER_INFERENCE_ERROR`, `503 / 2006 EMPTY_COMPLETION` | Exponential backoff with jitter, a few times |
| **Fix the request** | `400 / 1001 INVALID_REQUEST`, `400 / 1002 CONTEXT_LENGTH_EXCEEDED`, `422` validation errors, content-policy refusals | Don't resend unchanged. Change the request or the model. |
| **Stop and surface** | `401`, `403`, `402` and other billing or account limits, `500 / 3001 INTERNAL_ERROR`, `408` or `504 / 3007 GATEWAY_DEADLINE` | Don't retry. Show the error, fix credentials or billing, or contact support with the `trace_id`. |

Two cases catch people out:

* **`3001 / INTERNAL_ERROR` is never retryable**, while a mid-stream `2014 / PROVIDER_STREAM_ERROR` is. Same family of status codes, opposite decisions. That's why the classifier reads `retryable`, not the status.
* **`3007 / GATEWAY_DEADLINE`** means a non-streaming request ran past Gloo's response deadline. Sending it again meets the same deadline. Stream long generations, or lower the output limit or the reasoning effort.

The full list is in [Errors](https://docs.gloo.com/api-reference/general/errors.md#error-code-reference), and the `429` variants are explained in [Rate Limits](https://docs.gloo.com/api-reference/general/limits.md).

***

## Step 2: Classify Every Response

The classifier is the heart of the client. It reads the status, headers and body, and returns one decision. It is **fail-closed**: only an explicit `retryable: true` leads to a retry. A missing field, an unparseable body or an unknown error never does.

```python Python theme={null}
# File: gloo_client.py
import json
import logging
import os
import random
import time
from dataclasses import dataclass, field

import requests

log = logging.getLogger("gloo_client")

API_BASE = "https://platform.ai.gloo.com"

# Someone is waiting (an interactive request or a live stream): at most 2 retries,
# with waits of a couple of seconds at most.
INTERACTIVE_MAX_ATTEMPTS = 3   # the first request plus 2 retries
LIVE_STREAM_RETRIES = 2        # full retries before any text is shown
LIVE_BACKOFF_CAP_S = 2.0

# Background work nobody is watching, streaming or not: at most 5 requests in total.
BACKGROUND_MAX_ATTEMPTS = 5    # the first request plus 4 retries
BACKGROUND_STREAM_RETRIES = 4
BACKOFF_CAP_S = 30.0

USER_FALLBACK = "The service is temporarily unavailable. Please try again."


@dataclass
class Classified:
    kind: str                  # "success" | "retry" | "fix_request" | "stop"
    status: int | None = None
    code: str | None = None    # X-Gloo-Error-Code, error.name, or detail.code
    retry_after: float | None = None
    user_message: str | None = None
    trace_id: str | None = None
    request_id: str | None = None
    raw: dict = field(default_factory=dict)


class GlooAPIError(Exception):
    def __init__(self, info: Classified):
        super().__init__(f"{info.status} {info.code}: {info.user_message}")
        self.info = info


def _retry_after(headers, error: dict) -> float | None:
    value = headers.get("Retry-After")
    if value is not None:
        try:
            return max(0.0, float(value))
        except ValueError:
            pass  # not a number of seconds; fall back to backoff
    body_value = error.get("retry_after")
    if isinstance(body_value, (int, float)):
        return max(0.0, float(body_value))
    return None


def classify(status: int, headers, body) -> Classified:
    """Turn one HTTP response into a decision. Only an explicit retryable=true retries."""
    request_id = headers.get("X-Request-Id")
    if 200 <= status < 300:
        return Classified("success", status, request_id=request_id)

    body = body if isinstance(body, dict) else {}
    error = body.get("error") if isinstance(body.get("error"), dict) else None
    detail = body.get("detail") if isinstance(body.get("detail"), dict) else None
    source = error or detail or {}

    code = (
        headers.get("X-Gloo-Error-Code")
        or source.get("name")
        or (str(source["code"]) if "code" in source else None)
    )
    if source.get("retryable") is True:
        kind = "retry"
    elif status in (400, 404, 405, 413, 422):
        kind = "fix_request"
    else:
        kind = "stop"

    return Classified(
        kind=kind,
        status=status,
        code=code,
        retry_after=_retry_after(headers, source),
        user_message=source.get("message") or USER_FALLBACK,
        trace_id=source.get("trace_id"),
        request_id=request_id,
        raw=body,
    )
```

```typescript TypeScript theme={null}
// File: glooClient.ts
const API_BASE = "https://platform.ai.gloo.com";

// Someone is waiting (an interactive request or a live stream): at most 2 retries,
// with waits of a couple of seconds at most.
const INTERACTIVE_MAX_ATTEMPTS = 3; // the first request plus 2 retries
const LIVE_STREAM_RETRIES = 2; // full retries before any text is shown
const LIVE_BACKOFF_CAP_S = 2;

// Background work nobody is watching, streaming or not: at most 5 requests in total.
const BACKGROUND_MAX_ATTEMPTS = 5; // the first request plus 4 retries
const BACKGROUND_STREAM_RETRIES = 4;
const BACKOFF_CAP_S = 30;

// An upper bound on one streamed request, from the first byte to the last.
const STREAM_TIMEOUT_MS = 600_000;

const USER_FALLBACK = "The service is temporarily unavailable. Please try again.";

export type Kind = "success" | "retry" | "fix_request" | "stop";

export interface Classified {
  kind: Kind;
  status: number;
  code?: string;
  retryAfter?: number;
  userMessage?: string;
  traceId?: string;
  requestId?: string;
  raw?: unknown;
}

export class GlooAPIError extends Error {
  constructor(public info: Classified) {
    super(`${info.status} ${info.code}: ${info.userMessage}`);
  }
}

type Json = Record<string, any>;

function retryAfter(headers: Headers, error: Json): number | undefined {
  const header = headers.get("Retry-After");
  if (header !== null && header.trim() !== "" && !Number.isNaN(Number(header))) {
    return Math.max(0, Number(header));
  }
  return typeof error.retry_after === "number" ? Math.max(0, error.retry_after) : undefined;
}

export function classify(status: number, headers: Headers, body: unknown): Classified {
  const requestId = headers.get("X-Request-Id") ?? undefined;
  if (status >= 200 && status < 300) return { kind: "success", status, requestId };

  const b = (body && typeof body === "object" ? body : {}) as Json;
  const error = b.error && typeof b.error === "object" ? b.error : undefined;
  const detail = b.detail && typeof b.detail === "object" ? b.detail : undefined;
  const source: Json = error ?? detail ?? {};

  const code =
    headers.get("X-Gloo-Error-Code") ??
    source.name ??
    (source.code !== undefined ? String(source.code) : undefined);

  let kind: Kind;
  if (source.retryable === true) kind = "retry";
  else if ([400, 404, 405, 413, 422].includes(status)) kind = "fix_request";
  else kind = "stop";

  return {
    kind,
    status,
    code,
    retryAfter: retryAfter(headers, source),
    userMessage: source.message ?? USER_FALLBACK,
    traceId: source.trace_id,
    requestId,
    raw: body,
  };
}
```

The code does the following:

* Returns `success` for any `2xx`, keeping `X-Request-Id`
* Reads the `error` object first, then a `detail` object, so both error shapes work. A plain-string `detail` (an authentication failure) has no `retryable` field, so it is never retried
* Prefers the `X-Gloo-Error-Code` header for the code on the direct endpoints, then `error.name`
* Marks `400`, `404`, `405`, `413` and `422` as **fix the request**, and everything else that isn't retryable as **stop**
* Takes the wait from the `Retry-After` header, or from `error.retry_after` in the body
* Falls back to a safe, generic message for users when the body has none

***

## Step 3: Back Off With Jitter

When a retry is allowed, how long you wait matters as much as whether you retry. Two rules:

1. **When the response has `Retry-After`, wait at least that long.** The server knows when capacity returns.
2. **Otherwise, back off exponentially with jitter.** Doubling the wait gives a struggling provider room to recover; jitter spreads out clients that failed at the same moment, so they don't all retry in the same instant.

```python Python theme={null}
def backoff_delay(attempt: int) -> float:
    """Background backoff after attempt N: min(2**(N-1), 30) seconds plus up to 1s of jitter."""
    return min(2 ** (attempt - 1), BACKOFF_CAP_S) + random.uniform(0, 1)


def full_jitter_delay(retry: int, base: float = 0.5, cap: float = LIVE_BACKOFF_CAP_S) -> float:
    """Short backoff when someone is waiting: random() * min(cap, base * 2**retry)."""
    return random.random() * min(cap, base * (2 ** retry))
```

```typescript TypeScript theme={null}
/** Background backoff after attempt N: min(2**(N-1), 30) seconds plus up to 1s of jitter. */
export function backoffDelayS(attempt: number): number {
  return Math.min(2 ** (attempt - 1), BACKOFF_CAP_S) + Math.random();
}

/** Short backoff when someone is waiting: random() * min(cap, base * 2**retry). */
export function fullJitterDelayS(retry: number, base = 0.5, cap = LIVE_BACKOFF_CAP_S): number {
  return Math.random() * Math.min(cap, base * 2 ** retry);
}

const sleepS = (s: number) => new Promise<void>((r) => setTimeout(r, s * 1000));
```

`backoff_delay` is for background work, streaming or not: 1, 2, 4 and 8 seconds plus up to a second of jitter, never more than 30 seconds. `full_jitter_delay` is for when someone is waiting, on an interactive request or a live stream; it picks a random wait of at most 2 seconds.

### Retry budgets

Always bound the number of attempts. These are the budgets used in this tutorial, from [the retry rule](https://docs.gloo.com/best-practices/error-handling-and-retries.md#the-retry-rule) and [Handling Streaming Failures](https://docs.gloo.com/best-practices/completions-streaming-failures.md#retry-budgets-and-backoff):

| Context | Budget | Backoff |
| :- | :- | :- |
| Interactive request (someone is waiting) | 2 retries (3 requests in total) | `Retry-After` if it's 2s or less, else fail fast; otherwise full jitter, capped at 2s |
| Live stream, before any text is shown | 2 full retries | Same as an interactive request |
| Live stream, after text is shown | 1 automatic Continue, then user controls | Full jitter, capped at 2s |
| Background work, streaming or not | At most 5 requests in total, including the first | `Retry-After` if it's 30s or less, else stop and reschedule the job; otherwise 1s, 2s, 4s, 8s plus jitter, capped at 30s |

***

## Step 4: Wrap Your Requests

Now put the classifier and the backoff together in a `post` method. It sends the request, classifies the response, and either returns, waits and retries, or raises. A `background` flag picks the budget: by default it assumes someone is waiting for the answer.

```python Python theme={null}
def _headers() -> dict:
    return {
        "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
        "Content-Type": "application/json",
    }


def _json_or_none(response):
    try:
        return response.json()
    except ValueError:
        return None


@dataclass
class Result:
    data: dict
    attempts: int
    request_id: str | None
    redirected_from: str | None


class GlooClient:
    def __init__(self, session=None, sleep=time.sleep, timeout=(10, 300)):
        self.session = session or requests.Session()
        self.sleep = sleep
        self.timeout = timeout

    def post(self, path: str, body: dict, *, background: bool = False,
             extra_headers: dict | None = None) -> Result:
        """Send one non-streaming request, retrying only what the API says is retryable.

        background=False: someone is waiting. 2 retries, waits of 2s at most.
        background=True: nobody is waiting. 5 requests in total, waits of 30s at most.
        """
        if background:
            max_attempts, max_wait_s = BACKGROUND_MAX_ATTEMPTS, BACKOFF_CAP_S
        else:
            max_attempts, max_wait_s = INTERACTIVE_MAX_ATTEMPTS, LIVE_BACKOFF_CAP_S

        def delay(attempt: int) -> float:
            return backoff_delay(attempt) if background else full_jitter_delay(attempt - 1)

        headers = {**_headers(), **(extra_headers or {})}
        for attempt in range(1, max_attempts + 1):
            try:
                response = self.session.post(API_BASE + path, json=body,
                                             headers=headers, timeout=self.timeout)
            except (requests.ConnectionError, requests.Timeout) as exc:
                # No response at all: plausibly transient.
                if attempt == max_attempts:
                    raise
                wait = delay(attempt)
                log.warning("transport error %s; retry %d in %.1fs", exc, attempt, wait)
                self.sleep(wait)
                continue

            payload = _json_or_none(response)
            info = classify(response.status_code, response.headers, payload)
            if info.kind == "success":
                redirected = response.headers.get("X-Gloo-Model-Redirected-From")
                if redirected:
                    log.warning("model %s is deprecated; served by %s",
                                redirected, (payload or {}).get("model"))
                return Result(payload or {}, attempt, info.request_id, redirected)

            log.warning("status=%s code=%s retryable=%s trace_id=%s request_id=%s",
                        info.status, info.code, info.kind == "retry",
                        info.trace_id, info.request_id)
            if info.kind != "retry" or attempt == max_attempts:
                raise GlooAPIError(info)

            if info.retry_after is None:
                wait = delay(attempt)
            elif info.retry_after > max_wait_s:
                raise GlooAPIError(info)  # longer than this caller can wait: fail fast
            else:
                wait = info.retry_after + random.uniform(0, 0.5)
            self.sleep(wait)
        raise AssertionError("unreachable")
```

```typescript TypeScript theme={null}
function authHeaders(extra: Record<string, string> = {}): Record<string, string> {
  return {
    Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
    "Content-Type": "application/json",
    ...extra,
  };
}

async function jsonOrUndefined(res: Response): Promise<unknown> {
  try {
    return await res.json();
  } catch {
    return undefined;
  }
}

export interface Result {
  data: Json;
  attempts: number;
  requestId?: string;
  redirectedFrom?: string;
}

export interface PostOptions {
  /** false (default): someone is waiting. 2 retries, waits of 2s at most.
   *  true: nobody is waiting. 5 requests in total, waits of 30s at most. */
  background?: boolean;
  headers?: Record<string, string>;
  timeoutMs?: number;
}

export class GlooClient {
  constructor(
    protected fetchImpl: typeof fetch = fetch,
    protected sleep: (s: number) => Promise<void> = sleepS,
  ) {}

  async post(path: string, body: Json, opts: PostOptions = {}): Promise<Result> {
    const maxAttempts = opts.background ? BACKGROUND_MAX_ATTEMPTS : INTERACTIVE_MAX_ATTEMPTS;
    const maxWaitS = opts.background ? BACKOFF_CAP_S : LIVE_BACKOFF_CAP_S;
    const delay = (attempt: number) =>
      opts.background ? backoffDelayS(attempt) : fullJitterDelayS(attempt - 1);

    for (let attempt = 1; attempt <= maxAttempts; attempt++) {
      let res: Response;
      try {
        res = await this.fetchImpl(API_BASE + path, {
          method: "POST",
          headers: authHeaders(opts.headers),
          body: JSON.stringify(body),
          signal: AbortSignal.timeout(opts.timeoutMs ?? 300_000),
        });
      } catch (err) {
        // No response at all: plausibly transient.
        if (attempt === maxAttempts) throw err;
        const wait = delay(attempt);
        console.warn(`transport error ${err}; retry ${attempt} in ${wait.toFixed(1)}s`);
        await this.sleep(wait);
        continue;
      }

      const payload = await jsonOrUndefined(res);
      const info = classify(res.status, res.headers, payload);
      if (info.kind === "success") {
        const redirectedFrom = res.headers.get("X-Gloo-Model-Redirected-From") ?? undefined;
        if (redirectedFrom) {
          console.warn(`model ${redirectedFrom} is deprecated; served by ${(payload as Json)?.model}`);
        }
        return { data: (payload ?? {}) as Json, attempts: attempt, requestId: info.requestId, redirectedFrom };
      }

      console.warn(
        `status=${info.status} code=${info.code} retryable=${info.kind === "retry"} ` +
          `trace_id=${info.traceId} request_id=${info.requestId}`,
      );
      if (info.kind !== "retry" || attempt === maxAttempts) throw new GlooAPIError(info);

      let wait: number;
      if (info.retryAfter === undefined) wait = delay(attempt);
      else if (info.retryAfter > maxWaitS) throw new GlooAPIError(info); // too long: fail fast
      else wait = info.retryAfter + Math.random() * 0.5;
      await this.sleep(wait);
    }
    throw new Error("unreachable");
  }
}
```

The code does the following:

* Picks the budget from `background`, using the budgets in the table above
* Retries a network failure with no response (connection refused, reset, client timeout) with backoff, because there is no error object to read
* Retries a `retryable: true` error, waiting for `Retry-After` plus a little jitter, or backing off
* Raises immediately for **fix the request** and **stop**, so your code can handle them
* Raises straight away when the server asks for a longer wait than the budget allows, so the caller can tell the user to try again shortly or reschedule the job
* Logs `code`, `retryable`, `trace_id` and `X-Request-Id` on every failed attempt
* Warns when the response was served by a replacement for a deprecated model (`X-Gloo-Model-Redirected-From`)

Use it like any HTTP call. This request has someone waiting for it, so it keeps the default interactive budget:

```python Python theme={null}
client = GlooClient()
try:
    result = client.post("/ai/v2/guarded/chat/completions", {
        "model": "gloo-anthropic-claude-opus-5.5",
        "messages": [{"role": "user", "content": "Suggest three themes for a youth retreat."}],
    })
    print(result.data["choices"][0]["message"]["content"])
    print(f"attempts={result.attempts} request_id={result.request_id}")
except GlooAPIError as exc:
    if exc.info.kind == "fix_request":
        print("Change the request:", exc.info.code)
    else:
        print(exc.info.user_message)  # safe to show your users
```

```typescript TypeScript theme={null}
const client = new GlooClient();
try {
  const result = await client.post("/ai/v2/guarded/chat/completions", {
    model: "gloo-anthropic-claude-opus-5.5",
    messages: [{ role: "user", content: "Suggest three themes for a youth retreat." }],
  });
  console.log(result.data.choices[0].message.content);
  console.log(`attempts=${result.attempts} request_id=${result.requestId}`);
} catch (err) {
  if (err instanceof GlooAPIError && err.info.kind === "fix_request") {
    console.log("Change the request:", err.info.code);
  } else if (err instanceof GlooAPIError) {
    console.log(err.info.userMessage); // safe to show your users
  } else {
    throw err;
  }
}
```

If a `3003 / SERVICE_UNAVAILABLE` asks for 30 seconds, this request raises at once and your user sees the message instead of a frozen screen. For work nobody is watching, such as a nightly batch, pass `background=True` (`{ background: true }` in TypeScript) to get the larger budget:

```python Python theme={null}
result = client.post("/ai/v2/guarded/chat/completions", job_body, background=True)
```

```typescript TypeScript theme={null}
const result = await client.post("/ai/v2/guarded/chat/completions", jobBody, { background: true });
```

> [!WARNING]
>
> Don't stack retry layers. If you wrap this client in another retry loop, or call it from a job queue that retries failed jobs, the attempts multiply. Pick one layer to own retries.

***

## Step 5: Work With Provider Failover

Most models in the Gloo AI catalog are served by more than one provider. When the provider serving your request fails, Gloo AI retries it on another provider of **the same model**, on every chat and responses endpoint, with no configuration. Your client doesn't need to switch models or add a second provider to ride out an outage.

What changes is what a response looks like after failover:

| What you see | What it means | What your client does |
| :- | :- | :- |
| `model` is the model you asked for | Failover never substitutes a different model | Nothing. (A deprecated model ID with a replacement is the exception: `model` names the replacement and `X-Gloo-Model-Redirected-From` is set; see [Model Lifecycle](https://docs.gloo.com/api-guides/model-lifecycle.md#response-headers-on-a-redirected-request).) |
| The request took longer than usual | It waited for the failed attempt first | Set client timeouts with room for this; don't treat slow as failed |
| Fewer cached tokens | [Prompt caches](https://docs.gloo.com/ai-learning-center/gloo-ai-104/cost-and-speed.md#prompt-caching) are held per provider | Expect a dip in cache hits; see [Prompt Caching](https://docs.gloo.com/tutorials/prompt-caching.md#step-8-keep-the-cache-warm-in-production) |
| You're billed once | A failed attempt replaced by another provider is not billed | Nothing |
| An error | Failover couldn't help | Classify it as usual |

When an error does reach you, it means one of two things:

* **No provider could take the request.** You get `503` with `2002 / PROVIDER_UNAVAILABLE` and `retryable: true`. Retry with backoff: a later request benefits from Gloo having already routed around the failing provider. If it persists, try a different model.
* **The failure would happen on every provider.** An invalid request, an authorization problem, a content-policy block or a billing limit is returned straight away instead of being repeated elsewhere. Your classifier already marks these as fix the request or stop.

Two endpoints work differently:

* **[Anthropic Messages](https://docs.gloo.com/api-guides/anthropic-messages.md)** is served by Anthropic only, with no failover. A `529` there means retry after `retry-after`. The official Anthropic SDKs already do this, so use their built-in retries rather than this wrapper.
* **[Embeddings](https://docs.gloo.com/api-guides/embeddings.md)** returns `503` with `retryable: true` when the provider fails. The classifier handles it.

> [!NOTE]
>
> Failover works within a time budget for each request, so a request never waits indefinitely. For long generations, stream rather than raising your client timeout: a non-streaming request that runs past Gloo's response deadline gets `3007 / GATEWAY_DEADLINE`, which isn't retryable unchanged.

***

## Step 6: Recover Interrupted Streams

Streaming adds one complication: once a stream has started, the HTTP status is already `200`, so later failures arrive inside the stream. A streamed request can end four ways:

| Phase | What you observe | What to do |
| :- | :- | :- |
| Error before the stream | A non-`200` status (on the direct endpoints, this includes a stream that fails before its first byte) | Classify it like any response |
| Stream error event | A chunk with `finish_reason: "error"` and an `error` object | Retry or continue if `retryable` is `true` |
| Dropped connection | The stream ends with no `finish_reason` | Treat as plausibly transient |
| Content filter | `finish_reason: "content_filter"`, no `error` object | Final. Show it; never retry or continue |

The other question is whether the user has already seen text:

* **Nothing shown yet**: a full retry is invisible to the user. Retry up to the budget.
* **Text on screen**: a full retry would erase or duplicate it. Keep the text, and send one **Continue** request that asks the model to pick up where it stopped.

Gloo AI already continues a cut-off stream for you on models that support it, and retries a stream that fails before any text reaches you. This code handles what's left.

```python Python theme={null}
@dataclass
class StreamOutcome:
    text: str = ""
    shown: bool = False
    tool_call: bool = False
    finish_reason: str | None = None
    error: dict | None = None
    transport_error: str | None = None
    http_error: Classified | None = None

    @property
    def retryable(self) -> bool:
        if self.http_error is not None:
            return self.http_error.kind == "retry"
        if self.transport_error is not None:
            return True
        return bool(self.error and self.error.get("retryable") is True)


@dataclass
class StreamResult:
    status: str  # "complete" | "content_filter" | "interrupted" | "failed"
    text: str
    last: StreamOutcome
    full_retries: int = 0
    continued: bool = False


def continuation_messages(messages: list[dict], partial: str) -> list[dict]:
    return [*messages, {
        "role": "user",
        "content": ("The previous streamed answer was interrupted after this text:\n\n"
                    f"{partial}\n\n"
                    "Continue from that point without repeating the text above."),
    }]


class GlooStreamClient(GlooClient):
    def stream_once(self, path: str, body: dict, on_delta=None) -> StreamOutcome:
        out = StreamOutcome()
        try:
            with self.session.post(API_BASE + path, json={**body, "stream": True},
                                   headers=_headers(), stream=True,
                                   timeout=self.timeout) as response:
                if response.status_code != 200:
                    out.http_error = classify(response.status_code, response.headers,
                                              _json_or_none(response))
                    return out
                response.encoding = "utf-8"  # SSE is always UTF-8; don't guess from the header
                for line in response.iter_lines(decode_unicode=True):
                    if not line or not line.startswith("data:"):
                        continue
                    data = line[len("data:"):].strip()
                    if data == "[DONE]":
                        break
                    event = json.loads(data)  # a malformed event is a transport error
                    choice = (event.get("choices") or [{}])[0]
                    delta = choice.get("delta") or {}
                    if delta.get("content"):
                        out.text += delta["content"]
                        out.shown = True
                        if on_delta:
                            on_delta(delta["content"])
                    if delta.get("tool_calls"):
                        out.tool_call = True
                    if event.get("error"):
                        out.error = event["error"]
                    if choice.get("finish_reason"):
                        out.finish_reason = choice["finish_reason"]
                        break
        except (requests.ConnectionError, requests.Timeout,
                requests.exceptions.ChunkedEncodingError, json.JSONDecodeError) as exc:
            out.transport_error = str(exc)
        if out.finish_reason is None and out.error is None and out.http_error is None \
                and out.transport_error is None:
            out.transport_error = "stream ended without a finish_reason"
        return out

    def stream(self, path: str, body: dict, *, on_delta=None, background=False) -> StreamResult:
        budget = BACKGROUND_STREAM_RETRIES if background else LIVE_STREAM_RETRIES
        cap = BACKOFF_CAP_S if background else LIVE_BACKOFF_CAP_S
        retries = 0
        while True:
            out = self.stream_once(path, body, on_delta)
            if out.finish_reason in ("stop", "length", "tool_calls"):
                return StreamResult("complete", out.text, out, retries)
            if out.finish_reason == "content_filter":
                return StreamResult("content_filter", out.text, out, retries)
            if not out.retryable:
                return StreamResult("failed", out.text, out, retries)
            if out.shown and not background:
                return self._continue_once(path, body, out, on_delta)
            if retries >= budget:
                return StreamResult("failed", out.text, out, retries)
            wait = backoff_delay(retries + 1) if background else full_jitter_delay(retries)
            server_wait = out.http_error.retry_after if out.http_error else None
            if server_wait is None and out.error:
                server_wait = out.error.get("retry_after")
            if isinstance(server_wait, (int, float)):
                if server_wait > cap:  # longer than this caller can wait: fail fast
                    return StreamResult("failed", out.text, out, retries)
                wait = server_wait
            self.sleep(wait)
            retries += 1

    def _continue_once(self, path, body, first: StreamOutcome, on_delta) -> StreamResult:
        if first.tool_call:
            return StreamResult("interrupted", first.text, first)
        self.sleep(full_jitter_delay(0))
        follow = {**body, "messages": continuation_messages(body["messages"], first.text)}
        second = self.stream_once(path, follow, on_delta)
        if second.finish_reason == "content_filter":
            status = "content_filter"  # final: show it, never continue again
        else:
            status = "complete" if second.finish_reason in ("stop", "length") else "interrupted"
        return StreamResult(status, first.text + second.text, second, continued=True)
```

```typescript TypeScript theme={null}
export interface StreamOutcome {
  text: string;
  shown: boolean;
  toolCall: boolean;
  finishReason?: string;
  error?: Json;
  transportError?: string;
  httpError?: Classified;
}

export interface StreamResult {
  status: "complete" | "content_filter" | "interrupted" | "failed";
  text: string;
  last: StreamOutcome;
  fullRetries: number;
  continued: boolean;
}

function isRetryable(o: StreamOutcome): boolean {
  if (o.httpError) return o.httpError.kind === "retry";
  if (o.transportError) return true;
  return o.error?.retryable === true;
}

export function continuationMessages(messages: Json[], partial: string): Json[] {
  return [
    ...messages,
    {
      role: "user",
      content:
        "The previous streamed answer was interrupted after this text:\n\n" +
        `${partial}\n\nContinue from that point without repeating the text above.`,
    },
  ];
}

export class GlooStreamClient extends GlooClient {
  async streamOnce(path: string, body: Json, onDelta?: (t: string) => void): Promise<StreamOutcome> {
    const out: StreamOutcome = { text: "", shown: false, toolCall: false };
    try {
      const res = await this.fetchImpl(API_BASE + path, {
        method: "POST",
        headers: authHeaders(),
        body: JSON.stringify({ ...body, stream: true }),
        signal: AbortSignal.timeout(STREAM_TIMEOUT_MS),
      });
      if (res.status !== 200 || !res.body) {
        out.httpError = classify(res.status, res.headers, await jsonOrUndefined(res));
        return out;
      }
      const reader = res.body.getReader();
      const decoder = new TextDecoder();
      let buffer = "";
      read: while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split("\n");
        buffer = lines.pop() ?? "";
        for (const line of lines) {
          if (!line.startsWith("data:")) continue;
          const data = line.slice(5).trim();
          if (data === "[DONE]") break read;
          const event = JSON.parse(data); // a malformed event is a transport error
          const choice = event.choices?.[0] ?? {};
          const content = choice.delta?.content;
          if (content) {
            out.text += content;
            out.shown = true;
            onDelta?.(content);
          }
          if (choice.delta?.tool_calls) out.toolCall = true;
          if (event.error) out.error = event.error;
          if (choice.finish_reason) {
            out.finishReason = choice.finish_reason;
            break read;
          }
        }
      }
      await reader.cancel().catch(() => {});
    } catch (err) {
      out.transportError = String(err);
    }
    if (!out.finishReason && !out.error && !out.httpError && !out.transportError) {
      out.transportError = "stream ended without a finish_reason";
    }
    return out;
  }

  async stream(
    path: string,
    body: Json,
    opts: { onDelta?: (t: string) => void; background?: boolean } = {},
  ): Promise<StreamResult> {
    const budget = opts.background ? BACKGROUND_STREAM_RETRIES : LIVE_STREAM_RETRIES;
    const cap = opts.background ? BACKOFF_CAP_S : LIVE_BACKOFF_CAP_S;
    let retries = 0;
    while (true) {
      const out = await this.streamOnce(path, body, opts.onDelta);
      const done = (status: StreamResult["status"]): StreamResult => ({
        status, text: out.text, last: out, fullRetries: retries, continued: false,
      });
      if (["stop", "length", "tool_calls"].includes(out.finishReason ?? "")) return done("complete");
      if (out.finishReason === "content_filter") return done("content_filter");
      if (!isRetryable(out)) return done("failed");
      if (out.shown && !opts.background) return this.continueOnce(path, body, out, opts.onDelta);
      if (retries >= budget) return done("failed");

      let wait = opts.background ? backoffDelayS(retries + 1) : fullJitterDelayS(retries);
      const serverWait = out.httpError?.retryAfter ?? out.error?.retry_after;
      if (typeof serverWait === "number") {
        if (serverWait > cap) return done("failed"); // longer than this caller can wait
        wait = serverWait;
      }
      await this.sleep(wait);
      retries++;
    }
  }

  private async continueOnce(
    path: string, body: Json, first: StreamOutcome, onDelta?: (t: string) => void,
  ): Promise<StreamResult> {
    if (first.toolCall) {
      return { status: "interrupted", text: first.text, last: first, fullRetries: 0, continued: false };
    }
    await this.sleep(fullJitterDelayS(0));
    const follow = { ...body, messages: continuationMessages(body.messages, first.text) };
    const second = await this.streamOnce(path, follow, onDelta);
    const reason = second.finishReason;
    return {
      status: reason === "content_filter" ? "content_filter" // final: show it, never continue again
        : reason === "stop" || reason === "length" ? "complete" : "interrupted",
      text: first.text + second.text,
      last: second,
      fullRetries: 0,
      continued: true,
    };
  }
}
```

The code does the following:

* Classifies a non-`200` response with the same `classify` function, so a `401` before the stream is never retried
* Collects text as it arrives and records whether any was shown, and whether a [tool call](https://docs.gloo.com/ai-learning-center/gloo-ai-101/application-patterns.md#tool-use-or-function-calling) was emitted
* Ends on `stop`, `length` or `tool_calls` as complete, and on `content_filter` as final
* Stops on `retryable: false`, keeping any partial text
* Treats a dropped connection, a client timeout or a malformed event as a transport error, which is plausibly transient
* Retries before any text is shown: 2 full retries with short waits for live streams; 4 retries (5 requests in total) with `min(2^(attempt-1), 30)` seconds plus jitter for background jobs
* Honors a server-requested wait when it fits the budget, and fails fast when it doesn't (for example a 30-second `Retry-After` on a live stream)
* After text is shown, sends **one** Continue request, using a new user message that carries the partial answer
* Skips the automatic Continue when a tool call was emitted, so a side effect can't run twice

Use it with a callback that renders text as it arrives:

```python Python theme={null}
client = GlooStreamClient()
result = client.stream(
    "/ai/v2/guarded/chat/completions",
    {
        "model": "gloo-anthropic-claude-opus-5.5",
        "messages": [{"role": "user", "content": "Explain Romans 8 in simple terms."}],
    },
    on_delta=lambda text: print(text, end="", flush=True),
)
print()
if result.status == "interrupted":
    print("[The answer was cut off. Continue | Try again]")
elif result.status == "content_filter":
    pass  # final: the text shown is the answer, so offer no Continue or Try again
elif result.status == "failed":
    print("[Something went wrong. Please try again.]")
```

```typescript TypeScript theme={null}
const client = new GlooStreamClient();
const result = await client.stream(
  "/ai/v2/guarded/chat/completions",
  {
    model: "gloo-anthropic-claude-opus-5.5",
    messages: [{ role: "user", content: "Explain Romans 8 in simple terms." }],
  },
  { onDelta: (text) => process.stdout.write(text) },
);
console.log();
if (result.status === "interrupted") {
  console.log("[The answer was cut off. Continue | Try again]");
} else if (result.status === "content_filter") {
  // final: the text shown is the answer, so offer no Continue or Try again
} else if (result.status === "failed") {
  console.log("[Something went wrong. Please try again.]");
}
```

### Designing the interrupted state

When `status` is `interrupted`, the partial answer is still in `result.text` and still on screen. Keep it visible and mark it as incomplete. Offer **Continue** as the main action (it appends) and **Try again** as a secondary action that's clearly labelled as replacing the answer. A user-initiated Continue doesn't count against the automatic budget.

> [!WARNING]
>
> A Continue is a new request: it can repeat a phrase or drift from where the first stream was heading. For high-stakes, factual or doctrinal answers, prefer **Try again** or user review over an automatic Continue. See [Application-Level Continuation](https://docs.gloo.com/best-practices/completions-streaming-failures.md#application-level-continuation).

### Direct Responses streams

On [Direct Responses](https://docs.gloo.com/api-guides/direct-responses.md), a stream that fails after output has started ends with a `response.failed` event instead of an error chunk. `response.error` carries the string `code` (the same value as `X-Gloo-Error-Code`), `gloo_code`, `retryable` and, when it applies, `retry_after`. The same decisions apply: read `retryable`, check whether text was shown, and continue or surface.

***

## Step 7: Test the Failure Paths

Don't wait for a real outage to test your retry logic. Both clients take their transport and their sleep function as constructor arguments, so you can script failures and check the decisions without calling the API or waiting.

```python Python theme={null}
# File: test_gloo_client.py
import os

from requests.structures import CaseInsensitiveDict

os.environ.setdefault("GLOO_API_KEY", "test-key")
from gloo_client import GlooAPIError, GlooClient


class FakeResponse:
    def __init__(self, status, body, headers=None):
        self.status_code, self._body = status, body
        self.headers = CaseInsensitiveDict(headers or {})

    def json(self):
        return self._body


class FakeSession:
    def __init__(self, responses):
        self.responses, self.calls = list(responses), 0

    def post(self, *args, **kwargs):
        self.calls += 1
        return self.responses.pop(0)


def error(code, name, retryable):
    return {"error": {"code": code, "name": name, "retryable": retryable, "message": "m"}}


def expect_error(session, **kwargs) -> GlooAPIError:
    try:
        GlooClient(session=session, sleep=waits.append).post("/x", {}, **kwargs)
    except GlooAPIError as exc:
        return exc
    raise AssertionError("expected GlooAPIError, but the call succeeded")


waits = []
session = FakeSession([
    FakeResponse(429, error(2004, "RATE_LIMIT", True), {"Retry-After": "3"}),
    FakeResponse(200, {"model": "gloo-anthropic-claude-opus-5.5"}),
])
result = GlooClient(session=session, sleep=waits.append).post("/x", {}, background=True)
assert result.attempts == 2 and 3 <= waits[0] <= 3.5, "background honors Retry-After"

session = FakeSession([FakeResponse(503, error(3003, "SERVICE_UNAVAILABLE", True),
                                    {"Retry-After": "30"})])
expect_error(session)
assert session.calls == 1, "interactive fails fast on a 30s Retry-After"

session = FakeSession([FakeResponse(500, error(3001, "INTERNAL_ERROR", False))])
exc = expect_error(session, background=True)
assert exc.info.kind == "stop" and session.calls == 1, "3001 is not retried"

session = FakeSession([FakeResponse(503, error(2002, "PROVIDER_UNAVAILABLE", True))] * 5)
expect_error(session)
assert session.calls == 3, "interactive: 2 retries"

waits.clear()
session = FakeSession([FakeResponse(503, error(2002, "PROVIDER_UNAVAILABLE", True))] * 5)
expect_error(session, background=True)
assert session.calls == 5, "background: five requests in total"
assert all(w <= 31 for w in waits), "background waits are capped at 30s plus jitter"

print("✓ all failure paths behave as expected")
```

```typescript TypeScript theme={null}
// File: glooClient.test.ts
import { GlooAPIError, GlooClient } from "./glooClient";

process.env.GLOO_API_KEY ??= "test-key";

function check(condition: boolean, message: string) {
  if (!condition) throw new Error(`failed: ${message}`);
}
const error = (code: number, name: string, retryable: boolean) => ({
  error: { code, name, retryable, message: "m" },
});
const json = (status: number, body: unknown, headers: Record<string, string> = {}) =>
  new Response(JSON.stringify(body), { status, headers });

function fakeFetch(responses: Response[]) {
  const state = { calls: 0 };
  const impl = (async () => (state.calls++, responses.shift()!)) as unknown as typeof fetch;
  return { impl, state };
}
const waits: number[] = [];
const recordSleep = async (s: number) => { waits.push(s); };

async function expectError(fake: ReturnType<typeof fakeFetch>, opts = {}): Promise<GlooAPIError> {
  try {
    await new GlooClient(fake.impl, recordSleep).post("/x", {}, opts);
  } catch (err) {
    if (err instanceof GlooAPIError) return err;
    throw err;
  }
  throw new Error("failed: expected GlooAPIError, but the call succeeded");
}

async function main() {
  let fake = fakeFetch([
    json(429, error(2004, "RATE_LIMIT", true), { "Retry-After": "3" }),
    json(200, { model: "gloo-anthropic-claude-opus-5.5" }),
  ]);
  const result = await new GlooClient(fake.impl, recordSleep).post("/x", {}, { background: true });
  check(result.attempts === 2 && waits[0] >= 3 && waits[0] <= 3.5, "background honors Retry-After");

  fake = fakeFetch([json(503, error(3003, "SERVICE_UNAVAILABLE", true), { "Retry-After": "30" })]);
  await expectError(fake);
  check(fake.state.calls === 1, "interactive fails fast on a 30s Retry-After");

  fake = fakeFetch([json(500, error(3001, "INTERNAL_ERROR", false))]);
  const stop = await expectError(fake, { background: true });
  check(stop.info.kind === "stop" && fake.state.calls === 1, "3001 is not retried");

  const unavailable = () => json(503, error(2002, "PROVIDER_UNAVAILABLE", true));
  fake = fakeFetch(Array.from({ length: 5 }, unavailable));
  await expectError(fake);
  check(fake.state.calls === 3, "interactive: 2 retries");

  waits.length = 0;
  fake = fakeFetch(Array.from({ length: 5 }, unavailable));
  await expectError(fake, { background: true });
  check(fake.state.calls === 5, "background: five requests in total");
  check(waits.every((w) => w <= 31), "background waits are capped at 30s plus jitter");

  console.log("✓ all failure paths behave as expected");
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Run it:

```bash Python theme={null}
python test_gloo_client.py
```

```bash TypeScript theme={null}
npx tsx glooClient.test.ts
```

Your output should look similar to the following (the client logs each failed attempt):

```text theme={null}
status=429 code=RATE_LIMIT retryable=True trace_id=None request_id=None
status=503 code=SERVICE_UNAVAILABLE retryable=True trace_id=None request_id=None
status=500 code=INTERNAL_ERROR retryable=False trace_id=None request_id=None
status=503 code=PROVIDER_UNAVAILABLE retryable=True trace_id=None request_id=None
...
✓ all failure paths behave as expected
```

Each check fails loudly if the client returns when it should have raised. Add cases for the paths that matter to your application: a `400` content-policy refusal, a network failure with no response, a stream that drops after the first chunk, and a stream that ends with `content_filter`.

***

## Step 8: Log What Support Needs

When something fails repeatedly, the IDs in your logs are what let support find the exact request. For every failed attempt, record:

* `error.code`, `error.name`, `error.fault` and `error.retryable`
* `trace_id` from the error, and `X-Request-Id` on the direct endpoints (stream errors on the direct endpoints carry no `trace_id`, so `X-Request-Id` is the one to keep)
* `developer_message`, when present
* The model, the attempt number and the wait before the retry
* For streams: whether text was shown, its length, whether a Continue was sent, and whether a tool call was emitted

If inference keeps failing after retries, open a ticket through Support in Gloo AI Studio (see [Get help](https://docs.gloo.com/getting-started/support.md)) and include the `trace_id`.

***

## Troubleshooting

**Every error is retried, including `401` and `400`**
: Check that retries depend on `retryable === true` and nothing else. A classifier that retries on "any `5xx`" or "any error without `retryable: false`" is fail-open and will loop on errors that can't succeed.

**Requests hang for 30 seconds before failing**
: A `3003 / SERVICE_UNAVAILABLE` asked the client to wait for `Retry-After`, and the request was sent with `background=True`. For requests someone is waiting on, keep the default interactive budget: the client raises straight away and your UI can show "please try again shortly".

**A `429` is never retried**
: Not every `429` is retryable. A provider rate limit (`2004 / RATE_LIMIT`) is; a billing or account limit is not, and comes back with `retryable: false`. See [Rate Limits](https://docs.gloo.com/api-reference/general/limits.md).

**`408` or `504` with `3007 / GATEWAY_DEADLINE` on long answers**
: The non-streaming request ran past Gloo's response deadline. Use `GlooStreamClient`, or lower `max_tokens` or the reasoning effort.

**A reasoning model returns no text with `finish_reason: "length"`**
: That isn't an outage, and it isn't retried. The model spent the output limit [thinking](https://docs.gloo.com/ai-learning-center/gloo-ai-102/models-that-think.md). Raise `max_tokens` or lower the reasoning effort; see [Set your output limit for reasoning models](https://docs.gloo.com/api-guides/reasoning.md#set-your-output-limit-for-reasoning-models).

**A streamed answer appears twice after a failure**
: The client did a full retry after text was already shown. Track whether text was shown, and use Continue, not a full retry, once it has.

**The continued answer repeats the last sentence**
: Continuation is best effort. De-duplicate the overlap where the two parts meet before you render the second part.

***

## Next Steps

> **[Error Handling and Retries](https://docs.gloo.com/best-practices/error-handling-and-retries.md)**
>
> Retry policy, budgets and backoff across every endpoint.

> **[Handling Streaming Failures](https://docs.gloo.com/best-practices/completions-streaming-failures.md)**
>
> The full decision tree and reference implementations for streams.

> **[Errors Reference](https://docs.gloo.com/api-reference/general/errors.md)**
>
> Every error code, the error object, and streaming error delivery.

> **[Provider Failover](https://docs.gloo.com/api-guides/provider-failover.md)**
>
> When Gloo AI retries on another provider, and when it doesn't.

> **[Rate Limits](https://docs.gloo.com/api-reference/general/limits.md)**
>
> The different kinds of `429`, and which ones to retry.

> **[Prompt Caching](https://docs.gloo.com/tutorials/prompt-caching.md)**
>
> Cut cost and latency on the requests your client sends.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.