Error Handling and Retries

Most failed requests tell you what went wrong and whether trying again can help. Read that information to retry failures that clear on their own, stop quickly on the ones that won’t, and give support what it needs when something keeps failing. This guide covers non-streaming requests and the HTTP error responses that come back before a stream starts:

reading the error object instead of the status alone
deciding which errors to retry
backing off with jitter and honoring Retry-After
setting retry budgets
logging the right fields

Streaming requests can also fail after the HTTP 200 response has started, once your user may already be reading partial output. Those failures need lower retry ceilings and a different recovery strategy. See Handling Streaming Failures. This page and that one share the same rules for reading error.retryable and the same retry budgets.

For the full list of status codes and error codes, see the Errors and Rate Limits reference pages.

The Retry Rule

One rule covers every Gloo AI endpoint, streaming or not:

Retry only what the error marks retryable (error.retryable: true, or a dropped connection). Never retry 4xx authentication, validation, or billing errors.
Background work (batch jobs, ingestion, anything no one is watching): at most 5 requests in total, including the first. Wait for Retry-After first; otherwise wait min(2^(attempt-1), 30) seconds plus jitter.
Someone is waiting (interactive requests and live streams): at most 2 retries, with short backoff capped at a couple of seconds. See Handling Streaming Failures.
When Retry-After is longer than the caller can wait, such as the 30 seconds a 3003 / SERVICE_UNAVAILABLE usually asks for, don’t wait: fail fast with a clear message, or hand the work to a background job.
Honor X-RateLimit-* headers when present, but don’t assume a response includes them.

Read the Error, Not Just the Status

Completions and Responses errors include a structured error object. Use it for programmatic handling instead of parsing human-readable messages. See AI Error Object for every field.

Branch on error.retryable first. It says whether retrying the same request may succeed. Use error.name or error.code to tell exactly what happened.
Log error.name and error.code, not just the status. The status gives only the category; the name and code give you precise alerting and debugging.
On the direct endpoints, branch on the X-Gloo-Error-Code header and retryable, not on the status alone. Show message to your end users, and log developer_message for yourself. Never parse message or developer_message: their wording can change; the codes do not. See Direct Endpoint Errors.
Use detail to pinpoint validation failures. For 422 responses, the detail array tells you exactly which field failed: loc is the field path, msg is the reason, and input is the value you sent.

Which Errors to Retry

Retry only transient failures, and only when the request is safe to send again.

Response	Retry?	What to do
`429 / 2004 RATE_LIMIT` (the model’s provider is limiting requests)	Yes	Wait for `Retry-After`, then retry with bounded exponential backoff. This is not your account’s quota.
`503 / 3003 SERVICE_UNAVAILABLE`	Yes	Retry after `Retry-After`.
`504` provider timeouts and other `5xx` with `retryable: true`	Yes	Bounded retries with backoff.
Embeddings `503 / 2002 PROVIDER_UNAVAILABLE`	Yes	Bounded exponential backoff. If the same request keeps failing, change the request or model instead.
Endpoint-specific `429` quota	Depends	Honor `Retry-After` when present. Otherwise retry only when the operation is safe to repeat, with a bounded delay and jitter.
`3007 / GATEWAY_DEADLINE` (`408` on guarded and grounded, `504` on direct)	Not unchanged	Shorten the request or stream it.
`3001 / INTERNAL_ERROR`	Never	Surface the error; contact support with the `trace_id` if it persists.
`429 / 1005 SPENDING_LIMIT_EXCEEDED`	No	Retry only after the account’s spending-limit condition is resolved.
`429 / 1009 MONTHLY_SPEND_LIMIT_REACHED`	No	Access returns when the month resets (UTC) or support raises the limit.
`402 INSUFFICIENT_CREDIT`	No	Add credits, then retry.
`401`	No	The API key is missing, invalid, or revoked. A static key won’t become valid on retry; fix the key. See API Key Security.
`403` insufficient permissions	No	The key or organization lacks access to this resource. Check your access in Studio. See 403 Forbidden.
`403` from guardrails	Not unchanged	Show your user a helpful message rather than resending the same text. See Guarded endpoints.
`400` / `422`	No	Fix the request; `detail` names the field.

Not every 429 is retryable. Read the error object before you decide. Network failures with no response, such as connection timeouts and dropped connections, are likely transient: retry them with the same backoff and budget, and keep your user informed while you do.

Backoff with Jitter

Wait longer between each attempt, and add random jitter so many clients that failed at the same moment don’t all retry at the same moment.

If the response has Retry-After, wait at least that long. On the direct endpoints the same wait is also in the error body as retry_after. If that is longer than the caller can wait, fail fast or defer the work instead.
Otherwise, use exponential backoff with jitter, capped so a single wait never grows without bound: min(2^(attempt-1), 30) seconds plus jitter for background work, a couple of seconds at most when someone is waiting.
Stop retrying and surface the error when the error is not retryable, the budget is spent, or the limit requires an account or configuration change.

Honor X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset when present, but don’t assume a 429 includes them; they aren’t part of a universal Gloo API response contract.

Retry Example

This pseudocode follows the background budget: at most five total requests, including the first. It retries only when the response metadata, endpoint policy, and request safety all permit it, gives precedence to Retry-After, and excludes spending-limit errors.

attempt = 1
max_attempts = 5

loop:
  response = make_api_request()

  if response.status_code != 429:
    return response

  error = parse_error(response)

  if error is missing or malformed:
    return response

  if error.name == "SPENDING_LIMIT_EXCEEDED" or error.retryable != true:
    return response

  if endpoint_policy_allows_retry(response) != true:
    return response

  if request_is_safe_to_retry(response.request) != true:
    return response

  if attempt >= max_attempts:
    return response

  if response.headers contains "Retry-After":
    wait_seconds = parse_retry_after(response.headers["Retry-After"])
  else:
    wait_seconds = min(2 ** (attempt - 1), 30) + random_jitter()

  sleep(wait_seconds)
  attempt += 1

The same structure applies to retryable 5xx responses: let 5xx statuses through the first check alongside 429, and the error.retryable check after parse_error decides whether to retry.

New to retries? Production-Ready in the AI Learning Center builds a small retry helper that follows these rules.

Retry Budgets

Every retry loop needs a ceiling, and a user waiting on a screen needs a much smaller one than a background job.

Context	Ceiling	Source of the rule
Background or non-interactive work, streaming or not	At most 5 total requests, including the first, with waits capped at 30 seconds and an overall job timeout	Retry Example above
Someone is waiting: an interactive request, or a live stream before any visible content	2 retries, with backoff capped at a couple of seconds	Handling Streaming Failures
Live stream, after visible content	1 automatic Continue, then user-initiated controls	Handling Streaming Failures
`Retry-After` longer than the caller can wait	Don’t wait: fail fast with a clear message, or defer to a background job	The Retry Rule

Stop early the moment an error reports retryable: false, whatever the remaining budget.

Make Requests Safe to Retry

A retry sends the same request again. Before you retry, make sure doing it twice is harmless:

Requests that change data, such as uploads, need a way to detect a duplicate. See Duplicate Upload Results for how ingestion reports content it already has.
Tool calls with side effects, such as sending an email, should carry idempotency keys or operation IDs so a replayed call can be detected and dropped. See Tool Calls and Side Effects.

Batch and Ingestion Jobs

Jobs that send many requests in a row, such as uploading a content library, should:

Control call frequency with delays between API calls, and back off when a rate limit arrives.
Retry failed items with the same budget, and report success or failure for each item.
Track progress so you can see each item’s processing status.
Validate content before sending it, so a validation error is caught once instead of on every retry.

Log What Support Needs

For every failed request, log:

error.name, error.code, error.fault, and error.retryable
the trace_id from the error object (on the direct endpoints, also X-Request-Id, which every response carries)
the attempt number and the wait before it
the endpoint and model you called

Track success counts, error counts by error.name, and response times so you can see a rising failure rate before your users report it. Open a ticket with the trace_id for persistent inference issues. If inference errors continue after retrying with backoff, contact Support in Gloo AI Studio (bottom of the sidebar; see Get help) and include the error response’s trace_id so we can trace the exact request.

Summary

Practice	Minimum viable version
Read the error	Branch on `error.retryable`, then `error.name` / `error.code` (`X-Gloo-Error-Code` on direct)
Retry selectively	Only `retryable: true` and safe-to-repeat requests
Back off	`Retry-After` first, else exponential backoff with jitter and a cap
Bound it	At most 5 requests for background work; 2 retries when someone is waiting
Log it	`error.name`, `error.code`, `trace_id`, attempt number

Next steps

Handling Streaming Failures

Retry, continuation, and partial-output UX for interrupted streams.

Errors Reference

Every status code, error code, and the AI error object.

Rate Limits

Spending limits, endpoint quotas, and provider rate limits.

API Key Security

Storing, using, and replacing API keys safely.

Best Practices

Security

Requests

Reliability

Trust

Content

Error Handling and Retries

The Retry Rule

Read the Error, Not Just the Status

Which Errors to Retry

Backoff with Jitter

Retry Example

Retry Budgets

Make Requests Safe to Retry

Batch and Ingestion Jobs

Log What Support Needs

Summary

Next steps

Handling Streaming Failures

Errors Reference

Rate Limits

API Key Security

​The Retry Rule

​Read the Error, Not Just the Status

​Which Errors to Retry

​Backoff with Jitter

​Retry Example

​Retry Budgets

​Make Requests Safe to Retry

​Batch and Ingestion Jobs

​Log What Support Needs

​Summary

​Next steps

Handling Streaming Failures

Errors Reference

Rate Limits

API Key Security

The Retry Rule

Read the Error, Not Just the Status

Which Errors to Retry

Backoff with Jitter

Retry Example

Retry Budgets

Make Requests Safe to Retry

Batch and Ingestion Jobs

Log What Support Needs

Summary

Next steps