Part 3: Verification, Error Handling & Resilience

This is Part 3 of the Build an End-to-End RAG Pipeline series: the ingestion side of RAG. Parts 1 and 2 followed the happy path, but production code can’t assume it: requests fail, services blip, and you need to confirm that an operation actually took effect. This part builds a small resilient client and uses it to interpret API errors, retry transient failures, and verify ingestion health. The retrieval side (semantic search and grounded answers) is covered in the Building Custom Search tutorial and the Grounded Completions guide.

Gloo AI does not include a monitoring or health-check endpoint. Resilience is built from the same item APIs you’ve already used, plus disciplined error handling on the client side.

Pipeline at a Glance

Publisher setup (Studio)

Create the publisher that owns your content: Part 1.

Ingest content with metadata

Upload files and enrich them: Part 1.

Verify indexing

Poll item status until your content is searchable: Part 1.

Semantic search

Query your content. Deep dive: Building Custom Search.

Grounded completions with sources

Answer questions from your content with citations. Deep dive: Grounded Completions, which sets rag_publisher to your publisher name.

Content lifecycle

Update, bulk-edit, and delete content: Part 2.

Verification, errors & resilience

Error handling and retry patterns: covered below.

Prerequisites

Before starting, you need:

A Gloo AI Studio account
Your API key from the API Keys page. Keep it on your server and out of source control; see API Key Security
A publisher with credentials: see Part 1
Familiarity with the item APIs from Part 1 and Part 2
Data Engine access: Data Engine is currently at capacity for new organizations; see Part 1 for details and Billing & Plans to request access

Step 1: Parse Errors and Retry

A resilient client does two things on every request: it turns failures into a normalized error (status, code, message) that carries the body’s error.retryable flag, and it retries only the failures the API marks as transient. The authoritative signal is that retryable boolean (Errors): retry when it is true, and hand the error straight back to the caller when it is false. A 500 with retryable: false (for example 3001 INTERNAL_ERROR) will not succeed on a second attempt. The transient status set (500, 502, 503, 504) is only a fallback for responses whose error body is missing or unparseable, where there is no flag to read. For the general rules this step applies, including retry budgets and Retry-After, see Error Handling and Retries. The parser normalizes all of the Data Engine’s error shapes: {"detail": {"code", "message"}}, {"detail": "..."}, and {"error", "message"}.

import time
import requests

RETRYABLE = {500, 502, 503, 504}  # fallback when the body carries no retryable flag
MAX_RETRIES = 4


class ApiError(Exception):
    def __init__(self, status, code, message, retryable=None):
        super().__init__(f"[{status} {code}] {message}")
        self.status, self.code, self.message, self.retryable = status, code, message, retryable


def parse_error(response):
    try:
        body = response.json()
    except ValueError:
        return None, response.reason, None
    detail = body.get("detail") if isinstance(body, dict) else None
    if isinstance(detail, dict):
        flag = detail.get("retryable")
        return detail.get("code"), detail.get("message") or response.reason, flag
    if isinstance(detail, str):
        return None, detail, None
    return body.get("error"), body.get("message") or response.reason, None


def request(method, url, token, **kwargs):
    for attempt in range(MAX_RETRIES + 1):
        response = requests.request(
            method, url, headers={"Authorization": f"Bearer {token}"}, timeout=30, **kwargs
        )
        if not response.ok:
            code, message, retryable = parse_error(response)
            # error.retryable is authoritative; the status set is only the fallback
            # when the response carries no JSON error body.
            retry = retryable if retryable is not None else response.status_code in RETRYABLE
            if retry and attempt < MAX_RETRIES:
                delay = 2 ** attempt
                print(f"    Attempt {attempt + 1} failed ({response.status_code}); retrying in {delay}s")
                time.sleep(delay)
                continue
            raise ApiError(response.status_code, code, message, retryable)
        return response.json()

const RETRYABLE = new Set([500, 502, 503, 504]); // fallback when the body carries no retryable flag
const MAX_RETRIES = 4;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

class ApiError extends Error {
  constructor(status, code, message, retryable = null) {
    super(message);
    this.status = status;
    this.code = code;
    this.retryable = retryable;
  }
}

async function parseError(response) {
  const text = await response.text();
  let body;
  try {
    body = JSON.parse(text);
  } catch {
    return [null, response.statusText, null];
  }
  const detail = body?.detail;
  if (detail && typeof detail === "object") {
    return [detail.code ?? null, detail.message || response.statusText, detail.retryable ?? null];
  }
  if (typeof detail === "string") return [null, detail, null];
  return [body?.error ?? null, body?.message || response.statusText, null];
}

async function request(method, url, token, body) {
  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    const response = await fetch(url, {
      method,
      headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
      body: body ? JSON.stringify(body) : undefined,
    });
    if (!response.ok) {
      const [code, message, retryable] = await parseError(response);
      // error.retryable is authoritative; the status set is only the fallback
      // when the response carries no JSON error body.
      const retry = retryable !== null ? retryable : RETRYABLE.has(response.status);
      if (retry && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${response.status}); retrying in ${delay}s`);
        await sleep(delay * 1000);
        continue;
      }
      throw new ApiError(response.status, code, message, retryable);
    }
    return response.json();
  }
}

const RETRYABLE = new Set([500, 502, 503, 504]); // fallback when the body carries no retryable flag
const MAX_RETRIES = 4;
const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));

class ApiError extends Error {
  constructor(
    readonly status: number | null,
    readonly code: string | null,
    message: string,
    readonly retryable: boolean | null = null,
  ) {
    super(message);
  }
}

async function parseError(response: Response): Promise<[string | null, string, boolean | null]> {
  const text = await response.text();
  let body: any;
  try {
    body = JSON.parse(text);
  } catch {
    return [null, response.statusText, null];
  }
  const detail = body?.detail;
  if (detail && typeof detail === "object") {
    return [detail.code ?? null, detail.message || response.statusText, detail.retryable ?? null];
  }
  if (typeof detail === "string") return [null, detail, null];
  return [body?.error ?? null, body?.message || response.statusText, null];
}

async function request(method: string, url: string, token: string, body?: unknown): Promise<any> {
  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    const response = await fetch(url, {
      method,
      headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
      body: body ? JSON.stringify(body) : undefined,
    });
    if (!response.ok) {
      const [code, message, retryable] = await parseError(response);
      // error.retryable is authoritative; the status set is only the fallback
      // when the response carries no JSON error body.
      const retry = retryable !== null ? retryable : RETRYABLE.has(response.status);
      if (retry && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${response.status}); retrying in ${delay}s`);
        await sleep(delay * 1000);
        continue;
      }
      throw new ApiError(response.status, code, message, retryable);
    }
    return response.json();
  }
}

<?php

const RETRYABLE = [500, 502, 503, 504]; // fallback when the body carries no retryable flag
const MAX_RETRIES = 4;

class ApiError extends RuntimeException {
    public function __construct(
        public ?int $status,
        public ?string $apiCode,
        string $message,
        public ?bool $retryable = null,
    ) {
        parent::__construct($message);
    }
}

function parseError(int $status, string $body): array {
    $reasons = [400 => 'Bad Request', 403 => 'Forbidden', 404 => 'Not Found', 500 => 'Internal Server Error'];
    $reason = $reasons[$status] ?? "HTTP $status";
    $decoded = json_decode($body, true);
    $detail = is_array($decoded) ? ($decoded['detail'] ?? null) : null;
    if (is_array($detail)) {
        $flag = is_bool($detail['retryable'] ?? null) ? $detail['retryable'] : null;
        return [$detail['code'] ?? null, ($detail['message'] ?? '') ?: $reason, $flag];
    }
    if (is_string($detail)) return [null, $detail, null];
    return [$decoded['error'] ?? null, ($decoded['message'] ?? '') ?: $reason, null];
}

function request(string $method, string $url, string $token, ?array $body = null) {
    for ($attempt = 0; $attempt <= MAX_RETRIES; $attempt++) {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_CUSTOMREQUEST => $method,
            CURLOPT_HTTPHEADER => ["Authorization: Bearer $token", 'Content-Type: application/json'],
            CURLOPT_POSTFIELDS => $body !== null ? json_encode($body) : null,
            CURLOPT_TIMEOUT => 30,
        ]);
        $responseBody = (string) curl_exec($ch);
        $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        if ($status >= 400) {
            [$code, $message, $retryable] = parseError($status, $responseBody);
            // error.retryable is authoritative; the status set is only the fallback
            // when the response carries no JSON error body.
            $retry = $retryable ?? in_array($status, RETRYABLE, true);
            if ($retry && $attempt < MAX_RETRIES) {
                $delay = 2 ** $attempt;
                echo "    Attempt " . ($attempt + 1) . " failed ($status); retrying in {$delay}s\n";
                sleep($delay);
                continue;
            }
            throw new ApiError($status, $code, $message, $retryable);
        }
        return json_decode($responseBody, true);
    }
}

// fallback when the body carries no retryable flag
var retryable = map[int]bool{500: true, 502: true, 503: true, 504: true}

const maxRetries = 4

type ApiError struct {
	Status    int
	Code      string
	Message   string
	Retryable *bool
}

func (e *ApiError) Error() string { return fmt.Sprintf("[%d %s] %s", e.Status, e.Code, e.Message) }

// retryableField reads the error object's retryable boolean, if present.
func retryableField(obj map[string]any) *bool {
	if v, ok := obj["retryable"].(bool); ok {
		return &v
	}
	return nil
}

func parseError(status int, body []byte) (string, string, *bool) {
	reason := http.StatusText(status)
	var decoded map[string]any
	if json.Unmarshal(body, &decoded) != nil {
		return "", reason, nil
	}
	if detail, ok := decoded["detail"].(map[string]any); ok {
		return str(detail["code"]), orReason(str(detail["message"]), reason), retryableField(detail)
	}
	if detail, ok := decoded["detail"].(string); ok {
		return "", detail, nil
	}
	if errObj, ok := decoded["error"].(map[string]any); ok {
		return str(errObj["code"]), orReason(str(errObj["message"]), reason), retryableField(errObj)
	}
	return str(decoded["error"]), orReason(str(decoded["message"]), reason), nil
}

func request(method, url, token string, body []byte) (map[string]any, error) {
	for attempt := 0; attempt <= maxRetries; attempt++ {
		var reader io.Reader
		if body != nil {
			reader = bytes.NewReader(body)
		}
		req, _ := http.NewRequest(method, url, reader)
		req.Header.Set("Authorization", "Bearer "+token)
		req.Header.Set("Content-Type", "application/json")
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			return nil, &ApiError{0, "network_error", err.Error(), nil}
		}
		respBody, _ := io.ReadAll(resp.Body)
		resp.Body.Close()

		if resp.StatusCode >= 400 {
			code, message, bodyRetryable := parseError(resp.StatusCode, respBody)
			// error.retryable is authoritative; the status map is only the fallback
			// when the response carries no JSON error body.
			retry := retryable[resp.StatusCode]
			if bodyRetryable != nil {
				retry = *bodyRetryable
			}
			if retry && attempt < maxRetries {
				delay := 1 << attempt
				fmt.Printf("    Attempt %d failed (%d); retrying in %ds\n", attempt+1, resp.StatusCode, delay)
				time.Sleep(time.Duration(delay) * time.Second)
				continue
			}
			return nil, &ApiError{resp.StatusCode, code, message, bodyRetryable}
		}
		var result map[string]any
		json.Unmarshal(respBody, &result)
		return result, nil
	}
	return nil, &ApiError{0, "retries_exhausted", "Exhausted retries", nil}
}

static final Set<Integer> RETRYABLE = Set.of(500, 502, 503, 504); // fallback when the body carries no retryable flag
static final int MAX_RETRIES = 4;

static class ApiError extends RuntimeException {
    final Integer status;
    final String code;
    final Boolean retryable;
    ApiError(Integer status, String code, String message) {
        this(status, code, message, null);
    }
    ApiError(Integer status, String code, String message, Boolean retryable) {
        super(message);
        this.status = status;
        this.code = code;
        this.retryable = retryable;
    }
}

record ParsedError(String code, String message, Boolean retryable) {}

// Read the error object's retryable boolean, if present.
static Boolean retryable(JsonObject obj) {
    JsonElement value = obj.get("retryable");
    return value != null && value.isJsonPrimitive() && value.getAsJsonPrimitive().isBoolean()
        ? value.getAsBoolean() : null;
}

static ParsedError parseError(int status, String body) {
    String reason = switch (status) {
        case 400 -> "Bad Request"; case 403 -> "Forbidden"; case 404 -> "Not Found";
        default -> "HTTP " + status;
    };
    JsonObject obj = GSON.fromJson(body, JsonObject.class);
    JsonElement detail = obj == null ? null : obj.get("detail");
    if (detail != null && detail.isJsonObject()) {
        JsonObject d = detail.getAsJsonObject();
        return new ParsedError(jsonString(d, "code"), orReason(jsonString(d, "message"), reason), retryable(d));
    }
    if (detail != null && detail.isJsonPrimitive()) return new ParsedError(null, detail.getAsString(), null);
    JsonElement error = obj == null ? null : obj.get("error");
    if (error != null && error.isJsonObject()) {
        JsonObject e = error.getAsJsonObject();
        return new ParsedError(jsonString(e, "code"), orReason(jsonString(e, "message"), reason), retryable(e));
    }
    return new ParsedError(jsonString(obj, "error"), orReason(jsonString(obj, "message"), reason), null);
}

static JsonObject request(String method, String url, String token, String body)
        throws IOException, InterruptedException {
    for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
        HttpRequest req = HttpRequest.newBuilder()
            .uri(URI.create(url))
            .header("Authorization", "Bearer " + token)
            .header("Content-Type", "application/json")
            .method(method, body != null
                ? HttpRequest.BodyPublishers.ofString(body)
                : HttpRequest.BodyPublishers.noBody())
            .build();
        HttpResponse<String> resp = HTTP.send(req, HttpResponse.BodyHandlers.ofString());

        if (resp.statusCode() >= 400) {
            ParsedError parsed = parseError(resp.statusCode(), resp.body());
            // error.retryable is authoritative; the status set is only the fallback
            // when the response carries no JSON error body.
            boolean retry = parsed.retryable() != null
                ? parsed.retryable()
                : RETRYABLE.contains(resp.statusCode());
            if (retry && attempt < MAX_RETRIES) {
                int delay = 1 << attempt;
                System.out.printf("    Attempt %d failed (%d); retrying in %ds%n", attempt + 1, resp.statusCode(), delay);
                Thread.sleep(delay * 1000L);
                continue;
            }
            throw new ApiError(resp.statusCode(), parsed.code(), parsed.message(), parsed.retryable());
        }
        return GSON.fromJson(resp.body(), JsonObject.class);
    }
    throw new ApiError(null, "retries_exhausted", "Exhausted retries", null);
}

These snippets are simplified for readability. The cookbook client also retries network-level failures, and applies the same retry policy to multipart uploads. Both implement the same patterns.

Step 2: Interpret API Error Responses

With request and parse_error in place, catch the normalized error and read its status, code, and message. The calls below deliberately trigger three common failures: a missing item (404), a malformed ID (400), and a rejected token (403).

import uuid

cases = [
    ("Missing item (random UUID)", f"{ITEMS_URL}/{uuid.uuid4()}", token),
    ("Malformed item ID", f"{ITEMS_URL}/not-a-valid-uuid", token),
    ("Rejected bearer token", f"{ITEMS_URL}/{uuid.uuid4()}", "invalid-token"),
]
for label, url, tok in cases:
    try:
        request("GET", url, tok)
        print(f"  {label}: unexpectedly succeeded")
    except ApiError as e:
        print(f"  {label}: status={e.status} code={e.code!r} message={e.message!r}")

import { randomUUID } from "node:crypto";

const cases = [
  ["Missing item (random UUID)", `${ITEMS_URL}/${randomUUID()}`, token],
  ["Malformed item ID", `${ITEMS_URL}/not-a-valid-uuid`, token],
  ["Rejected bearer token", `${ITEMS_URL}/${randomUUID()}`, "invalid-token"],
];
for (const [label, url, tok] of cases) {
  try {
    await request("GET", url, tok);
    console.log(`  ${label}: unexpectedly succeeded`);
  } catch (e) {
    console.log(`  ${label}: status=${e.status} code='${e.code}' message='${e.message}'`);
  }
}

import { randomUUID } from "node:crypto";

const cases: Array<[string, string, string]> = [
  ["Missing item (random UUID)", `${ITEMS_URL}/${randomUUID()}`, token],
  ["Malformed item ID", `${ITEMS_URL}/not-a-valid-uuid`, token],
  ["Rejected bearer token", `${ITEMS_URL}/${randomUUID()}`, "invalid-token"],
];
for (const [label, url, tok] of cases) {
  try {
    await request("GET", url, tok);
    console.log(`  ${label}: unexpectedly succeeded`);
  } catch (e) {
    if (e instanceof ApiError) {
      console.log(`  ${label}: status=${e.status} code='${e.code}' message='${e.message}'`);
    }
  }
}

$cases = [
    ['Missing item (random UUID)', ITEMS_URL . '/' . guidv4(), $token],
    ['Malformed item ID', ITEMS_URL . '/not-a-valid-uuid', $token],
    ['Rejected bearer token', ITEMS_URL . '/' . guidv4(), 'invalid-token'],
];
foreach ($cases as [$label, $url, $tok]) {
    try {
        request('GET', $url, $tok);
        echo "  $label: unexpectedly succeeded\n";
    } catch (ApiError $e) {
        echo "  $label: status={$e->status} code='{$e->apiCode}' message='{$e->getMessage()}'\n";
    }
}

cases := []struct{ label, url, token string }{
	{"Missing item (random UUID)", itemsURL + "/" + newUUID(), token},
	{"Malformed item ID", itemsURL + "/not-a-valid-uuid", token},
	{"Rejected bearer token", itemsURL + "/" + newUUID(), "invalid-token"},
}
for _, tc := range cases {
	_, err := request("GET", tc.url, tc.token, nil)
	if err == nil {
		fmt.Printf("  %s: unexpectedly succeeded\n", tc.label)
		continue
	}
	var e *ApiError
	if errors.As(err, &e) {
		fmt.Printf("  %s: status=%d code='%s' message='%s'\n", tc.label, e.Status, e.Code, e.Message)
	}
}

record Case(String label, String url, String token) {}
List<Case> cases = List.of(
    new Case("Missing item (random UUID)", ITEMS_URL + "/" + UUID.randomUUID(), token),
    new Case("Malformed item ID", ITEMS_URL + "/not-a-valid-uuid", token),
    new Case("Rejected bearer token", ITEMS_URL + "/" + UUID.randomUUID(), "invalid-token"));
for (Case tc : cases) {
    try {
        request("GET", tc.url(), tc.token(), null);
        System.out.println("  " + tc.label() + ": unexpectedly succeeded");
    } catch (ApiError e) {
        System.out.printf("  %s: status=%s code='%s' message='%s'%n",
            tc.label(), e.status, e.code, e.getMessage());
    }
}

What You’ll See

  Missing item (random UUID): status=404 code='Item not found' message='The requested item does not exist or has been permanently deleted'
  Malformed item ID: status=400 code='Invalid item ID format' message='Item ID must be a valid UUID format'
  Rejected bearer token: status=403 code='Forbidden - insufficient permissions' message='Forbidden'

A rejected or malformed API key returns 403, not 401. A 403 can also be a legitimate permission denial (for example, an item that belongs to another publisher), so the client does not retry it. API keys are sent as-is on every request, so there is no token to refresh; if you get a 401 or 403, check that the key is correct and has not been revoked.On a legacy client-credentials integration, a genuinely expired access token typically returns 401, and the fix is to fetch a new token once and retry (not on 403). Migrating to an API key removes that step; see Legacy: client credentials and OAuth.

Step 3: Retry Transient Failures

The retry loop from Step 1 retries transient failures (a 503, a dropped connection) instead of surfacing them. Here it is in isolation, recovering from a service that fails twice before succeeding.

A healthy API won’t return a 5xx on demand, so this example simulates one to exercise the backoff path. In production the same path handles real 5xx responses and network errors.

RETRYABLE_DELAYS = [2 ** i for i in range(MAX_RETRIES)]

calls = 0
def flaky():
    global calls
    calls += 1
    if calls < 3:
        raise ApiError(503, "service_unavailable", "Service temporarily unavailable")
    return {"ok": True}

for attempt in range(MAX_RETRIES + 1):
    try:
        flaky()
        break
    except ApiError as e:
        retry = e.retryable if e.retryable is not None else e.status in RETRYABLE
        if retry and attempt < MAX_RETRIES:
            delay = RETRYABLE_DELAYS[attempt]
            print(f"    Attempt {attempt + 1} failed ({e.status}: {e.code}); retrying in {delay}s")
            time.sleep(delay)
        else:
            raise
print(f"  Succeeded after {calls} attempts")

let calls = 0;
const flaky = () => {
  calls += 1;
  if (calls < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
  return { ok: true };
};

for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
  try {
    flaky();
    break;
  } catch (e) {
    const retry = e.retryable !== null ? e.retryable : RETRYABLE.has(e.status);
    if (retry && attempt < MAX_RETRIES) {
      const delay = 2 ** attempt;
      console.log(`    Attempt ${attempt + 1} failed (${e.status}: ${e.code}); retrying in ${delay}s`);
      await sleep(delay * 1000);
    } else throw e;
  }
}
console.log(`  Succeeded after ${calls} attempts`);

let calls = 0;
const flaky = (): { ok: boolean } => {
  calls += 1;
  if (calls < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
  return { ok: true };
};

for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
  try {
    flaky();
    break;
  } catch (e) {
    if (e instanceof ApiError) {
      const retry = e.retryable !== null ? e.retryable : e.status !== null && RETRYABLE.has(e.status);
      if (retry && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${e.status}: ${e.code}); retrying in ${delay}s`);
        await sleep(delay * 1000);
      } else throw e;
    } else throw e;
  }
}
console.log(`  Succeeded after ${calls} attempts`);

$calls = 0;
$flaky = function () use (&$calls) {
    $calls++;
    if ($calls < 3) throw new ApiError(503, 'service_unavailable', 'Service temporarily unavailable');
    return ['ok' => true];
};

for ($attempt = 0; $attempt <= MAX_RETRIES; $attempt++) {
    try {
        $flaky();
        break;
    } catch (ApiError $e) {
        $retry = $e->retryable ?? in_array($e->status, RETRYABLE, true);
        if ($retry && $attempt < MAX_RETRIES) {
            $delay = 2 ** $attempt;
            echo "    Attempt " . ($attempt + 1) . " failed ({$e->status}: {$e->apiCode}); retrying in {$delay}s\n";
            sleep($delay);
        } else {
            throw $e;
        }
    }
}
echo "  Succeeded after $calls attempts\n";

calls := 0
flaky := func() error {
	calls++
	if calls < 3 {
		return &ApiError{Status: 503, Code: "service_unavailable", Message: "Service temporarily unavailable"}
	}
	return nil
}

for attempt := 0; attempt <= maxRetries; attempt++ {
	err := flaky()
	if err == nil {
		break
	}
	var e *ApiError
	retry := errors.As(err, &e) && retryable[e.Status]
	if e != nil && e.Retryable != nil {
		retry = *e.Retryable
	}
	if retry && attempt < maxRetries {
		delay := 1 << attempt
		fmt.Printf("    Attempt %d failed (%d: %s); retrying in %ds\n", attempt+1, e.Status, e.Code, delay)
		time.Sleep(time.Duration(delay) * time.Second)
	} else {
		log.Fatal(err)
	}
}
fmt.Printf("  Succeeded after %d attempts\n", calls)

int[] calls = {0};
Runnable flaky = () -> {
    calls[0]++;
    if (calls[0] < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
};

for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    try {
        flaky.run();
        break;
    } catch (ApiError e) {
        boolean retry = e.retryable != null
            ? e.retryable
            : e.status != null && RETRYABLE.contains(e.status);
        if (retry && attempt < MAX_RETRIES) {
            int delay = 1 << attempt;
            System.out.printf("    Attempt %d failed (%d: %s); retrying in %ds%n",
                attempt + 1, e.status, e.code, delay);
            Thread.sleep(delay * 1000L);
        } else {
            throw e;
        }
    }
}
System.out.printf("  Succeeded after %d attempts%n", calls[0]);

What You’ll See

    Attempt 1 failed (503: service_unavailable); retrying in 1s
    Attempt 2 failed (503: service_unavailable); retrying in 2s
  Succeeded after 3 attempts

Step 4: Verify Ingestion Health

With no health endpoint, you verify by checking the items you care about. Upload a batch, wait for indexing, then fetch each item’s status and roll it up into a summary, treating a 404 as “not found” rather than an error so one missing item doesn’t abort the check. The example deliberately includes a random ID to show that path.

# item_ids: uploaded and indexed via the resilient client (see the cookbook)
to_check = item_ids + [str(uuid.uuid4())]
summary = {"completed": 0, "pending": 0, "failed": 0, "not_found": 0}
for item_id in to_check:
    try:
        status = request("GET", f"{ITEMS_URL}/{item_id}", token).get("status", "").upper()
        if status == "COMPLETED":
            summary["completed"] += 1
        elif status in ("FAILED", "ERROR"):
            summary["failed"] += 1
        else:
            summary["pending"] += 1
    except ApiError as e:
        if e.status == 404:
            summary["not_found"] += 1
        else:
            raise
print(f"  Health: {summary['completed']} completed, {summary['pending']} pending, "
      f"{summary['failed']} failed, {summary['not_found']} not found")

// itemIds: uploaded and indexed via the resilient client (see the cookbook)
const toCheck = [...itemIds, randomUUID()];
const summary = { completed: 0, pending: 0, failed: 0, notFound: 0 };
for (const itemId of toCheck) {
  try {
    const status = ((await request("GET", `${ITEMS_URL}/${itemId}`, token)).status ?? "").toUpperCase();
    if (status === "COMPLETED") summary.completed += 1;
    else if (["FAILED", "ERROR"].includes(status)) summary.failed += 1;
    else summary.pending += 1;
  } catch (e) {
    if (e.status === 404) summary.notFound += 1;
    else throw e;
  }
}
console.log(
  `  Health: ${summary.completed} completed, ${summary.pending} pending, ` +
    `${summary.failed} failed, ${summary.notFound} not found`
);

// itemIds: uploaded and indexed via the resilient client (see the cookbook)
const toCheck = [...itemIds, randomUUID()];
const summary = { completed: 0, pending: 0, failed: 0, notFound: 0 };
for (const itemId of toCheck) {
  try {
    const status = ((await request("GET", `${ITEMS_URL}/${itemId}`, token)).status ?? "").toUpperCase();
    if (status === "COMPLETED") summary.completed += 1;
    else if (["FAILED", "ERROR"].includes(status)) summary.failed += 1;
    else summary.pending += 1;
  } catch (e) {
    if (e instanceof ApiError && e.status === 404) summary.notFound += 1;
    else throw e;
  }
}
console.log(
  `  Health: ${summary.completed} completed, ${summary.pending} pending, ` +
    `${summary.failed} failed, ${summary.notFound} not found`
);

// $itemIds: uploaded and indexed via the resilient client (see the cookbook)
$toCheck = array_merge($itemIds, [guidv4()]);
$summary = ['completed' => 0, 'pending' => 0, 'failed' => 0, 'not_found' => 0];
foreach ($toCheck as $itemId) {
    try {
        $status = strtoupper(request('GET', ITEMS_URL . '/' . $itemId, $token)['status'] ?? '');
        if ($status === 'COMPLETED') $summary['completed']++;
        elseif (in_array($status, ['FAILED', 'ERROR'], true)) $summary['failed']++;
        else $summary['pending']++;
    } catch (ApiError $e) {
        if ($e->status === 404) $summary['not_found']++;
        else throw $e;
    }
}
echo "  Health: {$summary['completed']} completed, {$summary['pending']} pending, "
    . "{$summary['failed']} failed, {$summary['not_found']} not found\n";

// itemIDs: uploaded and indexed via the resilient client (see the cookbook)
toCheck := append(append([]string{}, itemIDs...), newUUID())
completed, pending, failed, notFound := 0, 0, 0, 0
for _, id := range toCheck {
	item, err := request("GET", itemsURL+"/"+id, token, nil)
	if err != nil {
		var e *ApiError
		if errors.As(err, &e) && e.Status == 404 {
			notFound++
			continue
		}
		log.Fatal(err)
	}
	switch strings.ToUpper(str(item["status"])) {
	case "COMPLETED":
		completed++
	case "FAILED", "ERROR":
		failed++
	default:
		pending++
	}
}
fmt.Printf("  Health: %d completed, %d pending, %d failed, %d not found\n",
	completed, pending, failed, notFound)

// itemIds: uploaded and indexed via the resilient client (see the cookbook)
List<String> toCheck = new ArrayList<>(itemIds);
toCheck.add(UUID.randomUUID().toString());
int completed = 0, pending = 0, failed = 0, notFound = 0;
for (String itemId : toCheck) {
    try {
        String status = request("GET", ITEMS_URL + "/" + itemId, token, null)
            .get("status").getAsString().toUpperCase();
        if (status.equals("COMPLETED")) completed++;
        else if (status.equals("FAILED") || status.equals("ERROR")) failed++;
        else pending++;
    } catch (ApiError e) {
        if (e.status != null && e.status == 404) notFound++;
        else throw e;
    }
}
System.out.printf("  Health: %d completed, %d pending, %d failed, %d not found%n",
    completed, pending, failed, notFound);

What You’ll See

  Health: 2 completed, 0 pending, 0 failed, 1 not found

The two uploaded items report COMPLETED; the random ID is counted as not found instead of throwing.

Run the Complete Example

The cookbook ties it together (a resilient client that handles errors, retries, and a full upload → verify → clean-up health check) in all six languages. From the cookbook repository, install dependencies, copy .env.example to .env and add your credentials, then run it:

cd rag-pipeline-part-3/python
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env       # then add your API key and Publisher ID
python main.py

cd rag-pipeline-part-3/javascript
npm install
cp .env.example .env       # then add your API key and Publisher ID
npm start

cd rag-pipeline-part-3/typescript
npm install
cp .env.example .env       # then add your API key and Publisher ID
npm start

cd rag-pipeline-part-3/php
composer install
cp .env.example .env       # then add your API key and Publisher ID
php index.php

cd rag-pipeline-part-3/go
go mod tidy
cp .env.example .env       # then add your API key and Publisher ID
go run main.go

cd rag-pipeline-part-3/java
mvn compile
cp .env.example .env       # then add your API key and Publisher ID
mvn -q exec:java

You’ll see:

Step 1: Resilient client ready (error parsing, retry/backoff).

Step 2: Interpreting API error responses...
  Missing item (random UUID): status=404 code='Item not found' message='The requested item does not exist or has been permanently deleted'
  Malformed item ID: status=400 code='Invalid item ID format' message='Item ID must be a valid UUID format'
  Rejected bearer token: status=403 code='Forbidden - insufficient permissions' message='Forbidden'

Step 3: Retrying transient failures with backoff...
    Attempt 1 failed (503: service_unavailable); retrying in 1s
    Attempt 2 failed (503: service_unavailable); retrying in 2s
  Succeeded after 3 attempts

Step 4: Verifying ingestion health...
  Uploading and indexing a batch...
  Waiting for 2 item(s) to finish indexing...
  Health: 2 completed, 0 pending, 0 failed, 1 not found
  Cleaned up 2 item(s)

Done. The resilient client handled errors, retries, and verification end to end.

Working Code Sample

View Complete Code

Clone or browse the complete resilient client for all 6 languages (JavaScript, TypeScript, Python, PHP, Go, Java) with setup instructions and the sample content files.

Troubleshooting

A request fails immediately instead of retrying

Retry only when the error’s retryable flag is true; when it is false, fail fast regardless of status (a 500 such as 3001 INTERNAL_ERROR is not retried). Client errors (400, 401, 403, 404, 422) are not retryable and fail fast. The status fallback (500/502/503/504) applies only to responses whose error body is missing or unparseable.

Retries never stop / take too long

Check your backoff cap. With base-2 exponential backoff and MAX_RETRIES = 4, the waits are 1s, 2s, 4s, 8s. Lower MAX_RETRIES or cap the delay for latency-sensitive paths.

Empty error message

Some error responses carry a code but no message. The parser falls back to the HTTP reason phrase (for example, Forbidden) so you always have something to log.

403 on a token you believe is valid

A malformed or rejected token returns 403. Re-check your API key and confirm it has not been revoked. See the Authentication Tutorial.

Next Steps

That completes the Build an End-to-End RAG Pipeline series. You’ve set up a publisher and ingested content (Part 1), managed its lifecycle (Part 2), and made the integration resilient (Part 3). From here:

Building Custom Search: surface your content to users with the Search API
Grounded Completions: answer questions from your content with citations (set rag_publisher to your publisher name, the value the grounded guides document, rather than the Publisher ID used for ingestion)
Fold the resilient client into your own service for consistent error handling and retries on every Gloo AI call.

To learn how to judge the answers your pipeline produces, read When RAG Works and When It Doesn’t in the Learning Center.

Get Started

Responses & Completions

Production

Content

Search & Recommendations

Embeddings

Legacy

Part 3: Verification, Error Handling & Resilience

Pipeline at a Glance

Prerequisites

Step 1: Parse Errors and Retry

Step 2: Interpret API Error Responses

What You’ll See

Step 3: Retry Transient Failures

What You’ll See

Step 4: Verify Ingestion Health

What You’ll See

Run the Complete Example

Working Code Sample

View Complete Code

Troubleshooting

A request fails immediately instead of retrying

Retries never stop / take too long

Empty error message

403 on a token you believe is valid

Next Steps

​Pipeline at a Glance

​Prerequisites

​Step 1: Parse Errors and Retry

​Step 2: Interpret API Error Responses

​What You’ll See

​Step 3: Retry Transient Failures

​What You’ll See

​Step 4: Verify Ingestion Health

​What You’ll See

​Run the Complete Example

​Working Code Sample

View Complete Code

​Troubleshooting

​A request fails immediately instead of retrying

​Retries never stop / take too long

​Empty error message

​403 on a token you believe is valid

​Next Steps

Pipeline at a Glance

Prerequisites

Step 1: Parse Errors and Retry

Step 2: Interpret API Error Responses

What You’ll See

Step 3: Retry Transient Failures

What You’ll See

Step 4: Verify Ingestion Health

What You’ll See

Run the Complete Example

Working Code Sample

Troubleshooting

A request fails immediately instead of retrying

Retries never stop / take too long

Empty error message

403 on a token you believe is valid

Next Steps