Part 3: Verification, Error Handling & Resilience

This is Part 3 of the Build an End-to-End RAG Pipeline series. Parts 1 and 2 walked the happy path. Production code can’t assume it: requests fail, services blip, and you need to confirm that an operation actually took effect. This part builds a small resilient client and uses it to interpret API errors, retry transient failures, and verify ingestion health.

Gloo AI does not include a monitoring or health-check endpoint. Resilience is built from the same item APIs you’ve already used, plus disciplined error handling on the client side.

Pipeline at a Glance

Publisher setup (Studio)

Create the publisher that owns your content — Part 1.

Ingest content with metadata

Upload files and enrich them — Part 1.

Verify indexing

Poll item status until your content is searchable — Part 1.

Semantic search

Query your content — deep dive: Building Custom Search.

Grounded completions with sources

Answer questions from your content with citations — deep dive: Grounded Completions with RAG.

Content lifecycle

Update, bulk-edit, and delete content — Part 2.

Verification, errors & resilience

Error handling and retry patterns — covered below.

Prerequisites

Before starting, ensure you have:

A Gloo AI Studio account
Your Client ID and Client Secret from the API Credentials page
A publisher with credentials — see Part 1
Familiarity with the item APIs from Part 1 and Part 2

Step 1: Parse Errors and Retry

A resilient client does two things on every request: it turns failures into a normalized error (status, code, message) so callers can react to them, and it retries only transient failures — server errors (500, 502, 503, 504) and network blips — with exponential backoff. Client errors (400, 401, 403, 404, 422) are bugs in the request, not blips, so they fail fast. The Data Engine returns a few error shapes — {"detail": {"code", "message"}}, {"detail": "..."}, and {"error", "message"} — so the parser normalizes all of them.

import time
import requests

RETRYABLE = {500, 502, 503, 504}
MAX_RETRIES = 4


class ApiError(Exception):
    def __init__(self, status, code, message):
        super().__init__(f"[{status} {code}] {message}")
        self.status, self.code, self.message = status, code, message


def parse_error(response):
    try:
        body = response.json()
    except ValueError:
        return None, response.reason
    detail = body.get("detail") if isinstance(body, dict) else None
    if isinstance(detail, dict):
        return detail.get("code"), detail.get("message") or response.reason
    if isinstance(detail, str):
        return None, detail
    return body.get("error"), body.get("message") or response.reason


def request(method, url, token, **kwargs):
    for attempt in range(MAX_RETRIES + 1):
        response = requests.request(
            method, url, headers={"Authorization": f"Bearer {token}"}, timeout=30, **kwargs
        )
        if response.status_code in RETRYABLE and attempt < MAX_RETRIES:
            delay = 2 ** attempt
            print(f"    Attempt {attempt + 1} failed ({response.status_code}); retrying in {delay}s")
            time.sleep(delay)
            continue
        if not response.ok:
            code, message = parse_error(response)
            raise ApiError(response.status_code, code, message)
        return response.json()

const RETRYABLE = new Set([500, 502, 503, 504]);
const MAX_RETRIES = 4;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

class ApiError extends Error {
  constructor(status, code, message) {
    super(message);
    this.status = status;
    this.code = code;
  }
}

async function parseError(response) {
  const text = await response.text();
  let body;
  try {
    body = JSON.parse(text);
  } catch {
    return [null, response.statusText];
  }
  const detail = body?.detail;
  if (detail && typeof detail === "object") return [detail.code ?? null, detail.message || response.statusText];
  if (typeof detail === "string") return [null, detail];
  return [body?.error ?? null, body?.message || response.statusText];
}

async function request(method, url, token, body) {
  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    const response = await fetch(url, {
      method,
      headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
      body: body ? JSON.stringify(body) : undefined,
    });
    if (RETRYABLE.has(response.status) && attempt < MAX_RETRIES) {
      const delay = 2 ** attempt;
      console.log(`    Attempt ${attempt + 1} failed (${response.status}); retrying in ${delay}s`);
      await sleep(delay * 1000);
      continue;
    }
    if (!response.ok) {
      const [code, message] = await parseError(response);
      throw new ApiError(response.status, code, message);
    }
    return response.json();
  }
}

const RETRYABLE = new Set([500, 502, 503, 504]);
const MAX_RETRIES = 4;
const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));

class ApiError extends Error {
  constructor(readonly status: number | null, readonly code: string | null, message: string) {
    super(message);
  }
}

async function parseError(response: Response): Promise<[string | null, string]> {
  const text = await response.text();
  let body: any;
  try {
    body = JSON.parse(text);
  } catch {
    return [null, response.statusText];
  }
  const detail = body?.detail;
  if (detail && typeof detail === "object") return [detail.code ?? null, detail.message || response.statusText];
  if (typeof detail === "string") return [null, detail];
  return [body?.error ?? null, body?.message || response.statusText];
}

async function request(method: string, url: string, token: string, body?: unknown): Promise<any> {
  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    const response = await fetch(url, {
      method,
      headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
      body: body ? JSON.stringify(body) : undefined,
    });
    if (RETRYABLE.has(response.status) && attempt < MAX_RETRIES) {
      const delay = 2 ** attempt;
      console.log(`    Attempt ${attempt + 1} failed (${response.status}); retrying in ${delay}s`);
      await sleep(delay * 1000);
      continue;
    }
    if (!response.ok) {
      const [code, message] = await parseError(response);
      throw new ApiError(response.status, code, message);
    }
    return response.json();
  }
}

<?php

const RETRYABLE = [500, 502, 503, 504];
const MAX_RETRIES = 4;

class ApiError extends RuntimeException {
    public function __construct(public ?int $status, public ?string $apiCode, string $message) {
        parent::__construct($message);
    }
}

function parseError(int $status, string $body): array {
    $reasons = [400 => 'Bad Request', 403 => 'Forbidden', 404 => 'Not Found', 500 => 'Internal Server Error'];
    $reason = $reasons[$status] ?? "HTTP $status";
    $decoded = json_decode($body, true);
    $detail = is_array($decoded) ? ($decoded['detail'] ?? null) : null;
    if (is_array($detail)) return [$detail['code'] ?? null, ($detail['message'] ?? '') ?: $reason];
    if (is_string($detail)) return [null, $detail];
    return [$decoded['error'] ?? null, ($decoded['message'] ?? '') ?: $reason];
}

function request(string $method, string $url, string $token, ?array $body = null) {
    for ($attempt = 0; $attempt <= MAX_RETRIES; $attempt++) {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_CUSTOMREQUEST => $method,
            CURLOPT_HTTPHEADER => ["Authorization: Bearer $token", 'Content-Type: application/json'],
            CURLOPT_POSTFIELDS => $body !== null ? json_encode($body) : null,
            CURLOPT_TIMEOUT => 30,
        ]);
        $responseBody = (string) curl_exec($ch);
        $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        if (in_array($status, RETRYABLE, true) && $attempt < MAX_RETRIES) {
            $delay = 2 ** $attempt;
            echo "    Attempt " . ($attempt + 1) . " failed ($status); retrying in {$delay}s\n";
            sleep($delay);
            continue;
        }
        if ($status >= 400) {
            [$code, $message] = parseError($status, $responseBody);
            throw new ApiError($status, $code, $message);
        }
        return json_decode($responseBody, true);
    }
}

var retryable = map[int]bool{500: true, 502: true, 503: true, 504: true}

const maxRetries = 4

type ApiError struct {
	Status  int
	Code    string
	Message string
}

func (e *ApiError) Error() string { return fmt.Sprintf("[%d %s] %s", e.Status, e.Code, e.Message) }

func parseError(status int, body []byte) (string, string) {
	reason := http.StatusText(status)
	var decoded map[string]any
	if json.Unmarshal(body, &decoded) != nil {
		return "", reason
	}
	if detail, ok := decoded["detail"].(map[string]any); ok {
		return str(detail["code"]), orReason(str(detail["message"]), reason)
	}
	if detail, ok := decoded["detail"].(string); ok {
		return "", detail
	}
	return str(decoded["error"]), orReason(str(decoded["message"]), reason)
}

func request(method, url, token string, body []byte) (map[string]any, error) {
	for attempt := 0; attempt <= maxRetries; attempt++ {
		var reader io.Reader
		if body != nil {
			reader = bytes.NewReader(body)
		}
		req, _ := http.NewRequest(method, url, reader)
		req.Header.Set("Authorization", "Bearer "+token)
		req.Header.Set("Content-Type", "application/json")
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			return nil, &ApiError{0, "network_error", err.Error()}
		}
		respBody, _ := io.ReadAll(resp.Body)
		resp.Body.Close()

		if retryable[resp.StatusCode] && attempt < maxRetries {
			delay := 1 << attempt
			fmt.Printf("    Attempt %d failed (%d); retrying in %ds\n", attempt+1, resp.StatusCode, delay)
			time.Sleep(time.Duration(delay) * time.Second)
			continue
		}
		if resp.StatusCode >= 400 {
			code, message := parseError(resp.StatusCode, respBody)
			return nil, &ApiError{resp.StatusCode, code, message}
		}
		var result map[string]any
		json.Unmarshal(respBody, &result)
		return result, nil
	}
	return nil, &ApiError{0, "retries_exhausted", "Exhausted retries"}
}

static final Set<Integer> RETRYABLE = Set.of(500, 502, 503, 504);
static final int MAX_RETRIES = 4;

static class ApiError extends RuntimeException {
    final Integer status;
    final String code;
    ApiError(Integer status, String code, String message) {
        super(message);
        this.status = status;
        this.code = code;
    }
}

static String[] parseError(int status, String body) {
    String reason = switch (status) {
        case 400 -> "Bad Request"; case 403 -> "Forbidden"; case 404 -> "Not Found";
        default -> "HTTP " + status;
    };
    JsonObject obj = GSON.fromJson(body, JsonObject.class);
    JsonElement detail = obj == null ? null : obj.get("detail");
    if (detail != null && detail.isJsonObject()) {
        JsonObject d = detail.getAsJsonObject();
        return new String[] {jsonString(d, "code"), orReason(jsonString(d, "message"), reason)};
    }
    if (detail != null && detail.isJsonPrimitive()) return new String[] {null, detail.getAsString()};
    return new String[] {jsonString(obj, "error"), orReason(jsonString(obj, "message"), reason)};
}

static JsonObject request(String method, String url, String token, String body)
        throws IOException, InterruptedException {
    for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
        HttpRequest req = HttpRequest.newBuilder()
            .uri(URI.create(url))
            .header("Authorization", "Bearer " + token)
            .header("Content-Type", "application/json")
            .method(method, body != null
                ? HttpRequest.BodyPublishers.ofString(body)
                : HttpRequest.BodyPublishers.noBody())
            .build();
        HttpResponse<String> resp = HTTP.send(req, HttpResponse.BodyHandlers.ofString());

        if (RETRYABLE.contains(resp.statusCode()) && attempt < MAX_RETRIES) {
            int delay = 1 << attempt;
            System.out.printf("    Attempt %d failed (%d); retrying in %ds%n", attempt + 1, resp.statusCode(), delay);
            Thread.sleep(delay * 1000L);
            continue;
        }
        if (resp.statusCode() >= 400) {
            String[] parsed = parseError(resp.statusCode(), resp.body());
            throw new ApiError(resp.statusCode(), parsed[0], parsed[1]);
        }
        return GSON.fromJson(resp.body(), JsonObject.class);
    }
    throw new ApiError(null, "retries_exhausted", "Exhausted retries");
}

These snippets are simplified for readability. The cookbook client also refreshes the access token once on a 401, retries network-level failures, and applies the same retry policy to multipart uploads. Both implement the same patterns.

Step 2: Interpret API Error Responses

With request and parse_error in place, error handling becomes uniform: catch the normalized error and read its status, code, and message. The calls below deliberately trigger three common failures — a missing item (404), a malformed ID (400), and a rejected token (403).

import uuid

cases = [
    ("Missing item (random UUID)", f"{ITEMS_URL}/{uuid.uuid4()}", token),
    ("Malformed item ID", f"{ITEMS_URL}/not-a-valid-uuid", token),
    ("Rejected bearer token", f"{ITEMS_URL}/{uuid.uuid4()}", "invalid-token"),
]
for label, url, tok in cases:
    try:
        request("GET", url, tok)
        print(f"  {label}: unexpectedly succeeded")
    except ApiError as e:
        print(f"  {label}: status={e.status} code={e.code!r} message={e.message!r}")

import { randomUUID } from "node:crypto";

const cases = [
  ["Missing item (random UUID)", `${ITEMS_URL}/${randomUUID()}`, token],
  ["Malformed item ID", `${ITEMS_URL}/not-a-valid-uuid`, token],
  ["Rejected bearer token", `${ITEMS_URL}/${randomUUID()}`, "invalid-token"],
];
for (const [label, url, tok] of cases) {
  try {
    await request("GET", url, tok);
    console.log(`  ${label}: unexpectedly succeeded`);
  } catch (e) {
    console.log(`  ${label}: status=${e.status} code='${e.code}' message='${e.message}'`);
  }
}

import { randomUUID } from "node:crypto";

const cases: Array<[string, string, string]> = [
  ["Missing item (random UUID)", `${ITEMS_URL}/${randomUUID()}`, token],
  ["Malformed item ID", `${ITEMS_URL}/not-a-valid-uuid`, token],
  ["Rejected bearer token", `${ITEMS_URL}/${randomUUID()}`, "invalid-token"],
];
for (const [label, url, tok] of cases) {
  try {
    await request("GET", url, tok);
    console.log(`  ${label}: unexpectedly succeeded`);
  } catch (e) {
    if (e instanceof ApiError) {
      console.log(`  ${label}: status=${e.status} code='${e.code}' message='${e.message}'`);
    }
  }
}

$cases = [
    ['Missing item (random UUID)', ITEMS_URL . '/' . guidv4(), $token],
    ['Malformed item ID', ITEMS_URL . '/not-a-valid-uuid', $token],
    ['Rejected bearer token', ITEMS_URL . '/' . guidv4(), 'invalid-token'],
];
foreach ($cases as [$label, $url, $tok]) {
    try {
        request('GET', $url, $tok);
        echo "  $label: unexpectedly succeeded\n";
    } catch (ApiError $e) {
        echo "  $label: status={$e->status} code='{$e->apiCode}' message='{$e->getMessage()}'\n";
    }
}

cases := []struct{ label, url, token string }{
	{"Missing item (random UUID)", itemsURL + "/" + newUUID(), token},
	{"Malformed item ID", itemsURL + "/not-a-valid-uuid", token},
	{"Rejected bearer token", itemsURL + "/" + newUUID(), "invalid-token"},
}
for _, tc := range cases {
	_, err := request("GET", tc.url, tc.token, nil)
	if err == nil {
		fmt.Printf("  %s: unexpectedly succeeded\n", tc.label)
		continue
	}
	var e *ApiError
	if errors.As(err, &e) {
		fmt.Printf("  %s: status=%d code='%s' message='%s'\n", tc.label, e.Status, e.Code, e.Message)
	}
}

record Case(String label, String url, String token) {}
List<Case> cases = List.of(
    new Case("Missing item (random UUID)", ITEMS_URL + "/" + UUID.randomUUID(), token),
    new Case("Malformed item ID", ITEMS_URL + "/not-a-valid-uuid", token),
    new Case("Rejected bearer token", ITEMS_URL + "/" + UUID.randomUUID(), "invalid-token"));
for (Case tc : cases) {
    try {
        request("GET", tc.url(), tc.token(), null);
        System.out.println("  " + tc.label() + ": unexpectedly succeeded");
    } catch (ApiError e) {
        System.out.printf("  %s: status=%s code='%s' message='%s'%n",
            tc.label(), e.status, e.code, e.getMessage());
    }
}

What You’ll See

  Missing item (random UUID): status=404 code='Item not found' message='The requested item does not exist or has been permanently deleted'
  Malformed item ID: status=400 code='Invalid item ID format' message='Item ID must be a valid UUID format'
  Rejected bearer token: status=403 code='Forbidden - insufficient permissions' message='Forbidden'

A rejected or malformed token returns 403, not 401. A genuinely expired token typically returns 401 — which is why the cookbook client refreshes the token once on a 401 and retries. It does not refresh on 403, since 403 can be a legitimate permission denial (for example, an item that belongs to another publisher).

Step 3: Retry Transient Failures

Transient failures — a 503, a dropped connection — should be retried, not surfaced. The retry loop from Step 1 already does this; here it is in isolation, recovering from a service that fails twice before succeeding.

A healthy API won’t return a 5xx on demand, so this example simulates a transient failure to exercise the backoff path. In production the same path handles real 5xx responses and network errors.

RETRYABLE_DELAYS = [2 ** i for i in range(MAX_RETRIES)]

calls = 0
def flaky():
    global calls
    calls += 1
    if calls < 3:
        raise ApiError(503, "service_unavailable", "Service temporarily unavailable")
    return {"ok": True}

for attempt in range(MAX_RETRIES + 1):
    try:
        flaky()
        break
    except ApiError as e:
        if e.status in RETRYABLE and attempt < MAX_RETRIES:
            delay = RETRYABLE_DELAYS[attempt]
            print(f"    Attempt {attempt + 1} failed ({e.status}: {e.code}); retrying in {delay}s")
            time.sleep(delay)
        else:
            raise
print(f"  Succeeded after {calls} attempts")

let calls = 0;
const flaky = () => {
  calls += 1;
  if (calls < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
  return { ok: true };
};

for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
  try {
    flaky();
    break;
  } catch (e) {
    if (RETRYABLE.has(e.status) && attempt < MAX_RETRIES) {
      const delay = 2 ** attempt;
      console.log(`    Attempt ${attempt + 1} failed (${e.status}: ${e.code}); retrying in ${delay}s`);
      await sleep(delay * 1000);
    } else throw e;
  }
}
console.log(`  Succeeded after ${calls} attempts`);

let calls = 0;
const flaky = (): { ok: boolean } => {
  calls += 1;
  if (calls < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
  return { ok: true };
};

for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
  try {
    flaky();
    break;
  } catch (e) {
    if (e instanceof ApiError && e.status !== null && RETRYABLE.has(e.status) && attempt < MAX_RETRIES) {
      const delay = 2 ** attempt;
      console.log(`    Attempt ${attempt + 1} failed (${e.status}: ${e.code}); retrying in ${delay}s`);
      await sleep(delay * 1000);
    } else throw e;
  }
}
console.log(`  Succeeded after ${calls} attempts`);

$calls = 0;
$flaky = function () use (&$calls) {
    $calls++;
    if ($calls < 3) throw new ApiError(503, 'service_unavailable', 'Service temporarily unavailable');
    return ['ok' => true];
};

for ($attempt = 0; $attempt <= MAX_RETRIES; $attempt++) {
    try {
        $flaky();
        break;
    } catch (ApiError $e) {
        if (in_array($e->status, RETRYABLE, true) && $attempt < MAX_RETRIES) {
            $delay = 2 ** $attempt;
            echo "    Attempt " . ($attempt + 1) . " failed ({$e->status}: {$e->apiCode}); retrying in {$delay}s\n";
            sleep($delay);
        } else {
            throw $e;
        }
    }
}
echo "  Succeeded after $calls attempts\n";

calls := 0
flaky := func() error {
	calls++
	if calls < 3 {
		return &ApiError{503, "service_unavailable", "Service temporarily unavailable"}
	}
	return nil
}

for attempt := 0; attempt <= maxRetries; attempt++ {
	err := flaky()
	if err == nil {
		break
	}
	var e *ApiError
	if errors.As(err, &e) && retryable[e.Status] && attempt < maxRetries {
		delay := 1 << attempt
		fmt.Printf("    Attempt %d failed (%d: %s); retrying in %ds\n", attempt+1, e.Status, e.Code, delay)
		time.Sleep(time.Duration(delay) * time.Second)
	} else {
		log.Fatal(err)
	}
}
fmt.Printf("  Succeeded after %d attempts\n", calls)

int[] calls = {0};
Runnable flaky = () -> {
    calls[0]++;
    if (calls[0] < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
};

for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    try {
        flaky.run();
        break;
    } catch (ApiError e) {
        if (e.status != null && RETRYABLE.contains(e.status) && attempt < MAX_RETRIES) {
            int delay = 1 << attempt;
            System.out.printf("    Attempt %d failed (%d: %s); retrying in %ds%n",
                attempt + 1, e.status, e.code, delay);
            Thread.sleep(delay * 1000L);
        } else {
            throw e;
        }
    }
}
System.out.printf("  Succeeded after %d attempts%n", calls[0]);

What You’ll See

    Attempt 1 failed (503: service_unavailable); retrying in 1s
    Attempt 2 failed (503: service_unavailable); retrying in 2s
  Succeeded after 3 attempts

Step 4: Verify Ingestion Health

There’s no health endpoint, so you verify by checking the items you care about. Upload a batch, wait for indexing, then fetch each item’s status and roll it up into a summary — treating a 404 as “not found” rather than an error so one missing item doesn’t abort the check. The example deliberately includes a random ID to show that path.

# item_ids: uploaded and indexed via the resilient client (see the cookbook)
to_check = item_ids + [str(uuid.uuid4())]
summary = {"completed": 0, "pending": 0, "failed": 0, "not_found": 0}
for item_id in to_check:
    try:
        status = request("GET", f"{ITEMS_URL}/{item_id}", token).get("status", "").upper()
        if status == "COMPLETED":
            summary["completed"] += 1
        elif status in ("FAILED", "ERROR"):
            summary["failed"] += 1
        else:
            summary["pending"] += 1
    except ApiError as e:
        if e.status == 404:
            summary["not_found"] += 1
        else:
            raise
print(f"  Health: {summary['completed']} completed, {summary['pending']} pending, "
      f"{summary['failed']} failed, {summary['not_found']} not found")

// itemIds: uploaded and indexed via the resilient client (see the cookbook)
const toCheck = [...itemIds, randomUUID()];
const summary = { completed: 0, pending: 0, failed: 0, notFound: 0 };
for (const itemId of toCheck) {
  try {
    const status = ((await request("GET", `${ITEMS_URL}/${itemId}`, token)).status ?? "").toUpperCase();
    if (status === "COMPLETED") summary.completed += 1;
    else if (["FAILED", "ERROR"].includes(status)) summary.failed += 1;
    else summary.pending += 1;
  } catch (e) {
    if (e.status === 404) summary.notFound += 1;
    else throw e;
  }
}
console.log(
  `  Health: ${summary.completed} completed, ${summary.pending} pending, ` +
    `${summary.failed} failed, ${summary.notFound} not found`
);

// itemIds: uploaded and indexed via the resilient client (see the cookbook)
const toCheck = [...itemIds, randomUUID()];
const summary = { completed: 0, pending: 0, failed: 0, notFound: 0 };
for (const itemId of toCheck) {
  try {
    const status = ((await request("GET", `${ITEMS_URL}/${itemId}`, token)).status ?? "").toUpperCase();
    if (status === "COMPLETED") summary.completed += 1;
    else if (["FAILED", "ERROR"].includes(status)) summary.failed += 1;
    else summary.pending += 1;
  } catch (e) {
    if (e instanceof ApiError && e.status === 404) summary.notFound += 1;
    else throw e;
  }
}
console.log(
  `  Health: ${summary.completed} completed, ${summary.pending} pending, ` +
    `${summary.failed} failed, ${summary.notFound} not found`
);

// $itemIds: uploaded and indexed via the resilient client (see the cookbook)
$toCheck = array_merge($itemIds, [guidv4()]);
$summary = ['completed' => 0, 'pending' => 0, 'failed' => 0, 'not_found' => 0];
foreach ($toCheck as $itemId) {
    try {
        $status = strtoupper(request('GET', ITEMS_URL . '/' . $itemId, $token)['status'] ?? '');
        if ($status === 'COMPLETED') $summary['completed']++;
        elseif (in_array($status, ['FAILED', 'ERROR'], true)) $summary['failed']++;
        else $summary['pending']++;
    } catch (ApiError $e) {
        if ($e->status === 404) $summary['not_found']++;
        else throw $e;
    }
}
echo "  Health: {$summary['completed']} completed, {$summary['pending']} pending, "
    . "{$summary['failed']} failed, {$summary['not_found']} not found\n";

// itemIDs: uploaded and indexed via the resilient client (see the cookbook)
toCheck := append(append([]string{}, itemIDs...), newUUID())
completed, pending, failed, notFound := 0, 0, 0, 0
for _, id := range toCheck {
	item, err := request("GET", itemsURL+"/"+id, token, nil)
	if err != nil {
		var e *ApiError
		if errors.As(err, &e) && e.Status == 404 {
			notFound++
			continue
		}
		log.Fatal(err)
	}
	switch strings.ToUpper(str(item["status"])) {
	case "COMPLETED":
		completed++
	case "FAILED", "ERROR":
		failed++
	default:
		pending++
	}
}
fmt.Printf("  Health: %d completed, %d pending, %d failed, %d not found\n",
	completed, pending, failed, notFound)

// itemIds: uploaded and indexed via the resilient client (see the cookbook)
List<String> toCheck = new ArrayList<>(itemIds);
toCheck.add(UUID.randomUUID().toString());
int completed = 0, pending = 0, failed = 0, notFound = 0;
for (String itemId : toCheck) {
    try {
        String status = request("GET", ITEMS_URL + "/" + itemId, token, null)
            .get("status").getAsString().toUpperCase();
        if (status.equals("COMPLETED")) completed++;
        else if (status.equals("FAILED") || status.equals("ERROR")) failed++;
        else pending++;
    } catch (ApiError e) {
        if (e.status != null && e.status == 404) notFound++;
        else throw e;
    }
}
System.out.printf("  Health: %d completed, %d pending, %d failed, %d not found%n",
    completed, pending, failed, notFound);

What You’ll See

  Health: 2 completed, 0 pending, 0 failed, 1 not found

The two uploaded items report COMPLETED; the random ID is counted as not found instead of throwing.

Run the Complete Example

The cookbook ties it together — a resilient client that handles errors, retries, and a full upload → verify → clean-up health check — in all six languages. From the cookbook repository, install dependencies, copy .env.example to .env and add your credentials, then run it:

cd rag-pipeline-part-3/python
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
python main.py

cd rag-pipeline-part-3/javascript
npm install
cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
npm start

cd rag-pipeline-part-3/typescript
npm install
cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
npm start

cd rag-pipeline-part-3/php
composer install
cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
php index.php

cd rag-pipeline-part-3/go
go mod tidy
cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
go run main.go

cd rag-pipeline-part-3/java
mvn compile
cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
mvn -q exec:java

You’ll see:

Step 1: Resilient client ready (token refresh, error parsing, retry/backoff).

Step 2: Interpreting API error responses...
  Missing item (random UUID): status=404 code='Item not found' message='The requested item does not exist or has been permanently deleted'
  Malformed item ID: status=400 code='Invalid item ID format' message='Item ID must be a valid UUID format'
  Rejected bearer token: status=403 code='Forbidden - insufficient permissions' message='Forbidden'

Step 3: Retrying transient failures with backoff...
    Attempt 1 failed (503: service_unavailable); retrying in 1s
    Attempt 2 failed (503: service_unavailable); retrying in 2s
  Succeeded after 3 attempts

Step 4: Verifying ingestion health...
  Uploading and indexing a batch...
  Waiting for 2 item(s) to finish indexing...
  Health: 2 completed, 0 pending, 0 failed, 1 not found
  Cleaned up 2 item(s)

Done. The resilient client handled errors, retries, and verification end to end.

Working Code Sample

View Complete Code

Clone or browse the complete resilient client for all 6 languages (JavaScript, TypeScript, Python, PHP, Go, Java) with setup instructions and the sample content files.

Troubleshooting

A request fails immediately instead of retrying

That’s intended for client errors (400, 401, 403, 404, 422) — they won’t succeed on retry. Only 500/502/503/504 and network failures are retried.

Retries never stop / take too long

Check your backoff cap. With base-2 exponential backoff and MAX_RETRIES = 4, the waits are 1s, 2s, 4s, 8s. Lower MAX_RETRIES or cap the delay for latency-sensitive paths.

Empty error message

Some error responses carry a code but no message. The parser falls back to the HTTP reason phrase (for example, Forbidden) so you always have something to log.

403 on a token you believe is valid

A malformed or rejected token returns 403. Re-check the token; if it’s genuinely expired you’ll typically get a 401, which the cookbook client refreshes automatically. See the Authentication Tutorial.

Next Steps

That completes the Build an End-to-End RAG Pipeline series — you’ve set up a publisher and ingested content (Part 1), managed its lifecycle (Part 2), and made the integration resilient (Part 3). From here:

Building Custom Search — surface your content to users
Grounded Completions with RAG — answer questions from your content with citations
Fold the resilient client into your own service so every Gloo AI call gets consistent error handling and retries.

​Pipeline at a Glance

​Prerequisites

​Step 1: Parse Errors and Retry

​Step 2: Interpret API Error Responses

​What You’ll See

​Step 3: Retry Transient Failures

​What You’ll See

​Step 4: Verify Ingestion Health

​What You’ll See

​Run the Complete Example

​Working Code Sample

View Complete Code

​Troubleshooting

​A request fails immediately instead of retrying

​Retries never stop / take too long

​Empty error message

​403 on a token you believe is valid

​Next Steps

Pipeline at a Glance

Prerequisites

Step 1: Parse Errors and Retry

Step 2: Interpret API Error Responses

What You’ll See

Step 3: Retry Transient Failures

What You’ll See

Step 4: Verify Ingestion Health

What You’ll See

Run the Complete Example

Working Code Sample

Troubleshooting

A request fails immediately instead of retrying

Retries never stop / take too long

Empty error message

403 on a token you believe is valid

Next Steps