> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Part 3: Verification, Error Handling & Resilience

> Harden your Gloo AI integration: interpret API errors, retry transient failures with backoff, and verify ingestion health.

This is **Part 3 of the Build an End-to-End RAG Pipeline series**. Parts 1 and 2 walked the happy path. Production code can't assume it: requests fail, services blip, and you need to confirm that an operation actually took effect. This part builds a small **resilient client** and uses it to interpret API errors, retry transient failures, and verify ingestion health.

<Info>
  Gloo AI does not include a monitoring or health-check endpoint. Resilience is built from the same item APIs you've already used, plus disciplined error handling on the client side.
</Info>

## Pipeline at a Glance

<Steps>
  <Step title="Publisher setup (Studio)">
    Create the publisher that owns your content — [Part 1](/tutorials/rag-pipeline-part-1).
  </Step>

  <Step title="Ingest content with metadata">
    Upload files and enrich them — [Part 1](/tutorials/rag-pipeline-part-1).
  </Step>

  <Step title="Verify indexing">
    Poll item status until your content is searchable — [Part 1](/tutorials/rag-pipeline-part-1).
  </Step>

  <Step title="Semantic search">
    Query your content — deep dive: [Building Custom Search](/tutorials/search).
  </Step>

  <Step title="Grounded completions with sources">
    Answer questions from your content with citations — deep dive: [Grounded Completions with RAG](/tutorials/completions-grounded).
  </Step>

  <Step title="Content lifecycle">
    Update, bulk-edit, and delete content — [Part 2](/tutorials/rag-pipeline-part-2).
  </Step>

  <Step title="Verification, errors & resilience">
    Error handling and retry patterns — **covered below**.
  </Step>
</Steps>

## Prerequisites

Before starting, ensure you have:

* A Gloo AI Studio account
* Your Client ID and Client Secret from the [API Credentials page](/studio/manage-api-credentials)
* A **publisher** with credentials — see [Part 1](/tutorials/rag-pipeline-part-1)
* Familiarity with the item APIs from [Part 1](/tutorials/rag-pipeline-part-1) and [Part 2](/tutorials/rag-pipeline-part-2)

***

## Step 1: Parse Errors and Retry

A resilient client does two things on every request: it turns failures into a **normalized error** (status, code, message) so callers can react to them, and it **retries only transient failures** — server errors (500, 502, 503, 504) and network blips — with exponential backoff. Client errors (400, 401, 403, 404, 422) are bugs in the request, not blips, so they fail fast.

The Data Engine returns a few error shapes — `{"detail": {"code", "message"}}`, `{"detail": "..."}`, and `{"error", "message"}` — so the parser normalizes all of them.

<CodeGroup>
  ```python Python theme={null}
  import time
  import requests

  RETRYABLE = {500, 502, 503, 504}
  MAX_RETRIES = 4


  class ApiError(Exception):
      def __init__(self, status, code, message):
          super().__init__(f"[{status} {code}] {message}")
          self.status, self.code, self.message = status, code, message


  def parse_error(response):
      try:
          body = response.json()
      except ValueError:
          return None, response.reason
      detail = body.get("detail") if isinstance(body, dict) else None
      if isinstance(detail, dict):
          return detail.get("code"), detail.get("message") or response.reason
      if isinstance(detail, str):
          return None, detail
      return body.get("error"), body.get("message") or response.reason


  def request(method, url, token, **kwargs):
      for attempt in range(MAX_RETRIES + 1):
          response = requests.request(
              method, url, headers={"Authorization": f"Bearer {token}"}, timeout=30, **kwargs
          )
          if response.status_code in RETRYABLE and attempt < MAX_RETRIES:
              delay = 2 ** attempt
              print(f"    Attempt {attempt + 1} failed ({response.status_code}); retrying in {delay}s")
              time.sleep(delay)
              continue
          if not response.ok:
              code, message = parse_error(response)
              raise ApiError(response.status_code, code, message)
          return response.json()
  ```

  ```javascript JavaScript theme={null}
  const RETRYABLE = new Set([500, 502, 503, 504]);
  const MAX_RETRIES = 4;
  const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

  class ApiError extends Error {
    constructor(status, code, message) {
      super(message);
      this.status = status;
      this.code = code;
    }
  }

  async function parseError(response) {
    const text = await response.text();
    let body;
    try {
      body = JSON.parse(text);
    } catch {
      return [null, response.statusText];
    }
    const detail = body?.detail;
    if (detail && typeof detail === "object") return [detail.code ?? null, detail.message || response.statusText];
    if (typeof detail === "string") return [null, detail];
    return [body?.error ?? null, body?.message || response.statusText];
  }

  async function request(method, url, token, body) {
    for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
      const response = await fetch(url, {
        method,
        headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
        body: body ? JSON.stringify(body) : undefined,
      });
      if (RETRYABLE.has(response.status) && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${response.status}); retrying in ${delay}s`);
        await sleep(delay * 1000);
        continue;
      }
      if (!response.ok) {
        const [code, message] = await parseError(response);
        throw new ApiError(response.status, code, message);
      }
      return response.json();
    }
  }
  ```

  ```typescript TypeScript theme={null}
  const RETRYABLE = new Set([500, 502, 503, 504]);
  const MAX_RETRIES = 4;
  const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));

  class ApiError extends Error {
    constructor(readonly status: number | null, readonly code: string | null, message: string) {
      super(message);
    }
  }

  async function parseError(response: Response): Promise<[string | null, string]> {
    const text = await response.text();
    let body: any;
    try {
      body = JSON.parse(text);
    } catch {
      return [null, response.statusText];
    }
    const detail = body?.detail;
    if (detail && typeof detail === "object") return [detail.code ?? null, detail.message || response.statusText];
    if (typeof detail === "string") return [null, detail];
    return [body?.error ?? null, body?.message || response.statusText];
  }

  async function request(method: string, url: string, token: string, body?: unknown): Promise<any> {
    for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
      const response = await fetch(url, {
        method,
        headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
        body: body ? JSON.stringify(body) : undefined,
      });
      if (RETRYABLE.has(response.status) && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${response.status}); retrying in ${delay}s`);
        await sleep(delay * 1000);
        continue;
      }
      if (!response.ok) {
        const [code, message] = await parseError(response);
        throw new ApiError(response.status, code, message);
      }
      return response.json();
    }
  }
  ```

  ```php PHP theme={null}
  <?php

  const RETRYABLE = [500, 502, 503, 504];
  const MAX_RETRIES = 4;

  class ApiError extends RuntimeException {
      public function __construct(public ?int $status, public ?string $apiCode, string $message) {
          parent::__construct($message);
      }
  }

  function parseError(int $status, string $body): array {
      $reasons = [400 => 'Bad Request', 403 => 'Forbidden', 404 => 'Not Found', 500 => 'Internal Server Error'];
      $reason = $reasons[$status] ?? "HTTP $status";
      $decoded = json_decode($body, true);
      $detail = is_array($decoded) ? ($decoded['detail'] ?? null) : null;
      if (is_array($detail)) return [$detail['code'] ?? null, ($detail['message'] ?? '') ?: $reason];
      if (is_string($detail)) return [null, $detail];
      return [$decoded['error'] ?? null, ($decoded['message'] ?? '') ?: $reason];
  }

  function request(string $method, string $url, string $token, ?array $body = null) {
      for ($attempt = 0; $attempt <= MAX_RETRIES; $attempt++) {
          $ch = curl_init($url);
          curl_setopt_array($ch, [
              CURLOPT_RETURNTRANSFER => true,
              CURLOPT_CUSTOMREQUEST => $method,
              CURLOPT_HTTPHEADER => ["Authorization: Bearer $token", 'Content-Type: application/json'],
              CURLOPT_POSTFIELDS => $body !== null ? json_encode($body) : null,
              CURLOPT_TIMEOUT => 30,
          ]);
          $responseBody = (string) curl_exec($ch);
          $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
          curl_close($ch);

          if (in_array($status, RETRYABLE, true) && $attempt < MAX_RETRIES) {
              $delay = 2 ** $attempt;
              echo "    Attempt " . ($attempt + 1) . " failed ($status); retrying in {$delay}s\n";
              sleep($delay);
              continue;
          }
          if ($status >= 400) {
              [$code, $message] = parseError($status, $responseBody);
              throw new ApiError($status, $code, $message);
          }
          return json_decode($responseBody, true);
      }
  }
  ```

  ```go Go theme={null}
  var retryable = map[int]bool{500: true, 502: true, 503: true, 504: true}

  const maxRetries = 4

  type ApiError struct {
  	Status  int
  	Code    string
  	Message string
  }

  func (e *ApiError) Error() string { return fmt.Sprintf("[%d %s] %s", e.Status, e.Code, e.Message) }

  func parseError(status int, body []byte) (string, string) {
  	reason := http.StatusText(status)
  	var decoded map[string]any
  	if json.Unmarshal(body, &decoded) != nil {
  		return "", reason
  	}
  	if detail, ok := decoded["detail"].(map[string]any); ok {
  		return str(detail["code"]), orReason(str(detail["message"]), reason)
  	}
  	if detail, ok := decoded["detail"].(string); ok {
  		return "", detail
  	}
  	return str(decoded["error"]), orReason(str(decoded["message"]), reason)
  }

  func request(method, url, token string, body []byte) (map[string]any, error) {
  	for attempt := 0; attempt <= maxRetries; attempt++ {
  		var reader io.Reader
  		if body != nil {
  			reader = bytes.NewReader(body)
  		}
  		req, _ := http.NewRequest(method, url, reader)
  		req.Header.Set("Authorization", "Bearer "+token)
  		req.Header.Set("Content-Type", "application/json")
  		resp, err := http.DefaultClient.Do(req)
  		if err != nil {
  			return nil, &ApiError{0, "network_error", err.Error()}
  		}
  		respBody, _ := io.ReadAll(resp.Body)
  		resp.Body.Close()

  		if retryable[resp.StatusCode] && attempt < maxRetries {
  			delay := 1 << attempt
  			fmt.Printf("    Attempt %d failed (%d); retrying in %ds\n", attempt+1, resp.StatusCode, delay)
  			time.Sleep(time.Duration(delay) * time.Second)
  			continue
  		}
  		if resp.StatusCode >= 400 {
  			code, message := parseError(resp.StatusCode, respBody)
  			return nil, &ApiError{resp.StatusCode, code, message}
  		}
  		var result map[string]any
  		json.Unmarshal(respBody, &result)
  		return result, nil
  	}
  	return nil, &ApiError{0, "retries_exhausted", "Exhausted retries"}
  }
  ```

  ```java Java theme={null}
  static final Set<Integer> RETRYABLE = Set.of(500, 502, 503, 504);
  static final int MAX_RETRIES = 4;

  static class ApiError extends RuntimeException {
      final Integer status;
      final String code;
      ApiError(Integer status, String code, String message) {
          super(message);
          this.status = status;
          this.code = code;
      }
  }

  static String[] parseError(int status, String body) {
      String reason = switch (status) {
          case 400 -> "Bad Request"; case 403 -> "Forbidden"; case 404 -> "Not Found";
          default -> "HTTP " + status;
      };
      JsonObject obj = GSON.fromJson(body, JsonObject.class);
      JsonElement detail = obj == null ? null : obj.get("detail");
      if (detail != null && detail.isJsonObject()) {
          JsonObject d = detail.getAsJsonObject();
          return new String[] {jsonString(d, "code"), orReason(jsonString(d, "message"), reason)};
      }
      if (detail != null && detail.isJsonPrimitive()) return new String[] {null, detail.getAsString()};
      return new String[] {jsonString(obj, "error"), orReason(jsonString(obj, "message"), reason)};
  }

  static JsonObject request(String method, String url, String token, String body)
          throws IOException, InterruptedException {
      for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
          HttpRequest req = HttpRequest.newBuilder()
              .uri(URI.create(url))
              .header("Authorization", "Bearer " + token)
              .header("Content-Type", "application/json")
              .method(method, body != null
                  ? HttpRequest.BodyPublishers.ofString(body)
                  : HttpRequest.BodyPublishers.noBody())
              .build();
          HttpResponse<String> resp = HTTP.send(req, HttpResponse.BodyHandlers.ofString());

          if (RETRYABLE.contains(resp.statusCode()) && attempt < MAX_RETRIES) {
              int delay = 1 << attempt;
              System.out.printf("    Attempt %d failed (%d); retrying in %ds%n", attempt + 1, resp.statusCode(), delay);
              Thread.sleep(delay * 1000L);
              continue;
          }
          if (resp.statusCode() >= 400) {
              String[] parsed = parseError(resp.statusCode(), resp.body());
              throw new ApiError(resp.statusCode(), parsed[0], parsed[1]);
          }
          return GSON.fromJson(resp.body(), JsonObject.class);
      }
      throw new ApiError(null, "retries_exhausted", "Exhausted retries");
  }
  ```
</CodeGroup>

<Tip>
  These snippets are **simplified for readability**. The cookbook client also refreshes the access token once on a `401`, retries network-level failures, and applies the same retry policy to multipart uploads. Both implement the same patterns.
</Tip>

## Step 2: Interpret API Error Responses

With `request` and `parse_error` in place, error handling becomes uniform: catch the normalized error and read its `status`, `code`, and `message`. The calls below deliberately trigger three common failures — a missing item (404), a malformed ID (400), and a rejected token (403).

<CodeGroup>
  ```python Python theme={null}
  import uuid

  cases = [
      ("Missing item (random UUID)", f"{ITEMS_URL}/{uuid.uuid4()}", token),
      ("Malformed item ID", f"{ITEMS_URL}/not-a-valid-uuid", token),
      ("Rejected bearer token", f"{ITEMS_URL}/{uuid.uuid4()}", "invalid-token"),
  ]
  for label, url, tok in cases:
      try:
          request("GET", url, tok)
          print(f"  {label}: unexpectedly succeeded")
      except ApiError as e:
          print(f"  {label}: status={e.status} code={e.code!r} message={e.message!r}")
  ```

  ```javascript JavaScript theme={null}
  import { randomUUID } from "node:crypto";

  const cases = [
    ["Missing item (random UUID)", `${ITEMS_URL}/${randomUUID()}`, token],
    ["Malformed item ID", `${ITEMS_URL}/not-a-valid-uuid`, token],
    ["Rejected bearer token", `${ITEMS_URL}/${randomUUID()}`, "invalid-token"],
  ];
  for (const [label, url, tok] of cases) {
    try {
      await request("GET", url, tok);
      console.log(`  ${label}: unexpectedly succeeded`);
    } catch (e) {
      console.log(`  ${label}: status=${e.status} code='${e.code}' message='${e.message}'`);
    }
  }
  ```

  ```typescript TypeScript theme={null}
  import { randomUUID } from "node:crypto";

  const cases: Array<[string, string, string]> = [
    ["Missing item (random UUID)", `${ITEMS_URL}/${randomUUID()}`, token],
    ["Malformed item ID", `${ITEMS_URL}/not-a-valid-uuid`, token],
    ["Rejected bearer token", `${ITEMS_URL}/${randomUUID()}`, "invalid-token"],
  ];
  for (const [label, url, tok] of cases) {
    try {
      await request("GET", url, tok);
      console.log(`  ${label}: unexpectedly succeeded`);
    } catch (e) {
      if (e instanceof ApiError) {
        console.log(`  ${label}: status=${e.status} code='${e.code}' message='${e.message}'`);
      }
    }
  }
  ```

  ```php PHP theme={null}
  $cases = [
      ['Missing item (random UUID)', ITEMS_URL . '/' . guidv4(), $token],
      ['Malformed item ID', ITEMS_URL . '/not-a-valid-uuid', $token],
      ['Rejected bearer token', ITEMS_URL . '/' . guidv4(), 'invalid-token'],
  ];
  foreach ($cases as [$label, $url, $tok]) {
      try {
          request('GET', $url, $tok);
          echo "  $label: unexpectedly succeeded\n";
      } catch (ApiError $e) {
          echo "  $label: status={$e->status} code='{$e->apiCode}' message='{$e->getMessage()}'\n";
      }
  }
  ```

  ```go Go theme={null}
  cases := []struct{ label, url, token string }{
  	{"Missing item (random UUID)", itemsURL + "/" + newUUID(), token},
  	{"Malformed item ID", itemsURL + "/not-a-valid-uuid", token},
  	{"Rejected bearer token", itemsURL + "/" + newUUID(), "invalid-token"},
  }
  for _, tc := range cases {
  	_, err := request("GET", tc.url, tc.token, nil)
  	if err == nil {
  		fmt.Printf("  %s: unexpectedly succeeded\n", tc.label)
  		continue
  	}
  	var e *ApiError
  	if errors.As(err, &e) {
  		fmt.Printf("  %s: status=%d code='%s' message='%s'\n", tc.label, e.Status, e.Code, e.Message)
  	}
  }
  ```

  ```java Java theme={null}
  record Case(String label, String url, String token) {}
  List<Case> cases = List.of(
      new Case("Missing item (random UUID)", ITEMS_URL + "/" + UUID.randomUUID(), token),
      new Case("Malformed item ID", ITEMS_URL + "/not-a-valid-uuid", token),
      new Case("Rejected bearer token", ITEMS_URL + "/" + UUID.randomUUID(), "invalid-token"));
  for (Case tc : cases) {
      try {
          request("GET", tc.url(), tc.token(), null);
          System.out.println("  " + tc.label() + ": unexpectedly succeeded");
      } catch (ApiError e) {
          System.out.printf("  %s: status=%s code='%s' message='%s'%n",
              tc.label(), e.status, e.code, e.getMessage());
      }
  }
  ```
</CodeGroup>

### What You'll See

```
  Missing item (random UUID): status=404 code='Item not found' message='The requested item does not exist or has been permanently deleted'
  Malformed item ID: status=400 code='Invalid item ID format' message='Item ID must be a valid UUID format'
  Rejected bearer token: status=403 code='Forbidden - insufficient permissions' message='Forbidden'
```

<Note>
  A rejected or malformed token returns **403**, not 401. A genuinely *expired* token typically returns 401 — which is why the cookbook client refreshes the token once on a 401 and retries. It does **not** refresh on 403, since 403 can be a legitimate permission denial (for example, an item that belongs to another publisher).
</Note>

## Step 3: Retry Transient Failures

Transient failures — a 503, a dropped connection — should be retried, not surfaced. The retry loop from Step 1 already does this; here it is in isolation, recovering from a service that fails twice before succeeding.

<Note>
  A healthy API won't return a 5xx on demand, so this example **simulates** a transient failure to exercise the backoff path. In production the same path handles real 5xx responses and network errors.
</Note>

<CodeGroup>
  ```python Python theme={null}
  RETRYABLE_DELAYS = [2 ** i for i in range(MAX_RETRIES)]

  calls = 0
  def flaky():
      global calls
      calls += 1
      if calls < 3:
          raise ApiError(503, "service_unavailable", "Service temporarily unavailable")
      return {"ok": True}

  for attempt in range(MAX_RETRIES + 1):
      try:
          flaky()
          break
      except ApiError as e:
          if e.status in RETRYABLE and attempt < MAX_RETRIES:
              delay = RETRYABLE_DELAYS[attempt]
              print(f"    Attempt {attempt + 1} failed ({e.status}: {e.code}); retrying in {delay}s")
              time.sleep(delay)
          else:
              raise
  print(f"  Succeeded after {calls} attempts")
  ```

  ```javascript JavaScript theme={null}
  let calls = 0;
  const flaky = () => {
    calls += 1;
    if (calls < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
    return { ok: true };
  };

  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    try {
      flaky();
      break;
    } catch (e) {
      if (RETRYABLE.has(e.status) && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${e.status}: ${e.code}); retrying in ${delay}s`);
        await sleep(delay * 1000);
      } else throw e;
    }
  }
  console.log(`  Succeeded after ${calls} attempts`);
  ```

  ```typescript TypeScript theme={null}
  let calls = 0;
  const flaky = (): { ok: boolean } => {
    calls += 1;
    if (calls < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
    return { ok: true };
  };

  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    try {
      flaky();
      break;
    } catch (e) {
      if (e instanceof ApiError && e.status !== null && RETRYABLE.has(e.status) && attempt < MAX_RETRIES) {
        const delay = 2 ** attempt;
        console.log(`    Attempt ${attempt + 1} failed (${e.status}: ${e.code}); retrying in ${delay}s`);
        await sleep(delay * 1000);
      } else throw e;
    }
  }
  console.log(`  Succeeded after ${calls} attempts`);
  ```

  ```php PHP theme={null}
  $calls = 0;
  $flaky = function () use (&$calls) {
      $calls++;
      if ($calls < 3) throw new ApiError(503, 'service_unavailable', 'Service temporarily unavailable');
      return ['ok' => true];
  };

  for ($attempt = 0; $attempt <= MAX_RETRIES; $attempt++) {
      try {
          $flaky();
          break;
      } catch (ApiError $e) {
          if (in_array($e->status, RETRYABLE, true) && $attempt < MAX_RETRIES) {
              $delay = 2 ** $attempt;
              echo "    Attempt " . ($attempt + 1) . " failed ({$e->status}: {$e->apiCode}); retrying in {$delay}s\n";
              sleep($delay);
          } else {
              throw $e;
          }
      }
  }
  echo "  Succeeded after $calls attempts\n";
  ```

  ```go Go theme={null}
  calls := 0
  flaky := func() error {
  	calls++
  	if calls < 3 {
  		return &ApiError{503, "service_unavailable", "Service temporarily unavailable"}
  	}
  	return nil
  }

  for attempt := 0; attempt <= maxRetries; attempt++ {
  	err := flaky()
  	if err == nil {
  		break
  	}
  	var e *ApiError
  	if errors.As(err, &e) && retryable[e.Status] && attempt < maxRetries {
  		delay := 1 << attempt
  		fmt.Printf("    Attempt %d failed (%d: %s); retrying in %ds\n", attempt+1, e.Status, e.Code, delay)
  		time.Sleep(time.Duration(delay) * time.Second)
  	} else {
  		log.Fatal(err)
  	}
  }
  fmt.Printf("  Succeeded after %d attempts\n", calls)
  ```

  ```java Java theme={null}
  int[] calls = {0};
  Runnable flaky = () -> {
      calls[0]++;
      if (calls[0] < 3) throw new ApiError(503, "service_unavailable", "Service temporarily unavailable");
  };

  for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
      try {
          flaky.run();
          break;
      } catch (ApiError e) {
          if (e.status != null && RETRYABLE.contains(e.status) && attempt < MAX_RETRIES) {
              int delay = 1 << attempt;
              System.out.printf("    Attempt %d failed (%d: %s); retrying in %ds%n",
                  attempt + 1, e.status, e.code, delay);
              Thread.sleep(delay * 1000L);
          } else {
              throw e;
          }
      }
  }
  System.out.printf("  Succeeded after %d attempts%n", calls[0]);
  ```
</CodeGroup>

### What You'll See

```
    Attempt 1 failed (503: service_unavailable); retrying in 1s
    Attempt 2 failed (503: service_unavailable); retrying in 2s
  Succeeded after 3 attempts
```

## Step 4: Verify Ingestion Health

There's no health endpoint, so you verify by checking the items you care about. Upload a batch, wait for indexing, then fetch each item's status and roll it up into a summary — treating a 404 as "not found" rather than an error so one missing item doesn't abort the check. The example deliberately includes a random ID to show that path.

<CodeGroup>
  ```python Python theme={null}
  # item_ids: uploaded and indexed via the resilient client (see the cookbook)
  to_check = item_ids + [str(uuid.uuid4())]
  summary = {"completed": 0, "pending": 0, "failed": 0, "not_found": 0}
  for item_id in to_check:
      try:
          status = request("GET", f"{ITEMS_URL}/{item_id}", token).get("status", "").upper()
          if status == "COMPLETED":
              summary["completed"] += 1
          elif status in ("FAILED", "ERROR"):
              summary["failed"] += 1
          else:
              summary["pending"] += 1
      except ApiError as e:
          if e.status == 404:
              summary["not_found"] += 1
          else:
              raise
  print(f"  Health: {summary['completed']} completed, {summary['pending']} pending, "
        f"{summary['failed']} failed, {summary['not_found']} not found")
  ```

  ```javascript JavaScript theme={null}
  // itemIds: uploaded and indexed via the resilient client (see the cookbook)
  const toCheck = [...itemIds, randomUUID()];
  const summary = { completed: 0, pending: 0, failed: 0, notFound: 0 };
  for (const itemId of toCheck) {
    try {
      const status = ((await request("GET", `${ITEMS_URL}/${itemId}`, token)).status ?? "").toUpperCase();
      if (status === "COMPLETED") summary.completed += 1;
      else if (["FAILED", "ERROR"].includes(status)) summary.failed += 1;
      else summary.pending += 1;
    } catch (e) {
      if (e.status === 404) summary.notFound += 1;
      else throw e;
    }
  }
  console.log(
    `  Health: ${summary.completed} completed, ${summary.pending} pending, ` +
      `${summary.failed} failed, ${summary.notFound} not found`
  );
  ```

  ```typescript TypeScript theme={null}
  // itemIds: uploaded and indexed via the resilient client (see the cookbook)
  const toCheck = [...itemIds, randomUUID()];
  const summary = { completed: 0, pending: 0, failed: 0, notFound: 0 };
  for (const itemId of toCheck) {
    try {
      const status = ((await request("GET", `${ITEMS_URL}/${itemId}`, token)).status ?? "").toUpperCase();
      if (status === "COMPLETED") summary.completed += 1;
      else if (["FAILED", "ERROR"].includes(status)) summary.failed += 1;
      else summary.pending += 1;
    } catch (e) {
      if (e instanceof ApiError && e.status === 404) summary.notFound += 1;
      else throw e;
    }
  }
  console.log(
    `  Health: ${summary.completed} completed, ${summary.pending} pending, ` +
      `${summary.failed} failed, ${summary.notFound} not found`
  );
  ```

  ```php PHP theme={null}
  // $itemIds: uploaded and indexed via the resilient client (see the cookbook)
  $toCheck = array_merge($itemIds, [guidv4()]);
  $summary = ['completed' => 0, 'pending' => 0, 'failed' => 0, 'not_found' => 0];
  foreach ($toCheck as $itemId) {
      try {
          $status = strtoupper(request('GET', ITEMS_URL . '/' . $itemId, $token)['status'] ?? '');
          if ($status === 'COMPLETED') $summary['completed']++;
          elseif (in_array($status, ['FAILED', 'ERROR'], true)) $summary['failed']++;
          else $summary['pending']++;
      } catch (ApiError $e) {
          if ($e->status === 404) $summary['not_found']++;
          else throw $e;
      }
  }
  echo "  Health: {$summary['completed']} completed, {$summary['pending']} pending, "
      . "{$summary['failed']} failed, {$summary['not_found']} not found\n";
  ```

  ```go Go theme={null}
  // itemIDs: uploaded and indexed via the resilient client (see the cookbook)
  toCheck := append(append([]string{}, itemIDs...), newUUID())
  completed, pending, failed, notFound := 0, 0, 0, 0
  for _, id := range toCheck {
  	item, err := request("GET", itemsURL+"/"+id, token, nil)
  	if err != nil {
  		var e *ApiError
  		if errors.As(err, &e) && e.Status == 404 {
  			notFound++
  			continue
  		}
  		log.Fatal(err)
  	}
  	switch strings.ToUpper(str(item["status"])) {
  	case "COMPLETED":
  		completed++
  	case "FAILED", "ERROR":
  		failed++
  	default:
  		pending++
  	}
  }
  fmt.Printf("  Health: %d completed, %d pending, %d failed, %d not found\n",
  	completed, pending, failed, notFound)
  ```

  ```java Java theme={null}
  // itemIds: uploaded and indexed via the resilient client (see the cookbook)
  List<String> toCheck = new ArrayList<>(itemIds);
  toCheck.add(UUID.randomUUID().toString());
  int completed = 0, pending = 0, failed = 0, notFound = 0;
  for (String itemId : toCheck) {
      try {
          String status = request("GET", ITEMS_URL + "/" + itemId, token, null)
              .get("status").getAsString().toUpperCase();
          if (status.equals("COMPLETED")) completed++;
          else if (status.equals("FAILED") || status.equals("ERROR")) failed++;
          else pending++;
      } catch (ApiError e) {
          if (e.status != null && e.status == 404) notFound++;
          else throw e;
      }
  }
  System.out.printf("  Health: %d completed, %d pending, %d failed, %d not found%n",
      completed, pending, failed, notFound);
  ```
</CodeGroup>

### What You'll See

```
  Health: 2 completed, 0 pending, 0 failed, 1 not found
```

The two uploaded items report `COMPLETED`; the random ID is counted as `not found` instead of throwing.

## Run the Complete Example

The cookbook ties it together — a resilient client that handles errors, retries, and a full upload → verify → clean-up health check — in all six languages. From the [cookbook repository](https://github.com/GlooDeveloper/gloo-ai-docs-cookbook), install dependencies, copy `.env.example` to `.env` and add your credentials, then run it:

<CodeGroup>
  ```bash Python theme={null}
  cd rag-pipeline-part-3/python
  python3 -m venv venv
  source venv/bin/activate  # On Windows: venv\Scripts\activate
  pip install -r requirements.txt
  cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
  python main.py
  ```

  ```bash JavaScript theme={null}
  cd rag-pipeline-part-3/javascript
  npm install
  cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
  npm start
  ```

  ```bash TypeScript theme={null}
  cd rag-pipeline-part-3/typescript
  npm install
  cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
  npm start
  ```

  ```bash PHP theme={null}
  cd rag-pipeline-part-3/php
  composer install
  cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
  php index.php
  ```

  ```bash Go theme={null}
  cd rag-pipeline-part-3/go
  go mod tidy
  cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
  go run main.go
  ```

  ```bash Java theme={null}
  cd rag-pipeline-part-3/java
  mvn compile
  cp .env.example .env       # then add your Client ID, Secret, and Publisher ID
  mvn -q exec:java
  ```
</CodeGroup>

You'll see:

```
Step 1: Resilient client ready (token refresh, error parsing, retry/backoff).

Step 2: Interpreting API error responses...
  Missing item (random UUID): status=404 code='Item not found' message='The requested item does not exist or has been permanently deleted'
  Malformed item ID: status=400 code='Invalid item ID format' message='Item ID must be a valid UUID format'
  Rejected bearer token: status=403 code='Forbidden - insufficient permissions' message='Forbidden'

Step 3: Retrying transient failures with backoff...
    Attempt 1 failed (503: service_unavailable); retrying in 1s
    Attempt 2 failed (503: service_unavailable); retrying in 2s
  Succeeded after 3 attempts

Step 4: Verifying ingestion health...
  Uploading and indexing a batch...
  Waiting for 2 item(s) to finish indexing...
  Health: 2 completed, 0 pending, 0 failed, 1 not found
  Cleaned up 2 item(s)

Done. The resilient client handled errors, retries, and verification end to end.
```

## Working Code Sample

<Card title="View Complete Code" icon="github" href="https://github.com/GlooDeveloper/gloo-ai-docs-cookbook/tree/main/rag-pipeline-part-3">
  Clone or browse the complete resilient client for all 6 languages (JavaScript, TypeScript, Python, PHP, Go, Java) with setup instructions and the sample content files.
</Card>

## Troubleshooting

### A request fails immediately instead of retrying

That's intended for client errors (400, 401, 403, 404, 422) — they won't succeed on retry. Only 500/502/503/504 and network failures are retried.

### Retries never stop / take too long

Check your backoff cap. With base-2 exponential backoff and `MAX_RETRIES = 4`, the waits are 1s, 2s, 4s, 8s. Lower `MAX_RETRIES` or cap the delay for latency-sensitive paths.

### Empty error message

Some error responses carry a code but no message. The parser falls back to the HTTP reason phrase (for example, `Forbidden`) so you always have something to log.

### 403 on a token you believe is valid

A malformed or rejected token returns 403. Re-check the token; if it's genuinely expired you'll typically get a 401, which the cookbook client refreshes automatically. See the [Authentication Tutorial](/tutorials/authentication).

## Next Steps

That completes the **Build an End-to-End RAG Pipeline** series — you've set up a publisher and ingested content (Part 1), managed its lifecycle (Part 2), and made the integration resilient (Part 3). From here:

1. **[Building Custom Search](/tutorials/search)** — surface your content to users
2. **[Grounded Completions with RAG](/tutorials/completions-grounded)** — answer questions from your content with citations
3. Fold the resilient client into your own service so every Gloo AI call gets consistent error handling and retries.