> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Testing, Evaluation and Observability

> Best practices for testing Gloo AI features before you ship, comparing models before you switch, and logging what you need to debug production requests and work with support.

A prompt change, a new model or a different [reasoning effort](https://docs.gloo.com/ai-learning-center/gloo-ai-104/cost-and-speed.md#reasoning-effort) can quietly make answers worse, and without the right logs you won't know until users tell you. This page is for developers who want to catch regressions before they ship and have the data to explain what happened when something goes wrong in production.

> [!TIP]
>
> New to measuring model quality? [Metrics and Evaluation](https://docs.gloo.com/ai-learning-center/gloo-ai-101/metrics-and-evaluation.md) in the AI Learning Center introduces the common measures, from benchmarks to human review.

This page covers how to:

* build a test set from your own examples
* score answers in ways you can repeat
* compare models and settings before you switch
* log what each request did
* use your logs to debug and to work with support

## Build a test set from your own examples

General [evaluation](https://docs.gloo.com/ai-learning-center/gloo-ai-101/metrics-and-evaluation.md) benchmarks tell you little about your app. A small set of your own examples tells you a lot.

* **Start with 30 to 100 cases.** Take them from real requests (with personal details removed), from your product's requirements, and from bugs you've fixed.
* **Cover the hard parts:** short and long inputs, ambiguous questions, questions your app should decline, inputs in other languages, and inputs that try to override your instructions.
* **Write down what a good answer has,** not the exact words: the facts it must include, the format it must follow, what it must not do.
* **Keep it in version control** next to your prompts, and add a case every time a user reports a bad answer.

## Score answers in ways you can repeat

Model output varies from run to run, so score properties, not exact strings.

* **Check with code first.** Does the JSON parse and match the schema? Did the tool call name the right tool with valid arguments? Is the answer under the length limit? Did it finish (`finish_reason` or `status`)? These checks are cheap and never disagree with themselves.
* **Use a rubric for the rest.** For tone, helpfulness and accuracy, have reviewers, or a separate model given a written rubric, score each answer. Spot-check the model's scores against a person's before you trust them.
* **Run each case more than once.** One run can be lucky. Three to five runs per case show how consistent the answer is.
* **Don't count on `temperature` to make runs repeatable.** Some models don't take it, for example Claude 5 models and OpenAI GPT-5 and later models while they reason. The request still runs, and your value is listed in `ignored_params`. See [Sampling parameters](https://docs.gloo.com/api-guides/direct-completions.md#sampling-parameters).
* **Track cost and time too.** Record `usage` and response time for every case, so a change that improves quality at three times the cost is visible.

Grounded and decision features need their own checks: see [Evaluate Before You Ship](https://docs.gloo.com/best-practices/trustworthy-grounded-applications.md#evaluate-before-you-ship) for [grounded answers](https://docs.gloo.com/ai-learning-center/gloo-ai-103/the-generation-step.md#grounded-vs-ungrounded-generation) and [Choose thresholds from your own data](https://docs.gloo.com/best-practices/system-one.md#choose-thresholds-from-your-own-data) for [System One](https://docs.gloo.com/api-guides/system-one.md).

## Compare before you switch

Run the whole test set before any change that could affect answers: a new model, a new prompt, a different reasoning effort or output limit, a new [tool](https://docs.gloo.com/ai-learning-center/gloo-ai-104/tools-and-mcp.md) definition, or a switch between [guarded](https://docs.gloo.com/api-guides/endpoint-types.md#guarded-endpoints) and [direct endpoints](https://docs.gloo.com/api-guides/endpoint-types.md#direct-endpoints).

* **Change one thing at a time,** so you know what caused a difference.
* **Pin the `model`** in tests and in production, so the model you tested is the model you run. See [Choosing, Pinning and Updating Models](https://docs.gloo.com/best-practices/choosing-and-updating-models.md).
* **Compare side by side.** Put the old and new answers for each case next to each other, with scores, `usage` and time. The [Chat Playground](https://docs.gloo.com/studio/playground-chat.md#comparing-models-side-by-side) is useful for a first look at a few prompts.
* **Decide your bar in advance:** for example, no case that passed before may fail, and average cost may not rise by more than 20%.
* **Roll out gradually** where you can, and watch the production signals below before moving all traffic.

When a model you use is deprecated, run your test set against the replacement before you change your configuration, since its answers, reasoning and price can differ.

## Log what each request did

For every request, log one structured record. Leave out prompt and answer text unless you need it and can protect it.

| Field | Where it comes from |
| :- | :- |
| Endpoint, requested model, feature or user action | Your request |
| Model that answered | `model` in the response |
| Redirected model | `X-Gloo-Model-Redirected-From`, `Deprecation` and `Sunset` headers ([Model Lifecycle](https://docs.gloo.com/api-guides/model-lifecycle.md#response-headers-on-a-redirected-request)) |
| Parameters the model didn't use | `ignored_params` in the body, or the `X-Ignored-Params` header on a stream |
| Tokens | `usage`, including cached and reasoning tokens where reported |
| How it ended | `finish_reason` (Chat Completions) or `status` (Responses) |
| Time | Total time, and time to first token on a stream |
| Request ID | `X-Request-Id` on every direct response; `trace_id` in Completions V2 response bodies and in error bodies |
| Errors | `error.code`, `error.name`, `error.retryable` and `X-Gloo-Error-Code` on direct ([Log What Support Needs](https://docs.gloo.com/best-practices/error-handling-and-retries.md#log-what-support-needs)) |

On the direct endpoints, an `X-Request-Id` you send on the request is returned on the response, so you can use your own ID end to end.

```python Python theme={null}
import json
import logging
import time

import requests

log = logging.getLogger("gloo")


def post_and_log(url: str, headers: dict, body: dict, feature: str) -> requests.Response:
    started = time.monotonic()
    record = {"feature": feature, "endpoint": url, "requested_model": body.get("model")}
    try:
        response = requests.post(url, headers=headers, json=body, timeout=120)
    except requests.RequestException as exc:
        # Timeouts and connection failures never reach a response; log the attempt, then re-raise.
        record.update({"seconds": round(time.monotonic() - started, 3), "error_name": type(exc).__name__})
        log.warning(json.dumps(record))
        raise
    record.update({
        "status_code": response.status_code,
        "seconds": round(time.monotonic() - started, 3),
        "request_id": response.headers.get("X-Request-Id"),
        "redirected_from": response.headers.get("X-Gloo-Model-Redirected-From"),
        "sunset": response.headers.get("Sunset"),
        "error_code": response.headers.get("X-Gloo-Error-Code"),
    })
    try:
        data = response.json()
    except ValueError:
        data = {}
    if response.ok:
        choice = (data.get("choices") or [{}])[0]
        record.update({
            "model": data.get("model"),
            "finish": choice.get("finish_reason") or data.get("status"),
            "usage": data.get("usage"),
            "ignored_params": data.get("ignored_params"),
            "trace_id": data.get("trace_id"),
        })
    else:
        # Most errors carry an `error` object; some direct-endpoint refusals use a `detail` object.
        error = data.get("error") or {}
        detail = data["detail"] if isinstance(data.get("detail"), dict) else {}
        record.update({
            "error_name": error.get("name") or detail.get("code"),
            "retryable": error.get("retryable", detail.get("retryable")),
            "trace_id": error.get("trace_id") or detail.get("trace_id"),
        })
    log.info(json.dumps(record))
    return response
```

```typescript TypeScript theme={null}
async function postAndLog(url: string, headers: Record<string, string>, body: any, feature: string): Promise<Response> {
  const started = performance.now();
  const record: Record<string, unknown> = { feature, endpoint: url, requested_model: body.model };
  let response: Response;
  try {
    response = await fetch(url, {
      method: "POST",
      headers: { ...headers, "Content-Type": "application/json" },
      body: JSON.stringify(body),
    });
  } catch (err) {
    // Network failures and aborts never reach a response; log the attempt, then re-throw.
    record.seconds = Math.round(performance.now() - started) / 1000;
    record.error_name = err instanceof Error ? err.name : String(err);
    console.warn(JSON.stringify(record));
    throw err;
  }
  Object.assign(record, {
    status_code: response.status,
    seconds: Math.round(performance.now() - started) / 1000,
    request_id: response.headers.get("X-Request-Id"),
    redirected_from: response.headers.get("X-Gloo-Model-Redirected-From"),
    sunset: response.headers.get("Sunset"),
    error_code: response.headers.get("X-Gloo-Error-Code"),
  });
  const data: any = await response.clone().json().catch(() => ({}));
  if (response.ok) {
    record.model = data.model;
    record.finish = data.choices?.[0]?.finish_reason ?? data.status;
    record.usage = data.usage;
    record.ignored_params = data.ignored_params;
    record.trace_id = data.trace_id;
  } else {
    // Most errors carry an `error` object; some direct-endpoint refusals use a `detail` object.
    const detail = typeof data.detail === "object" && data.detail !== null ? data.detail : {};
    record.error_name = data.error?.name ?? detail.code;
    record.retryable = data.error?.retryable ?? detail.retryable;
    record.trace_id = data.error?.trace_id ?? detail.trace_id;
  }
  console.log(JSON.stringify(record));
  return response;
}
```

For streams, log the same record when the stream ends, from the headers and the final event, and add whether any output was shown before a failure. See the [Observability Checklist](https://docs.gloo.com/best-practices/completions-streaming-failures.md#observability-checklist).

## Watch production

* **Track rates, not single events:** error rate by `error.name`, cut-off rate, `content_filter` rate, redirect count, cost per feature and response time. A rise matters more than any one request.
* **Alert on model changes.** Any `X-Gloo-Model-Redirected-From`, or a `Sunset` without it, means a model you use is retiring. See [Choosing, Pinning and Updating Models](https://docs.gloo.com/best-practices/choosing-and-updating-models.md#watch-the-response-headers).
* **Reconcile with Studio.** The [API Usage](https://docs.gloo.com/studio/api-usage.md) page shows spend, requests and tokens, filterable by [API key](https://docs.gloo.com/studio/manage-api-credentials.md). Give each application and environment its own key, so a spike points to a source.
* **Feed problems back into tests.** Sample answers regularly, especially refusals, cut-offs and low-rated replies, and add the bad ones to your test set.

## Handle retries without hiding problems

Retry only what's retryable, with backoff and a budget, and log every attempt, not just the last one; a request that succeeds on its third try is a signal. See [Error Handling and Retries](https://docs.gloo.com/best-practices/error-handling-and-retries.md) and the [Build a Resilient Client](https://docs.gloo.com/tutorials/resilient-client.md) tutorial.

When errors persist after retrying, [contact support](https://docs.gloo.com/getting-started/support.md#contact-support) with the `trace_id` from the error body, and the `X-Request-Id` on direct endpoints, so the exact request can be traced.

## Summary

* Keep a version-controlled test set of your own examples, and add every reported bad answer to it.
* Score properties with code first and a rubric second; run each case several times.
* Change one thing at a time, pin the model, and compare quality, cost and time before you switch.
* Log model, redirects, `ignored_params`, `usage`, how the request ended, timing and request IDs for every request.
* Watch rates over time, alert on retiring models, and include `trace_id` or `X-Request-Id` when you contact support.

## Next steps

> [card] [Error Handling and Retries](https://docs.gloo.com/best-practices/error-handling-and-retries.md)
>
> Which errors to retry, backoff, budgets and what to log.

> [card] [Build a Resilient Client](https://docs.gloo.com/tutorials/resilient-client.md)
>
> Put retries, failover and logging into code, step by step.

> [card] [Metrics and Evaluation](https://docs.gloo.com/ai-learning-center/gloo-ai-101/metrics-and-evaluation.md)
>
> The lesson: benchmarks, latency and human evaluation.

> [card] [Building Trustworthy Grounded Applications](https://docs.gloo.com/best-practices/trustworthy-grounded-applications.md#evaluate-before-you-ship)
>
> Evaluation for answers grounded in your own content.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.