Testing, Evaluation and Observability

A prompt change, a new model or a different reasoning effort can quietly make answers worse, and without the right logs you won’t know until users tell you. This page is for developers who want to catch regressions before they ship and have the data to explain what happened when something goes wrong in production.

New to measuring model quality? Metrics and Evaluation in the AI Learning Center introduces the common measures, from benchmarks to human review.

This page covers how to:

build a test set from your own examples
score answers in ways you can repeat
compare models and settings before you switch
log what each request did
use your logs to debug and to work with support

Build a test set from your own examples

General evaluation benchmarks tell you little about your app. A small set of your own examples tells you a lot.

Start with 30 to 100 cases. Take them from real requests (with personal details removed), from your product’s requirements, and from bugs you’ve fixed.
Cover the hard parts: short and long inputs, ambiguous questions, questions your app should decline, inputs in other languages, and inputs that try to override your instructions.
Write down what a good answer has, not the exact words: the facts it must include, the format it must follow, what it must not do.
Keep it in version control next to your prompts, and add a case every time a user reports a bad answer.

Score answers in ways you can repeat

Model output varies from run to run, so score properties, not exact strings.

Check with code first. Does the JSON parse and match the schema? Did the tool call name the right tool with valid arguments? Is the answer under the length limit? Did it finish (finish_reason or status)? These checks are cheap and never disagree with themselves.
Use a rubric for the rest. For tone, helpfulness and accuracy, have reviewers, or a separate model given a written rubric, score each answer. Spot-check the model’s scores against a person’s before you trust them.
Run each case more than once. One run can be lucky. Three to five runs per case show how consistent the answer is.
Don’t count on temperature to make runs repeatable. Some models don’t take it, for example Claude 5 models and OpenAI GPT-5 and later models while they reason. The request still runs, and your value is listed in ignored_params. See Sampling parameters.
Track cost and time too. Record usage and response time for every case, so a change that improves quality at three times the cost is visible.

Grounded and decision features need their own checks: see Evaluate Before You Ship for grounded answers and Choose thresholds from your own data for System One.

Compare before you switch

Run the whole test set before any change that could affect answers: a new model, a new prompt, a different reasoning effort or output limit, a new tool definition, or a switch between guarded and direct endpoints.

Change one thing at a time, so you know what caused a difference.
Pin the model in tests and in production, so the model you tested is the model you run. See Choosing, Pinning and Updating Models.
Compare side by side. Put the old and new answers for each case next to each other, with scores, usage and time. The Chat Playground is useful for a first look at a few prompts.
Decide your bar in advance: for example, no case that passed before may fail, and average cost may not rise by more than 20%.
Roll out gradually where you can, and watch the production signals below before moving all traffic.

When a model you use is deprecated, run your test set against the replacement before you change your configuration, since its answers, reasoning and price can differ.

Log what each request did

For every request, log one structured record. Leave out prompt and answer text unless you need it and can protect it.

Field	Where it comes from
Endpoint, requested model, feature or user action	Your request
Model that answered	`model` in the response
Redirected model	`X-Gloo-Model-Redirected-From`, `Deprecation` and `Sunset` headers (Model Lifecycle)
Parameters the model didn’t use	`ignored_params` in the body, or the `X-Ignored-Params` header on a stream
Tokens	`usage`, including cached and reasoning tokens where reported
How it ended	`finish_reason` (Chat Completions) or `status` (Responses)
Time	Total time, and time to first token on a stream
Request ID	`X-Request-Id` on every direct response; `trace_id` in Completions V2 response bodies and in error bodies
Errors	`error.code`, `error.name`, `error.retryable` and `X-Gloo-Error-Code` on direct (Log What Support Needs)

On the direct endpoints, an X-Request-Id you send on the request is returned on the response, so you can use your own ID end to end.

import json
import logging
import time

import requests

log = logging.getLogger("gloo")


def post_and_log(url: str, headers: dict, body: dict, feature: str) -> requests.Response:
    started = time.monotonic()
    record = {"feature": feature, "endpoint": url, "requested_model": body.get("model")}
    try:
        response = requests.post(url, headers=headers, json=body, timeout=120)
    except requests.RequestException as exc:
        # Timeouts and connection failures never reach a response; log the attempt, then re-raise.
        record.update({"seconds": round(time.monotonic() - started, 3), "error_name": type(exc).__name__})
        log.warning(json.dumps(record))
        raise
    record.update({
        "status_code": response.status_code,
        "seconds": round(time.monotonic() - started, 3),
        "request_id": response.headers.get("X-Request-Id"),
        "redirected_from": response.headers.get("X-Gloo-Model-Redirected-From"),
        "sunset": response.headers.get("Sunset"),
        "error_code": response.headers.get("X-Gloo-Error-Code"),
    })
    try:
        data = response.json()
    except ValueError:
        data = {}
    if response.ok:
        choice = (data.get("choices") or [{}])[0]
        record.update({
            "model": data.get("model"),
            "finish": choice.get("finish_reason") or data.get("status"),
            "usage": data.get("usage"),
            "ignored_params": data.get("ignored_params"),
            "trace_id": data.get("trace_id"),
        })
    else:
        # Most errors carry an `error` object; some direct-endpoint refusals use a `detail` object.
        error = data.get("error") or {}
        detail = data["detail"] if isinstance(data.get("detail"), dict) else {}
        record.update({
            "error_name": error.get("name") or detail.get("code"),
            "retryable": error.get("retryable", detail.get("retryable")),
            "trace_id": error.get("trace_id") or detail.get("trace_id"),
        })
    log.info(json.dumps(record))
    return response

async function postAndLog(url: string, headers: Record<string, string>, body: any, feature: string): Promise<Response> {
  const started = performance.now();
  const record: Record<string, unknown> = { feature, endpoint: url, requested_model: body.model };
  let response: Response;
  try {
    response = await fetch(url, {
      method: "POST",
      headers: { ...headers, "Content-Type": "application/json" },
      body: JSON.stringify(body),
    });
  } catch (err) {
    // Network failures and aborts never reach a response; log the attempt, then re-throw.
    record.seconds = Math.round(performance.now() - started) / 1000;
    record.error_name = err instanceof Error ? err.name : String(err);
    console.warn(JSON.stringify(record));
    throw err;
  }
  Object.assign(record, {
    status_code: response.status,
    seconds: Math.round(performance.now() - started) / 1000,
    request_id: response.headers.get("X-Request-Id"),
    redirected_from: response.headers.get("X-Gloo-Model-Redirected-From"),
    sunset: response.headers.get("Sunset"),
    error_code: response.headers.get("X-Gloo-Error-Code"),
  });
  const data: any = await response.clone().json().catch(() => ({}));
  if (response.ok) {
    record.model = data.model;
    record.finish = data.choices?.[0]?.finish_reason ?? data.status;
    record.usage = data.usage;
    record.ignored_params = data.ignored_params;
    record.trace_id = data.trace_id;
  } else {
    // Most errors carry an `error` object; some direct-endpoint refusals use a `detail` object.
    const detail = typeof data.detail === "object" && data.detail !== null ? data.detail : {};
    record.error_name = data.error?.name ?? detail.code;
    record.retryable = data.error?.retryable ?? detail.retryable;
    record.trace_id = data.error?.trace_id ?? detail.trace_id;
  }
  console.log(JSON.stringify(record));
  return response;
}

For streams, log the same record when the stream ends, from the headers and the final event, and add whether any output was shown before a failure. See the Observability Checklist.

Watch production

Track rates, not single events: error rate by error.name, cut-off rate, content_filter rate, redirect count, cost per feature and response time. A rise matters more than any one request.
Alert on model changes. Any X-Gloo-Model-Redirected-From, or a Sunset without it, means a model you use is retiring. See Choosing, Pinning and Updating Models.
Reconcile with Studio. The API Usage page shows spend, requests and tokens, filterable by API key. Give each application and environment its own key, so a spike points to a source.
Feed problems back into tests. Sample answers regularly, especially refusals, cut-offs and low-rated replies, and add the bad ones to your test set.

Handle retries without hiding problems

Retry only what’s retryable, with backoff and a budget, and log every attempt, not just the last one; a request that succeeds on its third try is a signal. See Error Handling and Retries and the Build a Resilient Client tutorial. When errors persist after retrying, contact support with the trace_id from the error body, and the X-Request-Id on direct endpoints, so the exact request can be traced.

Summary

Keep a version-controlled test set of your own examples, and add every reported bad answer to it.
Score properties with code first and a rubric second; run each case several times.
Change one thing at a time, pin the model, and compare quality, cost and time before you switch.
Log model, redirects, ignored_params, usage, how the request ended, timing and request IDs for every request.
Watch rates over time, alert on retiring models, and include trace_id or X-Request-Id when you contact support.

Next steps

Error Handling and Retries

Which errors to retry, backoff, budgets and what to log.

Build a Resilient Client

Put retries, failover and logging into code, step by step.

Metrics and Evaluation

The lesson: benchmarks, latency and human evaluation.

Building Trustworthy Grounded Applications

Evaluation for answers grounded in your own content.

Best Practices

Security

Models and Cost

Requests

Reliability

Trust

Content

Testing, Evaluation and Observability

Build a test set from your own examples

Score answers in ways you can repeat

Compare before you switch

Log what each request did

Watch production

Handle retries without hiding problems

Summary

Next steps

Error Handling and Retries

Build a Resilient Client

Metrics and Evaluation

Building Trustworthy Grounded Applications

​Build a test set from your own examples

​Score answers in ways you can repeat

​Compare before you switch

​Log what each request did

​Watch production

​Handle retries without hiding problems

​Summary

​Next steps

Error Handling and Retries

Build a Resilient Client

Metrics and Evaluation

Building Trustworthy Grounded Applications

Build a test set from your own examples

Score answers in ways you can repeat

Compare before you switch

Log what each request did

Watch production

Handle retries without hiding problems

Summary

Next steps