> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Author evaluation sets, start and cancel metered runs, browse runs and compare results.

Evals are the platform's evaluation harness for measuring agent and workflow quality over a fixed test set. Over the API you can read sets and runs, author sets for agents you may write, start and cancel metered runs with scopes reaching both `agents:write` and `inference:use`, and compare runs.

## Visibility

The reads require `agents:read`, and each returns only what the caller may read:

* An eval set bound to an agent (`agentDefinitionId` set) is visible only when the caller holds `agents:read` at that agent's home org unit and the agent is not PERSONAL. A set bound to a missing agent is not visible.
* An unbound eval set is platform catalog: it is visible only when the caller holds `agents:read` at the deployment root.
* A run is visible only when its set is visible and, when the run names its own agent, that agent is visible too.

Keys and External Apps with platform-scope grants (every migrated API key) hold `agents:read` at every home, so they still see every set whose agent exists and is not PERSONAL. A caller whose grants are scoped to org units sees only the sets bound to agents in those units, and no unbound sets. A missing set and a set you may not read are the same `404 resource_missing`.

## Authoring

Creating, updating and archiving a set require `agents:write`, decided again at the set's home:

* A set bound to an agent: you must be able to read the agent (`agents:read` at its home org unit, not PERSONAL) and hold `agents:write` there. Creating a set for an agent you cannot read or write is `404` "Agent not found".
* An unbound (catalog) set: `agents:write` at the deployment root. A credential whose grants stop at an org unit cannot create or edit catalog sets.
* A set you cannot read is `404`, like a missing one; a set you can read but not write is `403` naming `agents:write`.

The bodies are strict, and any other key is `400 bad_request`:

| Field | Limit |
| - | - |
| `name` | 1-200 characters |
| `description` | at most 2,000 characters (`null` on update clears it) |
| `testCases` | 1-200 cases, each `{id, input, expectedOutput?, rubric?, weight?, tags?}` with a unique `id` of 1-100 characters, `input` of 1-20,000, `expectedOutput` up to 20,000, `rubric` up to 4,000 and `weight` above 0 up to 100 |
| `tags` (set and case) | at most 20, each at most 64 characters |
| all `testCases` together | at most 800,000 bytes as JSON (a set is stored as one record) |

`dimensions` and `goldenDatasetId` are refused: the eval runner scores a fixed set of dimensions and never reads either field. A new set is `active`; machines cannot change a set's status, agent binding, dimensions or golden dataset. Update replaces `testCases` as a whole; send `expectedUpdatedAt` (the `updatedAt` you read) to refuse a stale edit with `409 conflict`. An archived set cannot be edited (`409 invalid_state`), and archiving is one-way: a second archive answers `already_archived` and writes nothing.

**Authoring spends nothing.** Creating or editing a set starts no run. Sets created through the API are never run by the weekly regression evals, which run only sets created in the Console or seeded. For the same reason, the API cannot change the test cases of a Console or seeded set (`409 conflict`): you can rename, re-describe, retag or archive it, but what its weekly run executes is changed only in the Console.

A same-key `Idempotency-Key` retry replays the recorded answer only while you may still read and write the set. An answer above the 32 KiB idempotency storage ceiling (a set with large test cases) is never stored, so its retry answers `409 idempotency_result_unavailable` without writing again; read the set to reconcile. See [Idempotency](https://docs.gloo.com/forge/reference/idempotency.md#eval-set-write-replay).

## Running evaluations

`POST /api/v3/evals/{evalId}/runs` `{evalSetId, modelId}` queues a run and answers `201` with the `pending` `EvalRun`; `POST /api/v3/eval-runs/{runId}/cancel` `{id}` answers `200` with the run, now `cancelled`. Both bodies are strict.

Starting a run queues it. Each test case makes two metered model calls (the chosen model and the fixed judge model); cost accrues as cases run and is reported in `totalCost` (target plus judge) when the run finalizes. `completed` means finalized: `testCasesCompleted` below `testCasesTotal` means some cases never finished. Cancelling stops cases that have not started; cost already incurred is not refunded and recorded results remain. Eval spend is recorded under the platform's eval runner, not your organization's budget.

Machine-started runs are limited to 200 test cases and 2 in-progress runs per credential, and are refused while the platform's eval-runner, eval-judge or global budget is frozen or exhausted.

**Authority.** Both routes require `agents:write` and `inference:use`, and the credential's scopes must reach both: the authority is the pair. The run's set is then decided as for authoring: a set bound to an agent needs that agent readable (`agents:read` at its home, not PERSONAL) and `agents:write` there; an unbound set needs `agents:write` at the deployment root. A set or run you cannot see is `404`; one you can see but not write is `403` naming `agents:write`. Cancel also requires that the run's own agent, when it names one, is readable.

**Refusals.** Each is returned before anything is written or spent:

| Status | When |
| - | - |
| `400 bad_request` | an unknown key or empty value in the body, or "Model not available" (the `modelId` is not in the model catalog, is not active, is deprecated, or is not one the eval runner can call: it calls `anthropic/`, `gloo/` and `gloo-passthrough/` models) |
| `409 invalid_state` | "Cannot run an archived evaluation set" |
| `409 conflict` | "Eval set exceeds the 200-case run cap" (a set authored before the cap with more cases) |
| `409 conflict` | "Too many evaluation runs in progress for this credential (limit 2)": this credential already has 2 `pending` or `running` runs |
| `409 conflict` | "Too many evaluation runs are in progress to count this credential's runs; try again later": more than 1,000 runs are in progress across the deployment (or they are too large to read in one bounded check), so the limit cannot be checked |
| `409 conflict` | "Evaluation spend is blocked: …": an eval-runner, eval-judge or global budget is frozen by an operator or exhausted; the message names which |
| `409 invalid_state` | "Evaluation run already finished" (cancel of a `completed`, `failed` or `cancelled` run) |

A run judges every case with one fixed judge model across seven fixed dimensions; a set's `dimensions` and `goldenDatasetId` are not read. A run finalizes as `completed` even when some cases never finished, so compare `testCasesCompleted` with `testCasesTotal` (or use [compareRuns](#comparing-runs)) before trusting its scores. A cancelled run keeps `cancelled` and finalizes with the totals of the cases that ran. A run executes the set's test cases as they are when it is dispatched (a moment after the start is accepted), and each batch reads them when it runs: a case removed in between is recorded as failed without a model call, and a case edited in between runs as edited. Avoid editing a set's test cases while it has a run in progress.

A same-key `Idempotency-Key` retry replays the recorded answer only while the run stays visible to you and its set writable, and never starts or cancels a run again. See [Idempotency](https://docs.gloo.com/forge/reference/idempotency.md#eval-run-replay). Without a key, a repeated start queues a second run.

## Comparing runs

`GET /api/v3/eval-run-comparison?ids=<run>,<run>` compares 2-10 distinct runs, in the order given. It is computed on read and never stored. Every run must be visible to you; any hidden, missing or malformed id is one `404` for the whole request.

The response lists the `runs` (as `EvalRun`), `overallScoreDelta` and one entry per scored dimension with the run scores aligned to `runs` and a `delta`; each delta is the last run minus the first, or `null` when either is missing. `comparable` is `false`, with `reasons`, when:

* `different_eval_sets` — the runs belong to different eval sets;
* `not_completed` — a run is not `completed`;
* `incomplete` — a run has test cases that never finished or that failed. A run finalizes as `completed` even when some cases never finished, and a failed case is left out of its scores, so check this before trusting a delta.

The deltas are returned either way.

## Resource shape

* `EvalSet` (list): `id`, `name`, `description`, `agentDefinitionId`, `status`, `testCaseCount`, `tags`, `createdAt`, `updatedAt`.
* `EvalSetDetail` (get): the same fields with `testCases`, `dimensions` and `goldenDatasetId` instead of `testCaseCount`.
* `EvalRun` (get with `sub=runs`): `id`, `evalSetId`, `modelId`, `status`, `overallScore`, `aggregateScores`, `totalCost`, `totalTokens`, `testCasesTotal`, `testCasesCompleted`, `testCasesFailed`, `startedAt`, `completedAt`, `triggeredBy`, `startedByCaller`, `createdAt`.
* `EvalSetArchiveResult` (archive): `id`, `status` (`archived`), `disposition` (`archived` or `already_archived`).
* `EvalRunComparison` (compareRuns): `runs`, `comparable`, `reasons`, `overallScoreDelta`, `dimensions`.

An absent optional value is `null` (`[]` for `tags` and `dimensions`; `status` defaults to `"active"`). The set's creator, per-test-case run results and run errors are never returned.

A run's `triggeredBy` is `manual` (started in the Console), `cron` (the weekly regression evals) or `machine` (started through this API by an API key or External App); it never names the credential. `startedByCaller` is `true` only on a run your own credential started, so you can tell your runs from another credential's without learning its id.

## Operations

Reads:

* `GET /api/v3/evals` — [listEvals](https://docs.gloo.com/forge/reference/http-api/evals/list.md) — page the eval sets visible to the caller, newest first
* `GET /api/v3/evals/{evalId}` — [getEval](https://docs.gloo.com/forge/reference/http-api/evals/get.md) — fetch one visible eval set with its test cases
* `GET /api/v3/evals/{evalId}?sub=runs` — [getEval](https://docs.gloo.com/forge/reference/http-api/evals/get.md) — page that set's visible runs, newest first

Writes:

* `POST /api/v3/evals` — [createEvalSet](https://docs.gloo.com/forge/reference/http-api/evals/create.md) — create a set for an agent you may write, or a catalog set at the root
* `PATCH /api/v3/evals/{evalId}` — [updateEvalSet](https://docs.gloo.com/forge/reference/http-api/evals/update.md) — change a set's name, description, test cases or tags
* `DELETE /api/v3/evals/{evalId}` — [archiveEvalSet](https://docs.gloo.com/forge/reference/http-api/evals/archive.md) — archive a set, one-way

Runs (metered):

* `POST /api/v3/evals/{evalId}/runs` — [startEvalRun](https://docs.gloo.com/forge/reference/http-api/evals/startRun.md) — queue a run of a set you may write against a model
* `POST /api/v3/eval-runs/{runId}/cancel` — [cancelEvalRun](https://docs.gloo.com/forge/reference/http-api/evals/cancelRun.md) — cancel a pending or running run

Comparison:

* `GET /api/v3/eval-run-comparison?ids=` — [compareEvalRuns](https://docs.gloo.com/forge/reference/http-api/evals/compareRuns.md) — compare 2-10 visible runs

## Paging

`limit` (1–100, default 25; a larger value is `400` `too_big`) and `cursor` page both lists. A page can hold fewer rows than `limit`, or none, while `meta.hasMore` is still `true`: follow `meta.nextCursor` until `meta.hasMore` is `false`. A malformed `limit` is `400 bad_request`, and so is any `sub` value other than `runs`.

## Common workflow

```bash theme={null}
curl "$FORGE_BASE/api/v3/evals?limit=25" \
  -H "Authorization: Bearer $FORGE_TOKEN"

curl "$FORGE_BASE/api/v3/evals/$EVAL_SET_ID?sub=runs" \
  -H "Authorization: Bearer $FORGE_TOKEN"
```

```bash theme={null}
curl -X POST "$FORGE_BASE/api/v3/evals" \
  -H "Authorization: Bearer $FORGE_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"name": "Triage regression", "agentDefinitionId": "'"$AGENT_ID"'",
       "testCases": [{"id": "printer", "input": "Printer is on fire", "expectedOutput": "urgent"}]}'

curl -X POST "$FORGE_BASE/api/v3/evals/$EVAL_SET_ID/runs" \
  -H "Authorization: Bearer $FORGE_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"modelId": "'"$MODEL_ID"'"}'

curl "$FORGE_BASE/api/v3/eval-run-comparison?ids=$RUN_A,$RUN_B" \
  -H "Authorization: Bearer $FORGE_TOKEN"
```

Over MCP, `forge_call_operation` reaches these operations in the `agents` tool group: the reads as `listEvals` and `getEval` (with `sub: "runs"` for the set's runs), authoring as `createEvalSet`, `updateEvalSet` and `archiveEvalSet`, the comparison as `compareEvalRuns`, and metered runs as `startEvalRun` and `cancelEvalRun`.

## Related

* [Agents overview](https://docs.gloo.com/forge/build/define-an-agent.md) — conceptual background on agent quality and evaluation
* [Native HTTP quickstart](https://docs.gloo.com/forge/quickstart.md#raw-http) — call these endpoints with `fetch` and a bearer token


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.