Skip to main content
Evals are the platform’s evaluation harness for measuring agent and workflow quality over a fixed test set. Over the API you can read sets and runs, author sets for agents you may write, start and cancel metered runs with scopes reaching both agents:write and inference:use, and compare runs.

Visibility

The reads require agents:read, and each returns only what the caller may read:
  • An eval set bound to an agent (agentDefinitionId set) is visible only when the caller holds agents:read at that agent’s home org unit and the agent is not PERSONAL. A set bound to a missing agent is not visible.
  • An unbound eval set is platform catalog: it is visible only when the caller holds agents:read at the deployment root.
  • A run is visible only when its set is visible and, when the run names its own agent, that agent is visible too.
Keys and External Apps with platform-scope grants (every migrated API key) hold agents:read at every home, so they still see every set whose agent exists and is not PERSONAL. A caller whose grants are scoped to org units sees only the sets bound to agents in those units, and no unbound sets. A missing set and a set you may not read are the same 404 resource_missing.

Authoring

Creating, updating and archiving a set require agents:write, decided again at the set’s home:
  • A set bound to an agent: you must be able to read the agent (agents:read at its home org unit, not PERSONAL) and hold agents:write there. Creating a set for an agent you cannot read or write is 404 “Agent not found”.
  • An unbound (catalog) set: agents:write at the deployment root. A credential whose grants stop at an org unit cannot create or edit catalog sets.
  • A set you cannot read is 404, like a missing one; a set you can read but not write is 403 naming agents:write.
The bodies are strict, and any other key is 400 bad_request: dimensions and goldenDatasetId are refused: the eval runner scores a fixed set of dimensions and never reads either field. A new set is active; machines cannot change a set’s status, agent binding, dimensions or golden dataset. Update replaces testCases as a whole; send expectedUpdatedAt (the updatedAt you read) to refuse a stale edit with 409 conflict. An archived set cannot be edited (409 invalid_state), and archiving is one-way: a second archive answers already_archived and writes nothing. Authoring spends nothing. Creating or editing a set starts no run. Sets created through the API are never run by the weekly regression evals, which run only sets created in the Console or seeded. For the same reason, the API cannot change the test cases of a Console or seeded set (409 conflict): you can rename, re-describe, retag or archive it, but what its weekly run executes is changed only in the Console. A same-key Idempotency-Key retry replays the recorded answer only while you may still read and write the set. An answer above the 32 KiB idempotency storage ceiling (a set with large test cases) is never stored, so its retry answers 409 idempotency_result_unavailable without writing again; read the set to reconcile. See Idempotency.

Running evaluations

POST /api/v3/evals/{evalId}/runs {evalSetId, modelId} queues a run and answers 201 with the pending EvalRun; POST /api/v3/eval-runs/{runId}/cancel {id} answers 200 with the run, now cancelled. Both bodies are strict. Starting a run queues it. Each test case makes two metered model calls (the chosen model and the fixed judge model); cost accrues as cases run and is reported in totalCost (target plus judge) when the run finalizes. completed means finalized: testCasesCompleted below testCasesTotal means some cases never finished. Cancelling stops cases that have not started; cost already incurred is not refunded and recorded results remain. Eval spend is recorded under the platform’s eval runner, not your organization’s budget. Machine-started runs are limited to 200 test cases and 2 in-progress runs per credential, and are refused while the platform’s eval-runner, eval-judge or global budget is frozen or exhausted. Authority. Both routes require agents:write and inference:use, and the credential’s scopes must reach both: the authority is the pair. The run’s set is then decided as for authoring: a set bound to an agent needs that agent readable (agents:read at its home, not PERSONAL) and agents:write there; an unbound set needs agents:write at the deployment root. A set or run you cannot see is 404; one you can see but not write is 403 naming agents:write. Cancel also requires that the run’s own agent, when it names one, is readable. Refusals. Each is returned before anything is written or spent: A run judges every case with one fixed judge model across seven fixed dimensions; a set’s dimensions and goldenDatasetId are not read. A run finalizes as completed even when some cases never finished, so compare testCasesCompleted with testCasesTotal (or use compareRuns) before trusting its scores. A cancelled run keeps cancelled and finalizes with the totals of the cases that ran. A run executes the set’s test cases as they are when it is dispatched (a moment after the start is accepted), and each batch reads them when it runs: a case removed in between is recorded as failed without a model call, and a case edited in between runs as edited. Avoid editing a set’s test cases while it has a run in progress. A same-key Idempotency-Key retry replays the recorded answer only while the run stays visible to you and its set writable, and never starts or cancels a run again. See Idempotency. Without a key, a repeated start queues a second run.

Comparing runs

GET /api/v3/eval-run-comparison?ids=<run>,<run> compares 2-10 distinct runs, in the order given. It is computed on read and never stored. Every run must be visible to you; any hidden, missing or malformed id is one 404 for the whole request. The response lists the runs (as EvalRun), overallScoreDelta and one entry per scored dimension with the run scores aligned to runs and a delta; each delta is the last run minus the first, or null when either is missing. comparable is false, with reasons, when:
  • different_eval_sets — the runs belong to different eval sets;
  • not_completed — a run is not completed;
  • incomplete — a run has test cases that never finished or that failed. A run finalizes as completed even when some cases never finished, and a failed case is left out of its scores, so check this before trusting a delta.
The deltas are returned either way.

Resource shape

  • EvalSet (list): id, name, description, agentDefinitionId, status, testCaseCount, tags, createdAt, updatedAt.
  • EvalSetDetail (get): the same fields with testCases, dimensions and goldenDatasetId instead of testCaseCount.
  • EvalRun (get with sub=runs): id, evalSetId, modelId, status, overallScore, aggregateScores, totalCost, totalTokens, testCasesTotal, testCasesCompleted, testCasesFailed, startedAt, completedAt, triggeredBy, startedByCaller, createdAt.
  • EvalSetArchiveResult (archive): id, status (archived), disposition (archived or already_archived).
  • EvalRunComparison (compareRuns): runs, comparable, reasons, overallScoreDelta, dimensions.
An absent optional value is null ([] for tags and dimensions; status defaults to "active"). The set’s creator, per-test-case run results and run errors are never returned. A run’s triggeredBy is manual (started in the Console), cron (the weekly regression evals) or machine (started through this API by an API key or External App); it never names the credential. startedByCaller is true only on a run your own credential started, so you can tell your runs from another credential’s without learning its id.

Operations

Reads:
  • GET /api/v3/evals — listEvals — page the eval sets visible to the caller, newest first
  • GET /api/v3/evals/{evalId} — getEval — fetch one visible eval set with its test cases
  • GET /api/v3/evals/{evalId}?sub=runs — getEval — page that set’s visible runs, newest first
Writes:
  • POST /api/v3/evals — createEvalSet — create a set for an agent you may write, or a catalog set at the root
  • PATCH /api/v3/evals/{evalId} — updateEvalSet — change a set’s name, description, test cases or tags
  • DELETE /api/v3/evals/{evalId} — archiveEvalSet — archive a set, one-way
Runs (metered):
  • POST /api/v3/evals/{evalId}/runs — startEvalRun — queue a run of a set you may write against a model
  • POST /api/v3/eval-runs/{runId}/cancel — cancelEvalRun — cancel a pending or running run
Comparison:
  • GET /api/v3/eval-run-comparison?ids= — compareEvalRuns — compare 2-10 visible runs

Paging

limit (1–100, default 25; a larger value is 400 too_big) and cursor page both lists. A page can hold fewer rows than limit, or none, while meta.hasMore is still true: follow meta.nextCursor until meta.hasMore is false. A malformed limit is 400 bad_request, and so is any sub value other than runs.

Common workflow

Over MCP, forge_call_operation reaches these operations in the agents tool group: the reads as listEvals and getEval (with sub: "runs" for the set’s runs), authoring as createEvalSet, updateEvalSet and archiveEvalSet, the comparison as compareEvalRuns, and metered runs as startEvalRun and cancelEvalRun.