Skip to main content
POST
Start an evaluation run
Queues a metered run of an evaluation set you may write against modelId and answers 201 with the pending run. Starting a run queues it. Each test case makes two metered model calls (the chosen model and the fixed judge model); cost accrues as cases run and is reported in totalCost (target plus judge) when the run finalizes. completed means finalized: testCasesCompleted below testCasesTotal means some cases never finished. Eval spend is recorded under the platform’s eval runner, not your organization’s budget. Machine-started runs are limited to 200 test cases and 2 in-progress runs per credential, and are refused while the platform’s eval-runner, eval-judge or global budget is frozen or exhausted. See Evals.

Authorizations

Authorization
string
header
required

An External App: the JWT it obtains from POST /oauth/token with OAuth2 client_credentials. Pass as Authorization: Bearer <jwt>. An app's static companion key also authenticates as the app, but is deprecated and is not protected by Idempotency-Key.

Headers

Idempotency-Key
string

1–255 printable ASCII characters (0x21–0x7E, no spaces), scoped to the calling credential. A repeat with the same key and request never runs the write again, a 5xx included: once the operation's permissions pass again it answers the identical status and body of the first response (Idempotent-Replayed: true, also sent as Idempotency-Replayed: true), unless the operation's own replay policy withholds it; a first response that was not stored (over 32 KiB, or carrying a one-time secret) answers its first status with only data.id (Idempotency-Replay-Disclosure: id-only) when it named one; otherwise 409 idempotency_result_unavailable; a repeat while the first is still running is 409 idempotency_key_in_flight, and the same key with a different request is 422 idempotency_key_reuse. Honoured for External App JWTs and platform and Personal keys; an External App's static companion key is not protected.

Required string length: 1 - 255

Path Parameters

evalId
string
required
Minimum string length: 1

Body

application/json

The evaluation set to run and the model to run it against (a modelId from GET /api/v2/models/list). Starting a run queues it. Each test case makes two metered model calls (the chosen model and the fixed judge model); cost accrues as cases run and is reported in totalCost (target plus judge) when the run finalizes. completed means finalized: testCasesCompleted below testCasesTotal means some cases never finished. Cancelling stops cases that have not started; cost already incurred is not refunded and recorded results remain. Eval spend is recorded under the platform's eval runner, not your organization's budget. Machine-started runs are limited to 200 test cases and 2 in-progress runs per credential, and are refused while the platform's eval-runner, eval-judge or global budget is frozen or exhausted.

modelId
string
required
Minimum string length: 1

Response

Start an evaluation run

data
object
required

One run of a visible evaluation set. A run that names its own agent is returned only when the caller may read that agent. Per-test-case results and errors are not returned; absent scores, cost, tokens, counters and times are null.

meta
object
required