Skip to main content
POST
Create an evaluation set
Creates an active eval set for an agent you may write (agents:write at the agent’s home org unit, and the agent readable and not PERSONAL), or an unbound catalog set with agents:write at the deployment root. An agent you may not read or write is 404 “Agent not found”. dimensions and goldenDatasetId are refused: the eval runner never reads them. Creating a set starts no run and spends nothing. See Evals.

Authorizations

Authorization
string
header
required

An External App: the JWT it obtains from POST /oauth/token with OAuth2 client_credentials. Pass as Authorization: Bearer <jwt>. An app's static companion key also authenticates as the app, but is deprecated and is not protected by Idempotency-Key.

Headers

Idempotency-Key
string

1–255 printable ASCII characters (0x21–0x7E, no spaces), scoped to the calling credential. A repeat with the same key and request never runs the write again, a 5xx included: once the operation's permissions pass again it answers the identical status and body of the first response (Idempotent-Replayed: true, also sent as Idempotency-Replayed: true), unless the operation's own replay policy withholds it; a first response that was not stored (over 32 KiB, or carrying a one-time secret) answers its first status with only data.id (Idempotency-Replay-Disclosure: id-only) when it named one; otherwise 409 idempotency_result_unavailable; a repeat while the first is still running is 409 idempotency_key_in_flight, and the same key with a different request is 422 idempotency_key_reuse. Honoured for External App JWTs and platform and Personal keys; an External App's static companion key is not protected.

Required string length: 1 - 255

Body

application/json

A new evaluation set. Test case ids are unique within the set, and all test cases together are at most 800000 bytes as JSON. dimensions and goldenDatasetId are refused (the eval runner never reads them); the status starts active. Authoring a set starts no run and spends nothing.

name
string
required
Required string length: 1 - 200
testCases
object[]
required
Required array length: 1 - 200 elements
description
string
Maximum string length: 2000
agentDefinitionId
string
Minimum string length: 1
tags
string[]
Maximum array length: 20
Maximum string length: 64

Response

Create an evaluation set

data
object
required

An EvalSet with its test cases, dimensions and golden dataset id instead of the test-case count. The creator is never returned.

meta
object
required