JudgeVerdictV1. Adding a new system Judge is a platform contributor task: it requires source access, a seeded system agent, a subject contract, a dispatch integration, and tests. It is not the same as creating an ordinary Worker agent through the public agent-creation API.
For an operator reviewing an existing Judge, start with Judges in Forge Console.
1. Define the subject and rubric
State exactly what the Judge receives, which evidence it may use, which decisions it informs, and how a caller handles missing or low-confidence evidence. Keep the subject bounded and canonical so the same input can be compared across rubric revisions. Author a rubric skill withSKILL.md and forge.yaml using skill authoring. Describe the criteria, weights, score bands, examples, and output contract. The built-in architect and skill-quality rubric skills provide concrete examples.
Keep rubricVersion synchronized with the rubric that actually runs. A version label should identify an evaluative change, not just a file edit.
2. Register a system Judge
Use an existing system Judge as the complete pattern. The system-agent seed contract uses fields such asdisplayName, modelRef, skillIds, and toolSetIds; it is distinct from the stored agent shape and from HTTP request bodies.
The dispatch admission invariant requires an agent of type SYSTEM, subtype JUDGE, tagged judge. Its judgeMetadata includes the domain, rubric skill and version, output schema version, confidence requirement, and minConfidenceForAuthoritative. Configure the rubric skill for always activation and select an enabled model appropriate for the rubric. Grant only the tools the subject requires.
The threshold is a confidence cutoff: the dispatch marks a valid verdict authoritative when its confidence meets the configured minimum. Raising that minimum makes the cutoff stricter. It does not itself improve the Judge’s calibration or make a low-confidence result authoritative.
For agent-facing evaluation tools, register the Judge ID in the Judge registry. Those wrappers use an allowlist to prevent arbitrary system-agent dispatch. Internal callers use a fixed Judge ID and the dispatch admission invariant rather than that name-level allowlist.
3. Integrate dispatch
The internal entry point isdispatchJudge, called through ctx.runAction with its generated internal reference. Its required arguments are:
Optional arguments include a criterion subset, triggering dispatch ID, and parent evidence handle. The result contains the parsed verdict and its
calibrationRowId. Do not pass a subject-builder function as an action argument: build the serializable subject first.
Use the existing architect and skill-quality auto-triggers to understand triggering and deduplication. For panels, use dispatchJudgePanel and the current consensus contract; see Judges. Recurring triggers belong in the platform cron manifest, following the repository’s scheduling contract.
Dispatch governs the evaluation, composes the agent, parses the verdict, and records calibration evidence. Malformed output can receive one corrective retry; a budget timeout is a distinct failure. The caller must distinguish a valid verdict, a non-authoritative verdict, a held evaluation, and an execution failure instead of treating all returned or missing results as approval.
4. Verify the integration
Before enabling a Judge for a consequential decision:- Test representative accepted, rejected, ambiguous, and missing-evidence subjects.
- Confirm the emitted verdict validates against the canonical Judge verdict schema and names the expected subject and rubric.
- Exercise governance refusal, invalid output, timeout, and low-confidence handling.
- Verify the caller consumes the authoritative/advisory distinction and records the expected result.
- Inspect the verdict in Console and test any supported suggested action against its actual route.
5. Calibrate and maintain
Use recorded operator overrides and representative evaluations to find systematic errors in criteria, bands, or evidence selection. Change the rubric deliberately and compare equivalent subject sets across versions. Judge metrics are rolled up by the manifest-backedjudge metrics rollup job; their existence does not guarantee a consumer automatically disables a drifting Judge.
A confidence score is not a measured accuracy rate. Keep the consumer’s decision policy explicit, and define what happens when model, rubric, evidence, or subject shape changes. See Console Judges for the available monitoring and override surfaces.
Do it over the API
Registering and integrating a Judge is source work; there is no API call that creates one. Once it is deployed, these reads confirm that it is registered with the rubric and threshold you set and that its verdicts carry the expected subject. They use theFORGE_BASE set up in the Raw HTTP tab of the quickstart. The quickstart’s ACCESS_TOKEN does not carry every scope they need, so mint it again with scope=platform:read from an app whose roles hold them (see Requesting scopes at token mint), or send a Personal key whose scopes include them.
Find the Judge and check its rubric version, confidence threshold and calibration health:
id as JUDGE_ID:
platform:read held at the platform scope; a credential whose grant covers only some org units gets 403. Verdicts on a Personal agent, or on an agent that no longer exists, are left out, so a page can hold fewer than limit while meta.hasMore is true; follow meta.nextCursor to read on.
Over MCP, forge_call_read_operation reads the same verdicts. It is in the default start job:
Related
- Judges: what judges grade and how verdicts count.
- Judges in Forge Console: the catalog and admissions an operator manages.
- Author a skill: the skill format a rubric builds on.

