> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Judges

> The Judges Catalog tab: each Judge's calibration and drift, verdict overrides, outcome ratings, and the provisional, approved and deprecated tier lifecycle.

The **Judges** section (`/platform/judges`) is the operator surface for every SYSTEM/JUDGE agent the
platform ships. Judges evaluate primitives (skills, agents, procedures) and workflow outcomes
against versioned rubrics. This page shows their calibration health, surfaces active drift alerts,
and exposes the tier-lifecycle controls that decide whether a Judge's verdicts can block actions
or are advisory only.

The Judges entry has two tabs: **Catalog**, this page, and
[Admissions](https://docs.gloo.com/forge/console/platform/judges-admissions.md) (`/platform/judges/admissions`). The Catalog
tab is admin-only.

![Judges catalog](https://mintcdn.com/gloo-b243725a/Ndtg9NEpwGc2OULA/forge/images/console/platform/judges-light.png?fit=max\&auto=format\&n=Ndtg9NEpwGc2OULA\&q=85\&s=a93d886fc206b0e09c9e52d292c10ffe)

Judges → Catalog — Forge Console at `/platform/judges`

## What you can do

* **Check each Judge's health.** The list shows tier, agreement rate, calibration band, active drift and cost.
* **Open a Judge.** See its calibration windows, verdict distribution, drift alerts, recent verdicts and rubric.
* **Disagree with a verdict.** Record an override from the recent-verdicts table; overrides feed the Judge's agreement rate.
* **Promote or demote a Judge.** Move it between `provisional`, `approved` and `deprecated` from its detail page.
* **Rate an outcome.** Give a workflow outcome a star rating, recorded against the outcome Judge's verdict when one ran.

## Why this exists

Judges replace ad-hoc inline LLM evaluations with first-class, named agents that emit a
standardized verdict. Standardization unlocks three things:

* **Calibration**. Operators can disagree with any Judge from any verdict-surfacing page; those
  overrides feed a per-Judge agreement-rate metric that a daily aggregator computes over
  rolling 7d, 30d, and 90d windows.
* **Promotion gates**. Provisional Judges return verdicts but those verdicts are advisory only —
  caller code does not treat them as authoritative until the Judge is promoted. Promotion requires
  at least 100 dispatches and an agreement rate of 0.85 or higher over the last 30 days.
* **Drift defense**. When a Judge's agreement rate or verdict distribution tracks outside the
  configured thresholds, Forge raises a drift alert. The list page surfaces the highest-
  severity active alert per Judge; the per-Judge detail page shows the full history.

## List page

The list page (`/platform/judges`) shows one row per Judge agent.

| Column | What it tells you |
| - | - |
| **Judge** | Name, evaluation domain, and stable `agentId` for cross-referencing |
| **Tier** | `provisional` (verdicts advisory only), `approved` (verdicts can block), or `deprecated` |
| **Rubric** | Rubric skill version and output schema version |
| **Dispatches (30d)** | Count from the most-recent 30d metrics row |
| **Agreement** | Operator-agreement rate over 30d — color-coded green at 0.85+, yellow 0.75–0.85, red below |
| **Calibration** | Derived health band: green / yellow / red / gray (gray = insufficient override sample) |
| **Drift** | The highest-severity active drift alert, or — if none |
| **Avg cost** | Average evaluation cost in cents across all dispatches in the 30d window |

Clicking a row opens the per-Judge detail page.

The page also renders an **Active drift alerts** banner at the top when the drift detector has
written any active alerts. The banner shows the most-recent five with their severity and message.

## Per-Judge detail page

`/platform/judges/[judgeId]` is the inspection surface for a single Judge.

### Calibration windows

Three side-by-side cards show the most-recent metrics row for the 7-day, 30-day, and 90-day rolling
windows. Each card displays: total dispatches, agreement rate, average verdict score, and average
evaluation cost. Cards show an empty state when the aggregator hasn't written a row for that window
yet.

### Tier lifecycle controls

The **Tier lifecycle** card exposes two admin actions:

* **Promote to approved**. Enabled only when the Judge is currently `provisional` and the 30-day
  thresholds are met (100+ dispatches, agreement rate at or above 0.85). When the precondition
  isn't met, the button shows the blocker inline ("Needs 32 more 30d dispatches" or "Needs
  agreement rate ≥85% (currently 79.3%)").
* **Demote…**. Always available. Expands a form: choose target tier (`provisional` or `deprecated`)
  and supply a reason (20+ characters). Demotion is also how auto-demote works — when the daily
  drift detector observes agreement rate below 0.60, the platform service calls this mutation under
  its own identity with a generated reason.

### Verdict distribution

A horizontal segmented bar shows the pass / needs\_work / fail mix over the 30-day window with
per-segment counts and percentages.

### Drift alerts

Active drift alerts for this Judge are listed below the distribution chart. Each alert shows:
severity, signal name, measured value, threshold, fired-at timestamp, and the message authored by
the detector. The detector writes alerts when any of these signals trip:

* `agreement_rate_warning` (under 0.75 over 30d)
* `agreement_rate_critical` (under 0.60 — also triggers auto-demote)
* `verdict_distribution_skew` (no fails recorded for an extended period suggests the Judge isn't
  catching failures)
* `cost_regression` (per-evaluation cost climbed beyond the budget)
* `zero_dispatches` (no traffic over a configured window — possible wiring breakage)

### Recent verdicts

The recent-verdicts table shows the last 25 calibration rows for this Judge — newest first. Each
row surfaces the verdict label, aggregate score, confidence, authoritative-vs-advisory band, and
override status. Rows without an override get an inline **Disagree** button.

#### Disagreeing with a verdict

Clicking **Disagree** opens the override modal. The form asks:

* **Override type**. One of *Disagree*, *Partial agreement*, *Disagree with modification*, or
  *Actually, I agree*. The last option exists so operators can record alignment — agreement signal
  matters as much as disagreement signal for calibration.
* **Per-criterion adjustments (optional)**. If the verdict came with per-criterion scores, sliders
  let you re-score any criterion. Unchanged sliders are not recorded.
* **Rationale (required)**. A one-sentence explanation, minimum 20 characters (the same floor the
  Judge's own rationale field uses).

Submission records an operator override on the calibration row. The audit trail is the row
itself; a later override on the same row replaces the earlier one.

### Rubric viewer

The rubric viewer surfaces the Judge's structured rubric — criterion ids, names, weights,
description, and scoring guidance — followed by an optional **SKILL.md prose** drawer that exposes
the full markdown source the Judge consumes as a granted skill on every dispatch.

When the rubric skill has no structured `rubric` payload (older Judges authored before structured
rubrics), only the SKILL.md prose is shown.

## Rate an outcome

Operators with `workflows:use` who own the workflow (or are a Platform admin) can rate a
materialized outcome: 5 stars plus optional notes, from **Rate outcome** in the run header or from
the outcome's dossier. Every rating is stored on the outcome and shown on its dossier. When
`judge-workflow-outcome` evaluated that attempt, the rating is also recorded as an
operator override on the Judge's calibration row, and the Judge's verdict and criterion rationale
are shown above the stars so the operator can agree or override with full context. When no outcome
Judge ran, the rating stands alone as the operator's outcome signal.

## Verdict strip on artifact detail pages

Every primitive detail page surfaces its latest Judge verdict inline via a compact
**verdict strip**. The strip lives at the top of the page (just under the page header) and
shows the disposition (Pass / Needs work / Fail), the headline score, and the count of findings
the Judge flagged. Clicking the strip opens a **verdict drawer** with the full breakdown.

Strip mount points:

| Page | Judge | What the strip shows |
| - | - | - |
| `/production/workflows/[workflowId]` | `judge-architect` + `judge-workflow-outcome` | Latest design-quality verdict and (when a run exists) the most-recent outcome verdict |
| `/platform/skills/[skillId]` | `judge-skill-quality` | Latest skill-quality verdict |
| `/workforce/agents/[agentId]` | `judge-agent-quality` | Latest agent-quality verdict |
| `/platform/procedures/[procedureId]` | `judge-procedure-correctness` | Latest procedure-correctness verdict |
| Workflow Architect drawer header | `judge-architect` | Live verdict for the workflow under edit (compact form) |

States the strip can render:

* **Loading** — first paint, queries haven't resolved yet.
* **Judging…** — a judge dispatch is live against this artifact right now. The strip reads the
  platform's dispatch-state record (one row per judge dispatch, opened before the model call and
  closed on verdict or failure), so lifecycle actions like promoting a procedure can never leave
  the chip stuck on "Judging…" after a verdict has landed. A dispatch that dies without closing
  its record clears from the chip on its own once the dispatch's retry budget is exhausted.
* **Judge failed** — the most recent dispatch ended in failure and no verdict has ever landed.
  Hover the pill for the error kind and message. Self-clears on the next successful evaluation.
* **Re-judge failed** (amber pill beside the score) — a verdict exists, but the most recent
  re-evaluation attempt failed, so the score shown predates that failure.
* **Not yet judged** — the artifact has never been evaluated by this Judge.
* **Disposition + score + finding count + sparkline** — the standard ready state.

### Verdict drawer

The drawer renders:

* **Per-criterion sub-scores** with horizontal bars (one per rubric criterion).
* **Findings list** with severity icons. Findings that carry a `suggestedAction` render a
  one-click **deep-link button** that navigates to the corresponding remediation page —
  e.g. "Open node 4 in graph editor" jumps to the workflow editor at the flagged node.
* **Rationale**, collapsible — the Judge's reasoning summary.
* **Trajectory** — a sparkline of recent verdicts for this artifact, so operators can see
  whether scores are trending up or down across edits.
* **Rubric link** to the Judge's rubric skill.

The button appears only when the recorded finding includes a supported remediation target. Treat
the destination as the place to inspect and correct the artifact; opening it does not accept the
Judge verdict or change the admission.

## Key state to watch

* **Agreement.** Red means the 30-day agreement rate is below 0.75; a gray calibration band means too few overrides to judge.
* **Active drift alerts.** The banner and the Drift column show the most severe active alert; a critical agreement rate also demotes the Judge.
* **Promotion blockers.** **Promote to approved** names what is missing, such as dispatches or agreement rate.

## Related

* [Admissions](https://docs.gloo.com/forge/console/platform/judges-admissions.md) for synthesized primitives a Judge panel has scored.
* Platform [Synthesis](https://docs.gloo.com/forge/console/platform/synthesis-queue) at `/platform/synthesis` for provisional primitives awaiting
  operator review.
* [Agent Performance](https://docs.gloo.com/forge/console/analytics/agent-performance) for the broader agent leaderboard.
* [Approvals](https://docs.gloo.com/forge/console/production/approvals.md) and [My Work](https://docs.gloo.com/forge/console/my-work.md) for work
  currently waiting on a person.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.