Skip to main content
Use Knowledge → Search quality (/knowledge/quality) to evaluate whether retrieval finds the artifacts a representative question should return. Viewing requires knowledge:read; creating, editing, archiving, and running golden sets requires knowledge:manage.
Search quality (dark)Search quality (light)

Search quality — Forge Console at /knowledge/quality

What you can do

Run a useful evaluation

  1. Choose New golden set and name the behavior you want to measure, such as “Finance policy retrieval.”
  2. Choose the scopes the test searches. Add representative questions and the knowledge keys each should retrieve.
  3. Save the set, select it, and choose Run now.
  4. Wait for a completed run. Run scheduled confirms scheduling, not a successful evaluation.
  5. Open Show trend, inspect the score and failing questions, and compare with the baseline for the same retrieval configuration.
Keep a golden set stable once it has runs. If the desired behavior changes, create a new set rather than changing expected answers to hide a regression.

Read scores and history

Search quality (/knowledge/quality) answers “is retrieval any good?” It measures knowledge search against golden sets: lists of questions, each with the knowledge keys a good search returns for it. You see a golden set only when you can read every scope it searches, because a run shows which keys came back from those scopes. Archived golden sets are listed too, marked archived, so their past runs stay available. Viewing requires knowledge:read. Each golden set is a card showing:
  • its scope and the namespaces it searches;
  • the newest run’s recall@k and MRR, and how they compare with the baseline for the same embedding model, chunker version, k and ranking version;
  • Regressed when the newest run fell below that baseline, Invalid when it measured nothing, Baseline when the newest run is its configuration’s baseline, and OK otherwise. A set with no runs says “No runs yet” and shows no score.
  • for the first run under a new chunker version, which becomes the baseline for its configuration, a separately labelled comparison with the previous chunker’s baseline (same embedding model and k), and whether recall@k or MRR fell after the chunker change.
Show trend opens the set below the cards and puts it in the URL (?set=):
  • a line chart of recall@k and MRR over time for one retrieval configuration (embedding model, chunker version, k and ranking version): the newest valid run’s. Baseline runs are labelled. Runs under another configuration are counted, not plotted, because each configuration has its own baseline. Invalid runs are listed under the chart, not plotted.
  • the golden queries the newest valid run did not fully answer, worst first: the expected keys, the keys that came back (best first), recall and reciprocal rank. Each key opens the Knowledge Base searched for that key (a case key or a source ID). A key of the form entry:<id>, for an entry with neither, opens that entry.
Runs can be requested here or produced by the platform’s scheduled retrieval evaluation.

Authoring golden sets

With knowledge:manage you can also author golden sets and run them:
  • New golden set opens the editor: a name, one or more scopes you can read (Add scope, up to 50), and golden queries, each with its expected keys as a comma list (Add query, up to 100). The editor lists what to fix before it saves.
  • Edit on the selected set opens the same editor. A set with runs opens read-only except for its name and Archive, because its runs stay comparable only while its scopes and queries do. Archived sets keep their history; create a new set to change what is measured.
  • Run now schedules one run of the selected active set. It is refused for a minute after a run or an earlier Run now, even while that run is still queued. The button stays disabled for a minute after a run, and shows “Run scheduled” once the run is queued. The run records its embedding model, chunker version, k and ranking version, and is compared with that configuration’s baseline.
A run measures the ranking agents are served by knowledge search: native hits weighted by grade, read as a reader with no source access, so a source-restricted case never counts as retrieved. Its ranking version names that measurement. Runs recorded before it measured the store’s raw order and have no ranking version, so the first run under the served ranking starts its own baseline instead of being compared with them.

Interpret the metrics

Recall@k measures how many expected keys appear among the first k results. MRR rewards the first relevant result appearing early. A high aggregate score can still hide an important failing question; inspect the failing-query table before changing production configuration.

Recover from a problem

Key state to watch

  • Regressed — the newest run fell below the baseline for the same retrieval configuration.
  • Invalid — the run measured nothing; it is not a zero-quality verdict.
  • Run scheduled — the run is queued, not finished.
  • A changed configuration — a new embedding model, chunker version, k or ranking version starts its own baseline.
See Knowledge settings, Sources, and Health.
  • Health: re-embed and index status.
  • Settings: the ranking weights search applies.
  • Knowledge: search and browse artifacts.