> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> The signals Forge records and where they render: how a run's trace is assembled, how Analytics rollups stay current, and how to set saved views and alert rules.

This page is for operators running Forge in production. **Every durable signal Forge produces is recorded in the platform's own transactional store.** Workflow runs, governance decisions, token spend, audit records and gateway health live there, most of them written in the same transaction as the work they describe, and the Console reads those same records live. There is no separate observability backend, no out-of-band ingestion path, and no privileged log channel. What the operator sees is what the system did.

This page covers what the signals are, where they render, what they guarantee, and the features you configure on top of them: saved views and alert rules.

> [!NOTE]
>
> For trends over months and cost projections, use [Budgets & Cost](https://docs.gloo.com/forge/run/budgets-and-cost.md). For a tamper-evident compliance record rather than a debugging surface, go to [Audit](https://docs.gloo.com/forge/run/audit.md). To debug one workflow run end to end, start at **Observe → Traces**.

***

## How observability is built

Every signal takes the same path from the moment it happens to the moment you see it:

1. **Recorded with the work.** The workflow orchestrator, agent turns, Trust Fabric gates, sandbox VMs and scheduled jobs record what they did. Governance decisions are recorded in the same transaction as the dispatch decision. Some records are written slightly later, in a scheduled follow-up: an alert fire's audit event is one. A missing event is therefore not proof on its own that nothing happened.
2. **Stored by kind.** Events land in four durable streams — the unified event feed, the append-only audit record, token and cost data, and gateway health — described below.
3. **Correlated into a trace.** Every run carries a trace ID, and every step a span ID, so a run's whole span tree reassembles exactly.
4. **Rolled up for Analytics.** Scheduled jobs roll raw events into hourly, daily and weekly buckets, so dashboards stay fast at any volume.
5. **Rendered live.** The Console subscribes to the same records, so updates appear without refreshing.
6. **Acted on.** Alert rules count matching events and notify you; saved views let you and your rules reuse the same filter.

That shape trades nothing away: the audit entry a compliance officer queries six months later is the same record that appeared in the operator's live feed when it happened, and Analytics reads cheap pre-computed buckets without giving up the raw events underneath.

***

## Signal sources

Four streams produce everything you can observe. They differ in who writes them, what they hold, and how long they last.

| Signal | What it records | Kept for | Where you see it |
| - | - | - | - |
| **Platform events** | Execution events, the audit fan-out, routing fallbacks, webhooks, gateway events | 365 days | **Observe → Events** (the unified timeline) |
| **Audit events** | Append-only record of governed actions; each also appears in the unified feed | 365 days | **Observe → Run History** and **Trust Fabric → Decisions** |
| **Token and cost data** | Every model call with its tokens and the price at the moment of the call, rolled up hourly, daily and weekly | 90 days raw, then compacted | **Analytics** |
| **Gateway health** | Capability provider status, checked every 5 minutes | Live status; health snapshots are kept 7 days | The gateway chip on Observe, the **Needs Attention** panel |

Three guarantees follow:

1. **One live feed.** Everything on Observe → Events was recorded first and is read live. There is no client-side merge between "live" and "stored" data.
2. **The audit record cannot be edited.** Audit entries can only be appended; nothing updates or deletes them. Each one also appears in the live feed, but the audit entry is the record of truth.
3. **Cost is reproducible.** Each model call stores the price it was charged at, so Analytics does not change when model prices do.

***

## Operator surfaces — Observe and Analytics

Observability renders across two Forge Console sections.

**Observe (the Watchtower)** is the live and investigative surface:

| Tool | What it shows |
| - | - |
| **Events** | The unified live timeline: rich or compact display, run grouping, category chips, agent and gateway filters, CSV export. |
| **Run History** | Audit-grade, persistent view. Use it when the question is "what actually happened, and can I prove it later?" |
| **Logs** | Terminal-style raw runtime feed: level, message and agent context. Ephemeral. |
| **Traces** | Step-level execution detail: queue wait and execution time, conversation log, tool calls, token attribution. |
| **Alerts** | The alert rules you and your teammates have configured. |

Scheduled workflows and platform jobs, with next run time, last status and cron expression, are under **Platform → Schedules**.

**Analytics (the Ledger)** is the rollup surface for trend analysis — Cost, Throughput, Reliability and Agent Performance — backed by pre-computed buckets so it stays fast at any volume. See [Budgets & Cost](https://docs.gloo.com/forge/run/budgets-and-cost.md).

**Always on:** the **Needs Attention** panel appears on every Console page. It shows pending approvals with how long they have waited, recent failures (workflow failed, step failed, safety gate blocked, deployment failed) and the attention items below, so you see trouble without navigating away.

***

## Traces — debugging a single run

A run contains steps, and each step has one execution record: status, agent, model, token usage, cost, error, and two timestamps — when the orchestrator dispatched the step and when the agent returned. Queue wait is the gap between a predecessor's completion and the next step's start.

Traces answer the three core debugging questions:

| Question | Where to look |
| - | - |
| "Why did this step fail?" | The step's error and error code, plus the last assistant message in the conversation log |
| "Why was this step slow?" | The step's start-to-completion time, and the gap after its predecessor completed |
| "What did the agent decide?" | The conversation log: the agent's reasoning, tool calls and output |

Each step also carries token attribution — input, output and cached tokens, the model, and the cost at capture time — which links the trace directly to Analytics.

### How a run's trace is assembled

Every signal in a run is correlated two ways:

* **By run.** Step results, execution events, token events and audit events all carry the run's ID.
* **By trace context.** Each run gets a W3C-shaped trace ID (32 hex characters) and a root span ID when it is created. Each step inherits the run's trace ID, gets its own span ID, and names the run's root span as its parent. The span tree reassembles exactly, with no fuzzy matching.

Trace IDs start with the creation time in seconds, so traces from the same incident window sort together — which is what makes "everything that ran in the last 30 minutes" a usable query on the Traces page.

The trace context stays inside Forge: Forge does not send a `traceparent` header or export spans to an external tracing system, so a downstream system cannot join its own traces to a Forge run by trace ID. In the other direction, a Platform API request may carry a valid `traceparent`: the response echoes it beside `X-Request-Id`, and the request's log line pairs your trace ID with the request ID ([Trace context](https://docs.gloo.com/forge/reference/conventions.md#trace-context)). That trace ID is not stamped on any run the request starts.

Runs created before trace IDs existed do not appear on the Traces page, because they have no spans. The empty state links to **Production → Workflows** to launch a fresh run.

***

## Analytics rollups

Analytics reads pre-computed buckets instead of scanning months of raw events. Scheduled jobs roll each raw source up by hour, day or week:

| Rollup | Grain | What it covers |
| - | - | - |
| Workflow runs | Hourly and daily | Run counts, outcomes, duration distributions |
| Governance decisions | Hourly and daily | Allow, block and hold counts per gate, agent and action type |
| Approvals | Daily | Requests, decisions, time to resolution |
| Approval policies | Daily | How often each policy triggered |
| Step errors | Daily | Failures by error code, agent and gate |
| Budgets | Daily | Spend trajectory per budget envelope |
| Model cost | Daily | Cost and tokens by model, optionally by agent |
| Factories | Daily | Task completion, cost, worker utilization |
| Chat | Hourly | Message and session counts by channel |
| Gateway health | Hourly | Uptime and failure counts per capability provider |
| Agent activity | Hourly, daily and weekly | Turns, tool calls and errors per agent |

What this means on screen:

* **Freshness is visible.** Each bucket records when it was computed, so a dashboard can tell you how recent its numbers are.
* **Truncation is flagged.** If a period held more raw events than a rollup job could scan, the bucket is marked as overflowed. Drill into the raw events for that period before you trust its total.
* **Rollups converge.** Each job re-scans an overlapping window and rewrites the same buckets, so a re-run never double-counts.
* **Cost data compacts.** Hourly cost buckets compact into daily after 30 days and daily into weekly after 90 days; raw token events are removed at 90 days.

***

## Saved views — re-applying a filter

The Events, Run History and Logs pages keep their filters in the URL. A **saved view** stores that filter under a name so you can re-apply it with one click.

* **Personal by default.** Only you can read or change a view you create.
* **Shareable with your org unit.** Share a view and every operator in your org unit can read it. Only the owner can change it; sharing publishes a view, it does not delegate ownership.
* **Per page.** A view saved on Events does not appear on Run History or Logs.

Common patterns:

* **Team triage view** — share a "step failed, severity critical" view with your on-call team so everyone opens the same predicate.
* **Investigation snapshot** — save the filter you used during an incident and reopen it at the post-mortem.
* **Alert companion** — point an alert rule at a saved view, so the rule and the operator look at the same thing.

***

## Alerts

An **alert rule** counts matching events in a sliding window and notifies you when the count crosses a threshold — for example "page me on 3 or more errors in 5 minutes", or "tell me if the nightly sync stops reporting". Alert rules are separate from budget, cost and failure-rate alerts, which watch aggregate values rather than event counts.

Configure event-count rules in **Observe → Alerts**. **Analytics → Alerts** holds the budget, cost and failure-rate alerts.

### What a rule is made of

| Part | What you set |
| - | - |
| **Name** | Up to 120 characters. Appears in the alert title and the Slack message. |
| **Filter** | A saved view, or a filter written inline. With neither, every event in the window matches. |
| **Threshold** | A mode, a count and a window. The window is at least 1 minute. |
| **Sink** | **Notification** (with severity info, warn or critical) or **Slack** (with a channel ID). |
| **Suppression** | How long to stay quiet after firing. Default 15 minutes, minimum 1 minute. |
| **Enabled** | Disabled rules are not evaluated. |

### Threshold modes

| Mode | Fires when | Common use |
| - | - | - |
| **At least** | The count is greater than or equal to the threshold | "3 or more errors in 5 minutes" |
| **Exactly** | The count equals the threshold | A **deadman** alert, with a count of 0: fire when nothing matching arrives |

"At least 0" is refused when you save the rule, because it is always true. Use "exactly 0" for a deadman, or a count of 1 or more.

### How rules are evaluated

Forge evaluates every enabled rule once a minute. For each rule it skips the rule if it is inside its suppression window, counts the matching events in the window (up to 1,000), and fires if the threshold is met. The worst-case time from the triggering events to the alert is the window plus one minute.

A firing rule sends a message titled `Observability alert: <rule name>` with the body `<count> matching events in the last <N>min`, or `No matching events in the last <N>min (deadman)` for a deadman rule.

* **Notification sink** — the alert goes to the notification bus, which delivers it on every channel your deployment has enabled: the in-app bell, Slack, email and webhooks. In-app, the alert is broadcast to every signed-in caller across the deployment, not just people in the rule's org unit, so keep sensitive team or customer details out of rule names.
* **Slack sink** — the alert posts to the Slack channel you named. Use a channel ID (`C…` or `G…`), not a `#name`; the Forge bot must be a member of the channel. See [Slack — Alert Slack sink](https://docs.gloo.com/forge/run/slack.md#alert-slack-sink).

A failure in one sink never stops other rules from being evaluated. Every fire is also recorded as an `observability_alert_fired` audit event, which appears in the Events stream; exclude that event from a rule whose filter could otherwise match its own fires.

### Filters with several categories

A filter that selects several categories counts each category separately and combines the results. A rule can use at most 5 categories; split a broader saved view into several rules.

### Dormant rules

A rule that cannot be evaluated goes **dormant** rather than firing or silently counting zero (which would set off every deadman rule). The rule's row shows why:

| Reason | Meaning | Fix |
| - | - | - |
| Owner cannot read events | The event stream is deployment-wide, so a rule counts only while its owner holds `grants:read`, `policies:read` and `audit:read` at platform scope. | Restore the owner's access, or have someone who holds it recreate the rule. |
| Saved view deleted | The rule's saved view no longer exists. | Point the rule at another view, or write the filter inline. |
| Saved view not usable | The rule names a view that is neither its owner's nor shared into the rule's org unit. | Use the owner's own view or a view shared into that org unit. |

Forge records one `observability_alert_dormant` audit event each time a rule becomes dormant, not on every minute it stays dormant. When the rule evaluates again, it leaves the dormant state.

### Who can see and change rules

Anyone with `workflows:read` in an org unit can see every rule in it, with their own rules marked. Only a rule's owner — or a platform administrator — can change its threshold, enable or disable it, or remove it.

### Limits

| Limit | Value |
| - | - |
| Enabled rules per org unit | 50 |
| Minimum window | 60 seconds |
| Minimum suppression | 60 seconds |
| Events counted per rule per evaluation | 1,000 |
| Rules evaluated per minute | 500 |

The per-org-unit cap keeps the once-a-minute evaluation bounded, so one tenant's rules cannot crowd out another's.

***

## Attention items — what Needs Attention surfaces

The **Needs Attention** panel ranks four kinds of attention item into one list:

| Item | What raises it | What drives its severity |
| - | - | - |
| **Run failed** | A workflow run failed in the last 24 hours | Whether the failure was transient, persistent or a governance block |
| **Agent paused for budget** | An agent was paused because a budget ran out | Which budget level tripped: agent, envelope or hierarchy |
| **Approval aging** | An approval has waited more than an hour | Wait time and the policy that raised it |
| **Event delivery failed** | A notification or webhook delivery dead-lettered | Sink type and retry count |

Each item shows what changed in the six hours before it, drawn from the audit record, so you can see what preceded a failure. Acknowledge an item to clear it; acknowledgement persists across sessions.

***

## What machines can observe

Observe is an operator console. A machine credential (an External App or a platform API key) does not read it; it follows the surfaces built for machines:

| Need | Route | Permission (scope) |
| - | - | - |
| A run's status, outputs and results | [`GET /api/v3/runs`](https://docs.gloo.com/forge/reference/http-api/runs/list), [`GET /api/v3/runs/{runId}`](https://docs.gloo.com/forge/reference/http-api/runs/get) | `runs:read` |
| How a run progressed: checkpoints, handoffs, guardrail trips, interventions, cost milestones, errors and completion | [`GET /api/v3/runs/{runId}/events`](https://docs.gloo.com/forge/reference/http-api/runs/listEvents) | `runs:read` |
| Whether a run is held for a person | [`GET /api/v3/runs/{runId}/trust-status`](https://docs.gloo.com/forge/reference/http-api/runs/getTrustStatus) | `runs:read` |
| Being told when something happens | [Webhooks](https://docs.gloo.com/forge/run/webhooks.md), reconciled against [`GET /api/v3/events`](https://docs.gloo.com/forge/reference/http-api/events) | `events:read` plus each event type's scope |
| Spend and budgets | [`GET /api/v3/cost-summary`](https://docs.gloo.com/forge/reference/http-api/cost/summary), [`GET /api/v3/me/budget-status`](https://docs.gloo.com/forge/reference/http-api/budgets/getStatus) | `cost:read` |
| Platform health | [`GET /api/v3/health`](https://docs.gloo.com/forge/reference/http-api/health/get) | `platform:read` |

Every run read is decided at the run's home — the run's own org unit, or its workflow's when the run has none — and at every company the run carries. A run you cannot read is a `404`, never a partial answer.

**Paging.** `runs/listEvents` returns milestones oldest first, up to 100 per page (default 25). A page may be short while `meta.hasMore` is `true`: keep following `meta.nextCursor` until `hasMore` is `false`. The timeline is milestones only: tool calls, model messages, error payloads, agent identity and cost amounts are not returned. Use `runs/get` for status and outputs and `cost/summary` for cost.

**Reconciliation.** Webhook deliveries are retried independently and carry no order. Store the last `nextCursor` you processed from `GET /api/v3/events`, page forward from it after any outage, and deduplicate on the event `id`. The event log keeps 30 days.

**Console-only.** These Observe surfaces have no machine route, by decision rather than omission:

* **Logs and Observe settings** — deployment-wide raw runtime text with no organization fence; it may carry prompts or personal data.
* **Traces and trace snapshots** — raw tool input and output and model messages; snapshots are shareable copies. The run timeline above is the machine view.
* **Saved views** — a person's own filter state.
* **Observe and Analytics alert rules** — person-owned notification side effects over deployment-wide counts, or deployment administration. Machines subscribe to webhooks instead.
* **The Events stream and Run History** — the Events stream is deployment-wide; machines use the event log, runs and cost.

***

## Operator runbook

When you stand up the observability surface for the first time:

1. **Pin the Needs Attention panel open.** Pending approvals and recent failures are the two signals you do not want to navigate away from.
2. **Filter Events for incidents.** The category list is comprehensive but noisy. For incident response, filter to step failed, workflow failed, safety gate blocked and deployment failed, and save that as a shared view.
3. **Use Traces, not Logs, as your debugger.** Logs help when the runtime itself misbehaves. For most "why did this fail" questions, a trace's conversation log, tool calls and error fields are faster.
4. **Cross-check audit and events.** Every audit entry has a mirror in the event feed. The audit entry is the compliance record; the event is the operator view. When you find one, the other is one click away.
5. **Watch the gateway health chip.** A degraded gateway is the most common cause of cascading step failures. If the chip is red, test that hypothesis first.
6. **Give every critical path a deadman.** An "exactly 0 in 24 hours" rule on a job's completion event tells you when it stops running, which no error-count rule can.

***

## Do it over the API

These calls use the `FORGE_BASE` and `ACCESS_TOKEN` set up in the Raw HTTP tab of the [quickstart](https://docs.gloo.com/forge/quickstart.md). Saved views and alert rules are configured in Console only ([Observe → Alerts](https://docs.gloo.com/forge/console/observe/alerts.md)); a machine reads a run's progress from its timeline instead.

```bash theme={null}
curl -sS "$FORGE_BASE/api/v3/runs/$RUN_ID/events?limit=100" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  | jq '.data[] | {sequence, type, timestamp, durationMs, traceId}'
```

This needs `runs:read` at the run's home; a run you cannot read is a `404`. Each `type` is one of `checkpoint`, `handoff_initiated`, `handoff_completed`, `guardrail_tripped`, `intervention`, `completion`, `error` or `cost_milestone`. Follow `meta.nextCursor` while `meta.hasMore` is true.

Over MCP, `forge_call_read_operation` reads the same timeline:

```json theme={null}
{ "operationId": "listRunEvents", "input": { "runId": "<run id>", "limit": 100 } }
```

***

## Related

* [Audit](https://docs.gloo.com/forge/run/audit.md)
* [Budgets & Cost](https://docs.gloo.com/forge/run/budgets-and-cost.md)
* [Trust Fabric in production](https://docs.gloo.com/forge/run/trust-fabric.md)
* [Workflow execution](https://docs.gloo.com/forge/architecture/flows/workflow-execution.md)
* [Observe → Alerts](https://docs.gloo.com/forge/console/observe/alerts.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.