Skip to main content
This page is for operators running Forge in production. Every durable signal Forge produces is recorded in the platform’s own transactional store. Workflow runs, governance decisions, token spend, audit records and gateway health live there, most of them written in the same transaction as the work they describe, and the Console reads those same records live. There is no separate observability backend, no out-of-band ingestion path, and no privileged log channel. What the operator sees is what the system did. This page covers what the signals are, where they render, what they guarantee, and the features you configure on top of them: saved views and alert rules.
For trends over months and cost projections, use Budgets & Cost. For a tamper-evident compliance record rather than a debugging surface, go to Audit. To debug one workflow run end to end, start at Observe → Traces.

How observability is built

Every signal takes the same path from the moment it happens to the moment you see it:
  1. Recorded with the work. The workflow orchestrator, agent turns, Trust Fabric gates, sandbox VMs and scheduled jobs record what they did. Governance decisions are recorded in the same transaction as the dispatch decision. Some records are written slightly later, in a scheduled follow-up: an alert fire’s audit event is one. A missing event is therefore not proof on its own that nothing happened.
  2. Stored by kind. Events land in four durable streams — the unified event feed, the append-only audit record, token and cost data, and gateway health — described below.
  3. Correlated into a trace. Every run carries a trace ID, and every step a span ID, so a run’s whole span tree reassembles exactly.
  4. Rolled up for Analytics. Scheduled jobs roll raw events into hourly, daily and weekly buckets, so dashboards stay fast at any volume.
  5. Rendered live. The Console subscribes to the same records, so updates appear without refreshing.
  6. Acted on. Alert rules count matching events and notify you; saved views let you and your rules reuse the same filter.
That shape trades nothing away: the audit entry a compliance officer queries six months later is the same record that appeared in the operator’s live feed when it happened, and Analytics reads cheap pre-computed buckets without giving up the raw events underneath.

Signal sources

Four streams produce everything you can observe. They differ in who writes them, what they hold, and how long they last. Three guarantees follow:
  1. One live feed. Everything on Observe → Events was recorded first and is read live. There is no client-side merge between “live” and “stored” data.
  2. The audit record cannot be edited. Audit entries can only be appended; nothing updates or deletes them. Each one also appears in the live feed, but the audit entry is the record of truth.
  3. Cost is reproducible. Each model call stores the price it was charged at, so Analytics does not change when model prices do.

Operator surfaces — Observe and Analytics

Observability renders across two Forge Console sections. Observe (the Watchtower) is the live and investigative surface: Scheduled workflows and platform jobs, with next run time, last status and cron expression, are under Platform → Schedules. Analytics (the Ledger) is the rollup surface for trend analysis — Cost, Throughput, Reliability and Agent Performance — backed by pre-computed buckets so it stays fast at any volume. See Budgets & Cost. Always on: the Needs Attention panel appears on every Console page. It shows pending approvals with how long they have waited, recent failures (workflow failed, step failed, safety gate blocked, deployment failed) and the attention items below, so you see trouble without navigating away.

Traces — debugging a single run

A run contains steps, and each step has one execution record: status, agent, model, token usage, cost, error, and two timestamps — when the orchestrator dispatched the step and when the agent returned. Queue wait is the gap between a predecessor’s completion and the next step’s start. Traces answer the three core debugging questions: Each step also carries token attribution — input, output and cached tokens, the model, and the cost at capture time — which links the trace directly to Analytics.

How a run’s trace is assembled

Every signal in a run is correlated two ways:
  • By run. Step results, execution events, token events and audit events all carry the run’s ID.
  • By trace context. Each run gets a W3C-shaped trace ID (32 hex characters) and a root span ID when it is created. Each step inherits the run’s trace ID, gets its own span ID, and names the run’s root span as its parent. The span tree reassembles exactly, with no fuzzy matching.
Trace IDs start with the creation time in seconds, so traces from the same incident window sort together — which is what makes “everything that ran in the last 30 minutes” a usable query on the Traces page. The trace context stays inside Forge: Forge does not send a traceparent header or export spans to an external tracing system, so a downstream system cannot join its own traces to a Forge run by trace ID. In the other direction, a Platform API request may carry a valid traceparent: the response echoes it beside X-Request-Id, and the request’s log line pairs your trace ID with the request ID (Trace context). That trace ID is not stamped on any run the request starts. Runs created before trace IDs existed do not appear on the Traces page, because they have no spans. The empty state links to Production → Workflows to launch a fresh run.

Analytics rollups

Analytics reads pre-computed buckets instead of scanning months of raw events. Scheduled jobs roll each raw source up by hour, day or week: What this means on screen:
  • Freshness is visible. Each bucket records when it was computed, so a dashboard can tell you how recent its numbers are.
  • Truncation is flagged. If a period held more raw events than a rollup job could scan, the bucket is marked as overflowed. Drill into the raw events for that period before you trust its total.
  • Rollups converge. Each job re-scans an overlapping window and rewrites the same buckets, so a re-run never double-counts.
  • Cost data compacts. Hourly cost buckets compact into daily after 30 days and daily into weekly after 90 days; raw token events are removed at 90 days.

Saved views — re-applying a filter

The Events, Run History and Logs pages keep their filters in the URL. A saved view stores that filter under a name so you can re-apply it with one click.
  • Personal by default. Only you can read or change a view you create.
  • Shareable with your org unit. Share a view and every operator in your org unit can read it. Only the owner can change it; sharing publishes a view, it does not delegate ownership.
  • Per page. A view saved on Events does not appear on Run History or Logs.
Common patterns:
  • Team triage view — share a “step failed, severity critical” view with your on-call team so everyone opens the same predicate.
  • Investigation snapshot — save the filter you used during an incident and reopen it at the post-mortem.
  • Alert companion — point an alert rule at a saved view, so the rule and the operator look at the same thing.

Alerts

An alert rule counts matching events in a sliding window and notifies you when the count crosses a threshold — for example “page me on 3 or more errors in 5 minutes”, or “tell me if the nightly sync stops reporting”. Alert rules are separate from budget, cost and failure-rate alerts, which watch aggregate values rather than event counts. Configure event-count rules in Observe → Alerts. Analytics → Alerts holds the budget, cost and failure-rate alerts.

What a rule is made of

Threshold modes

“At least 0” is refused when you save the rule, because it is always true. Use “exactly 0” for a deadman, or a count of 1 or more.

How rules are evaluated

Forge evaluates every enabled rule once a minute. For each rule it skips the rule if it is inside its suppression window, counts the matching events in the window (up to 1,000), and fires if the threshold is met. The worst-case time from the triggering events to the alert is the window plus one minute. A firing rule sends a message titled Observability alert: <rule name> with the body <count> matching events in the last <N>min, or No matching events in the last <N>min (deadman) for a deadman rule.
  • Notification sink — the alert goes to the notification bus, which delivers it on every channel your deployment has enabled: the in-app bell, Slack, email and webhooks. In-app, the alert is broadcast to every signed-in caller across the deployment, not just people in the rule’s org unit, so keep sensitive team or customer details out of rule names.
  • Slack sink — the alert posts to the Slack channel you named. Use a channel ID (C… or G…), not a #name; the Forge bot must be a member of the channel. See Slack — Alert Slack sink.
A failure in one sink never stops other rules from being evaluated. Every fire is also recorded as an observability_alert_fired audit event, which appears in the Events stream; exclude that event from a rule whose filter could otherwise match its own fires.

Filters with several categories

A filter that selects several categories counts each category separately and combines the results. A rule can use at most 5 categories; split a broader saved view into several rules.

Dormant rules

A rule that cannot be evaluated goes dormant rather than firing or silently counting zero (which would set off every deadman rule). The rule’s row shows why: Forge records one observability_alert_dormant audit event each time a rule becomes dormant, not on every minute it stays dormant. When the rule evaluates again, it leaves the dormant state.

Who can see and change rules

Anyone with workflows:read in an org unit can see every rule in it, with their own rules marked. Only a rule’s owner — or a platform administrator — can change its threshold, enable or disable it, or remove it.

Limits

The per-org-unit cap keeps the once-a-minute evaluation bounded, so one tenant’s rules cannot crowd out another’s.

Attention items — what Needs Attention surfaces

The Needs Attention panel ranks four kinds of attention item into one list: Each item shows what changed in the six hours before it, drawn from the audit record, so you can see what preceded a failure. Acknowledge an item to clear it; acknowledgement persists across sessions.

What machines can observe

Observe is an operator console. A machine credential (an External App or a platform API key) does not read it; it follows the surfaces built for machines: Every run read is decided at the run’s home — the run’s own org unit, or its workflow’s when the run has none — and at every company the run carries. A run you cannot read is a 404, never a partial answer. Paging. runs/listEvents returns milestones oldest first, up to 100 per page (default 25). A page may be short while meta.hasMore is true: keep following meta.nextCursor until hasMore is false. The timeline is milestones only: tool calls, model messages, error payloads, agent identity and cost amounts are not returned. Use runs/get for status and outputs and cost/summary for cost. Reconciliation. Webhook deliveries are retried independently and carry no order. Store the last nextCursor you processed from GET /api/v3/events, page forward from it after any outage, and deduplicate on the event id. The event log keeps 30 days. Console-only. These Observe surfaces have no machine route, by decision rather than omission:
  • Logs and Observe settings — deployment-wide raw runtime text with no organization fence; it may carry prompts or personal data.
  • Traces and trace snapshots — raw tool input and output and model messages; snapshots are shareable copies. The run timeline above is the machine view.
  • Saved views — a person’s own filter state.
  • Observe and Analytics alert rules — person-owned notification side effects over deployment-wide counts, or deployment administration. Machines subscribe to webhooks instead.
  • The Events stream and Run History — the Events stream is deployment-wide; machines use the event log, runs and cost.

Operator runbook

When you stand up the observability surface for the first time:
  1. Pin the Needs Attention panel open. Pending approvals and recent failures are the two signals you do not want to navigate away from.
  2. Filter Events for incidents. The category list is comprehensive but noisy. For incident response, filter to step failed, workflow failed, safety gate blocked and deployment failed, and save that as a shared view.
  3. Use Traces, not Logs, as your debugger. Logs help when the runtime itself misbehaves. For most “why did this fail” questions, a trace’s conversation log, tool calls and error fields are faster.
  4. Cross-check audit and events. Every audit entry has a mirror in the event feed. The audit entry is the compliance record; the event is the operator view. When you find one, the other is one click away.
  5. Watch the gateway health chip. A degraded gateway is the most common cause of cascading step failures. If the chip is red, test that hypothesis first.
  6. Give every critical path a deadman. An “exactly 0 in 24 hours” rule on a job’s completion event tells you when it stops running, which no error-count rule can.

Do it over the API

These calls use the FORGE_BASE and ACCESS_TOKEN set up in the Raw HTTP tab of the quickstart. Saved views and alert rules are configured in Console only (Observe → Alerts); a machine reads a run’s progress from its timeline instead.
This needs runs:read at the run’s home; a run you cannot read is a 404. Each type is one of checkpoint, handoff_initiated, handoff_completed, guardrail_tripped, intervention, completion, error or cost_milestone. Follow meta.nextCursor while meta.hasMore is true. Over MCP, forge_call_read_operation reads the same timeline: