⚙️ Execution Phases
WORKFLOW EXECUTION — 8 PHASES
1
Run Start· Convex
workflowRuns.createCreates workflowRuns record with pinned YAML, version, gateway, role bindings, status “running”.
↓
2
Parse & Validate· Convex
interpreter.tsParses YAML, validates against Zod schema: unique step IDs, valid deps, non-empty commands.
↓
3
DAG Resolution· Convex
resolveExecutionLayers()Topological sort on step deps. Produces ordered execution layers — steps in a layer run in parallel.
↓
4
Governance Pipeline· Convex
evaluateGovernance()10-gate pipeline: health, agent, concurrency, rate, budget x2, trust x2, policy, approval.
passblockhold
if passed↓
5
Runtime Execution· Gateway (via Bridge)
dispatch.ts → BridgeConvex dispatches to Bridge. Bridge delegates to adapter. Results flow back via callback.
↓
6
Variable Scope & Output· Convex
coerceOutputs()Coerce outputs, validate against schema, store for downstream $step_id.stdout references.
↓
7
Failure Handling· Convex
on_failure policyon_failure policy: fail (stop run), continue (skip), retry (3x backoff), retry_once_then_escalate.
↓
8
Completion· Convex
workflowRuns statusAll steps resolved (completed, skipped, or failed with continue). Run status set to completed or failed.
📋 Phase Details
1. Run Start
A workflow run is created through one of two paths:- Manual trigger via an operator interface (public mutation)
- Programmatic trigger via the Platform API or another Convex function
- The pinned YAML (immutable for the lifetime of the run)
- Workflow version, gateway ID, role bindings from the active deployment
- Runtime args passed by the caller
- Initial status
"running"and astartedAttimestamp
2. Parse & Validate
Validation covers three categories:| Category | Checks |
|---|---|
| Structural | Unique step IDs, non-empty steps array, snake_case naming |
| Cross-reference | Every depends_on target exists, every stdin reference points to a valid step, every phase reference maps to a declared phase |
| Completeness | Every non-approval, non-sub-workflow, non-library step has a command |
library_step field), the interpreter resolves them from the stepLibrary table and merges defaults — the workflow definition always wins for fields it explicitly sets.
3. DAG Resolution
TheresolveExecutionLayers() function runs a topological sort variant:
- Infer dependencies from both explicit
depends_onarrays and implicitstdinreferences (pattern:$step_id.stdout) - Iterate until all steps are placed in layers. Each pass collects steps whose dependencies are all resolved into a new layer.
- Detect cycles — if a pass produces zero new resolvable steps while steps remain, the engine logs a circular-dependency error and forces remaining steps into a final layer.
4. Governance Pipeline
Every step dispatch runs through the full governance pipeline. See Governance Pipeline for the complete gate-by-gate breakdown. The three possible outcomes:| Disposition | Meaning | Effect |
|---|---|---|
| pass | All gates passed | Step is dispatched to Bridge |
| block | A gate rejected the dispatch | Step is marked failed with the gate’s error code |
| hold | A policy requires approval | An approval record is created; the interpreter sleeps until resolved |
5. Runtime Execution
Convex dispatches viadispatch.ts:
- Resolve the adapter for the target gateway
- Build the dispatch payload with command, stdin, context, timeout, and callback URL
- POST to Bridge
/api/v2/dispatch - Bridge delegates to
adapter.execute()with the normalizedExecutionRequest - The adapter runs the task on the gateway runtime
- Bridge delivers the
ExecutionResultto the Convex callback URL
Dispatch is fire-and-forget from Convex’s perspective. The interpreter moves on to dispatch other ready steps and waits for completion events.
6. Variable Scope
Step outputs are accessible to downstream steps via the$step_id.stdout reference pattern. The interpreter:
- Coerces outputs to expected types using
coerceOutputs() - Validates outputs against the step’s declared output schema using
validateOutputs() - Evaluates success criteria if the step defines them using
evaluateSuccessCriteria() - Stores outputs for downstream reference
7. Failure Handling
Each step declares anon_failure policy:
| Policy | Behavior |
|---|---|
fail | Stop the entire run immediately |
continue | Mark the step as skipped, advance the DAG |
retry | Retry up to 3 times with exponential backoff (1s, 2s, 4s) |
retry_once_then_escalate | Retry once, then create a escalation approval |
agent_busy), the interpreter also retries with backoff before applying the step’s failure policy.
8. Completion
The interpreter marks the run as completed when all steps are resolved. A step is “resolved” if it is in one of these terminal states:completed, skipped, failed (with continue policy), or cancelled.
If the run completes with any step failures, the run status reflects the worst outcome.
🔐 Ownership Matrix
| Phase | Owner | Key file |
|---|---|---|
| Run start | Convex | convex/workflowRuns.ts |
| Parse & validate | Convex | convex/interpreter.ts, packages/shared/src/lobsterx-schema.ts |
| DAG resolution | Convex | convex/interpreter.ts |
| Governance | Convex | convex/lib/governanceEngine.ts, convex/lib/gates.ts |
| Runtime execution | Gateway (via Bridge) | convex/dispatch.ts, packages/bridge/src/server.ts |
| Variable scope | Convex | convex/validation.ts |
| Failure handling | Convex | convex/interpreter.ts |
| Completion | Convex | convex/interpreter.ts, convex/workflowRuns.ts |
🛡️ Execution Hardening
Two cron jobs protect against stalled executions:| Cron | Frequency | Purpose |
|---|---|---|
| Workflow watchdog | Every 2 minutes | Detects runs where all steps resolved but the interpreter stalled. Resumes the workflow. |
| Step timeout enforcer | Every 60 seconds | Checks running steps against their timeout. Marks timed-out steps as failed and applies their on_failure policy. |