> ## Documentation Index
> Fetch the complete documentation index at: https://docs.strait.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# DAG Operations Playbook

> Operator runbook for diagnosing, recovering, and hardening workflow DAG executions in production.

This playbook is for on-call and platform operators handling stuck, slow, or failing workflow runs.

For engine internals, see [DAG Runtime](/concepts/dag-runtime). For the conceptual model, see [Workflows](/concepts/workflows).

## 1) Fast triage checklist

When a workflow appears stalled or degraded:

1. Fetch runtime graph: `GET /v1/workflow-runs/{workflowRunID}/graph`
2. Fetch decision stream: `GET /v1/workflow-runs/{workflowRunID}/explain`
3. Confirm workflow policy: `GET /v1/workflow-policies/{projectID}`
4. Inspect waiting event triggers (if `wait_for_event` / `sleep` is used)
5. Decide recovery scope: single-step retry vs subtree replay

## 2) What to look for

### Graph signals (`/graph`)

* `runnable` is non-empty but run does not progress: usually scheduler blockers (`max_parallel_steps`, `concurrency_key`, or `resource_class` limits).
* `critical_path_remaining_ms` growing over time: likely blocked long-running branch.
* High number of `pending` nodes with dependencies complete: progression callback/reconciliation issue.

### Explain signals (`/explain`)

Filter with `step_ref` and `decision_type` to reduce noise.

* `decision_type=resource`: step blocked by `resource_class` quota.
* `decision_type=concurrency`: `concurrency_key` lock contention.
* `decision_type=scheduler`: workflow-level parallelism cap reached.
* `decision_type=condition`: step intentionally skipped due to condition evaluation.

## 3) Recovery actions

### A) Retry one terminal step (smallest blast radius)

Use when failure is local and downstream branch should continue from that step.

```bash theme={null}
curl -X POST "https://strait.dev/v1/workflow-runs/{workflowRunID}/steps/{stepRef}/retry" \
  -H "Authorization: Bearer strait_live_abc123"
```

### B) Replay subtree (branch-scoped replay)

Use when the selected step and its descendants should be recomputed.

```bash theme={null}
curl -X POST "https://strait.dev/v1/workflow-runs/{workflowRunID}/steps/{stepRef}/replay-subtree" \
  -H "Authorization: Bearer strait_live_abc123"
```

### C) Let reaper reconcile stalled runs

Tune and enable auto-healing behavior:

* `WF_STALL_THRESHOLD` (default `15m`)
* `WF_STALL_ACTION`:
  * `log_only` (observe only)
  * `reconcile` (resume callback)
  * `fail_workflow` (terminal fail-safe)

## 4) Policy tuning guardrails

Set project policy via:

* `PUT /v1/workflow-policies/{projectID}`

Key fields:

* `max_fan_out`: controls branch explosion.
* `max_depth`: limits chain depth.
* `forbidden_step_types`: disallow risky/unsupported step kinds in a project.
* `require_approval_for_deploy`: enforce manual gate on deploy DAGs.

## 5) Common incident patterns

### Pattern: Deploy step never starts

Typical findings:

* `/graph`: step is runnable but not running.
* `/explain`: `resource` or `concurrency` blocks.

Actions:

1. Confirm there is no intentional policy restriction.
2. Retry failed terminal dependency if present.
3. If branch state is inconsistent, replay subtree from closest failed ancestor.

### Pattern: Workflow waits forever on external input

Typical findings:

* Waiting `wait_for_event` step.
* Event trigger still `waiting` near/after timeout.

Actions:

1. Verify event key and sender integration path.
2. Send/cancel event trigger depending on business intent.
3. If timeout handling is too strict, adjust workflow failure policy and timeout config in next version.

### Pattern: High fan-out DAG causes prolonged queue pressure

Typical findings:

* Very large runnable set.
* Frequent scheduler/resource block decisions.

Actions:

1. Reduce fan-out in DAG design.
2. Increase capacity in a controlled way (resource classes / worker sizing).
3. Enforce stricter `max_fan_out` policy to prevent recurrence.

## 6) Post-incident hardening

After recovery, capture and apply one or more of:

* Workflow policy update (`max_fan_out`, `max_depth`, forbidden types)
* Better `concurrency_key` strategy to avoid accidental serialization hotspots
* More accurate `resource_class` assignments per step
* Additional alerting on stalled run and explain decision rates
* Regression tests for the exact progression edge case observed

## 7) Minimum evidence to close incident

* Graph snapshot before/after recovery
* Explain decision sample for impacted step(s)
* Recovery action used (`retry` or `replay-subtree`) with timestamp
* Final workflow terminal state and run ID(s)
* Follow-up backlog item for permanent fix (policy/config/workflow design/test)