Skip to content

Failure triage

Every failed run is classified: is this mini-ork's own bug, or the consumer's? When it is mini-ork's, the failure is filed as a bug report and — with a second opt-in — promoted to a framework-edit epic that the scheduler fixes in an isolated worktree. Triage replaces "a run broke, go read its logs by hand" with a queued, kickoff-driven fix.

The failure already carries its own evidence: the node that died, its finish_reason, and its log. Attribution is automated from that evidence, so the only human decision is whether to spend budget on the fix.

How it fits together

flowchart TD
    A[failed run<br/>task_runs.status = 'failed'] --> B[triage_run]
    B --> C[run_events node_end rows<br/>+ node logs]
    C --> D[attribute: NodeFailure → blame + evidence]
    D -->|mini_ork| E[bug_reports row<br/>deduped by fingerprint]
    D -->|consumer / unknown| Z[stop — not our bug]
    E -->|--promote| F[epics row, recipe=framework-edit<br/>+ kickoff under kickoffs/auto]
    F --> G[scheduler.pick_ready]
    G --> H[mini-ork run framework-edit<br/>isolated worktree + gate + rollback]

The bug → epic → scheduler → run pipeline already existed in mini-ork. Triage is the producer that feeds it when a run itself fails.

Blame rules

Attribution is fail-closed: the first matching rule wins, and anything unrecognised is unknown.

Signal Verdict Why
finish_reason in verdict_fail / verdict_revise consumer A legitimate "the artifact is wrong" verdict, not a bug
error and traceback frames inside mini_ork/** or a shipped recipes/*/verifiers/* mini_ork The crash is in framework code
Failing node path outside the mini-ork tree (run-local overlay, consumer recipe) consumer The caller's code
timeout, cost_limit, interrupted, provider/auth error (401, bad model) unknown Infrastructure, not a code bug
Anything else unknown No evidence to blame anyone

Two subtleties matter:

  • finish_reason='error' is not enough on its own. It covers harness crashes and legitimate capability failures, so it can never decide blame. The traceback frames are the discriminator.
  • Framework frames are matched by path shape, not by the triager's checkout root. Runs execute from linked worktrees and vendored copies, so the failing frame is almost never under the root of the process doing the triage. (Matching by root would produce false unknowns — and worse, false consumer blame on vendored checkouts.)

Running triage

mini-ork triage --run <run_id>
mini-ork triage --latest
mini-ork triage --run <run_id> --dry-run
mini-ork triage --run <run_id> --promote
mini-ork triage --run <run_id> --json
Flag Effect
--run <run_id> The failed run to attribute
--latest Attribute the most recent failed run
--dry-run Print blame, evidence rules and the would-be kickoff; write nothing
--promote File the bug report and queue the framework-edit epic
--json Machine-readable output for scripts and dashboards

Exit codes

Triage mirrors the scriptable contract certify uses:

Code Meaning Scriptable response
0 Blamed on mini-ork — a fix is warranted Proceed with --promote, or open the report
2 Usage error / no run given Fix the invocation
3 Consumer or unknown blame — no fix Report to the caller; do not auto-promote

A 3 is not a failure of the triage tool — it is a correct "this is not ours" answer, and it is distinguishable from a crash (2) by design.

Automatic triage on failure

Off by default, and double opt-in. In the run's fail branch, execute calls triage only when both conditions hold:

Variable Effect
MO_FAILURE_TRIAGE=1 Enable attribution + bug filing
MO_FAILURE_TRIAGE_PROMOTE=1 Additionally promote to a schedulable epic

The two flags are separate on purpose: emitting is safe, spending budget is not. Filing a report costs nothing; a promoted epic launches model runs.

Triage is best-effort. It is wrapped so it can never raise into the run that just failed — a triage bug must not turn one failure into two.

Guards

Dedupe by fingerprint

Bug reports key on a sha256 fingerprint of:

recipe | node | finish_reason | first error line

The same failure recurring N times reuses one row instead of inflating frequency. Two distinct crashes in the same node stay two reports, because the first error line differs.

Kill switch

touch "${MINI_ORK_HOME}/state/.triage-kill"

The sentinel stops filing without editing environment variables on every launcher. Diagnosis still works — only the write is suppressed. This mirrors the self-improve .self-improve-kill convention, so one place turns off unattended writes for an operator.

Budget and concurrency

The fix epic is dispatched by the scheduler, which already honours the daily budget and MO_SCHED_MAX_PARALLEL. Triage does not add a second spend path; it adds work to the queue the scheduler already governs.

Reading an attribution

A triage report states, in order:

  1. The fingerprint and the failure's identity (recipe, node, finish_reason).
  2. The blame (mini_ork, consumer, unknown) and the evidence rule that matched — for a framework blame, the matched frame path shape.
  3. The first error line used for dedupe.
  4. For --promote, the kickoff that will be queued, including the target paths the fix may touch.

The evidence rule is part of the output because a wrong blame is worse than no blame: an operator can check the cited signal against the run's logs.

Repair-first, triage on give-up

Triage is not the first response to a failed run. The recommended order is:

  1. Repair — resume the run, or relaunch it if the failure was transient (provider error, timeout, cost pause). See Recovery.
  2. Triage — if the failure repeats and looks like framework code, attribute it.
  3. Escalate — promote to framework-edit, which fixes at the source in a worktree with the mandatory gate and rollback.

Promoting a transient failure wastes budget; repairing a real framework bug loops forever. The blame rules exist to tell these apart.

Failure modes

Symptom Likely cause Fix
Blame unknown on an obvious crash The traceback never reached node_end Check the node log; the error may have been swallowed by a provider timeout
Blame consumer on a vendored checkout Frame matched outside the tree Report it — path-shape matching should catch mini_ork/ frames; include the frame path
No report filed on a mini_ork blame MO_FAILURE_TRIAGE unset, or the kill sentinel exists Set the flag; check ${MINI_ORK_HOME}/state/.triage-kill
Report filed but no epic --promote / MO_FAILURE_TRIAGE_PROMOTE not set Add the flag
Duplicate reports for one bug The first error line changed between runs Inspect both rows; the fingerprint is doing its job if the lines differ
Epic queued but never runs Scheduler daily budget exhausted or parallel cap reached Check the board; raise the budget or wait for reset