Skip to content

Recovery

A failed or interrupted run is not a reason to start over. mini-ork keeps durable state for every node, so recovery reuses what already succeeded and re-dispatches only what did not. The hard part is choosing the right tool: the four recovery commands are not interchangeable.

Choose the path

flowchart TD
  F["run stopped or failed"] --> Q1{"still running<br/>on a remote node?"}
  Q1 -- yes --> RE["recover --strategy reattach"]
  Q1 -- no --> Q2{"cost-pause<br/>sentinel?"}
  Q2 -- yes --> RS["resume (clears sentinel)"]
  Q2 -- no --> Q3{"known bad node<br/>or stale evidence?"}
  Q3 -- yes --> RC["recover --from-node <id>"]
  Q3 -- no --> Q4{"process died<br/>mid-run?"}
  Q4 -- yes --> RR["run --resume <run-id>"]
  Q4 -- no --> RT["board retry <run-id>"]
Command Acts on What it does
run --resume <run-id> A run's step ledger Skip succeeded steps, re-dispatch failed or never-ran ones
resume A cost-pause sentinel Clear the pause and continue the same run
recover <run-id> The workflow DAG Reuse valid checkpoints, dispatch the minimal dependency closure
board retry <run-id> One failed run Re-dispatch it, subject to the retry gates

run --resume

mini-ork run --resume <run-id>
mini-ork run --resume <run-id> --checkpoint-id <step-id>

Semantics:

  • nodes that succeeded are skipped, with persisted outputs intact;
  • failed, errored or never-ran nodes are re-dispatched;
  • one [mini-ork-resume] line states the skip and dispatch sets before anything executes;
  • an unknown run id is a hard error — there is no silent fresh start.

Exit codes:

rc Meaning
0 Resumed and completed
2 Usage error — bad flags or missing run id
64 Unknown run
65 Corrupt or unreadable run state
70 Internal error

--checkpoint-id rewinds to an earlier step; everything from that step onward is re-dispatched. Rewinding is safe for the run's own state, but not for side effects already committed to the target tree — see the warning below.

resume (cost pause)

A run pauses when a budget or daily-spend cap trips. The pause is recorded as a sentinel, and mini-ork resume clears it and continues. The default pause interval is set by MO_PAUSE_EVERY_USD (default 25).

mini-ork resume            # clear the sentinel and continue

Do not clear a cost sentinel to "see if it works" — the next node will trip the same cap. Raise the cap first, or wait for the daily reset.

recover

recover is the DAG-aware tool. It walks the workflow, marks each node reusable via the engine's checkpoint predicate, and dispatches only the earliest non-reusable node plus its transitive dependents. It is read-only until it dispatches.

mini-ork recover <run-id> --status           # print reuse/rerun split, change nothing
mini-ork recover <run-id> --strategy pause   # compute the plan, do not dispatch
mini-ork recover <run-id>                    # default strategy: resume

Strategies:

Strategy Behaviour
resume (default) Start at the closure root, reuse every valid checkpoint
retry Start at the closure root as a fresh attempt
repair Like retry, with a bounded cost ceiling
pause Compute and print the plan; dispatch nothing
reattach Re-attach to a still-running remote process; the default when an unconsumed remote-process row exists for the first incomplete node
verify Re-run only the verifier chain; every upstream node must be reusable

--verify refuses when an upstream LLM node has no reusable checkpoint: <node> has no reusable checkpoint — use --strategy resume.

Options that matter in practice:

Flag Use
--from-node <id> Widen the rerun by rooting the closure at a specific node
--status Show the reuse/rerun split and the cost boundary without dispatching
--carry-patch <name> Apply a run-dir patch to the target tree before dispatch when the run was rolled back but the implementer checkpoint is being reused
--ack-change Proceed past a needs_change retry hint
--force Override a retryable: false hint and dispatch anyway
--lane <alias>=<lane> Pin a workflow lane alias for the resumed run

Lane pins are written to runs/<id>/config/agents.recover.yaml and logged to runs/<id>/recover-lanes.log; the resumed execute reads them through MINI_ORK_AGENTS.

Retry hints and the change gate

recover consults <run_dir>/retry-hint.json, produced by the retry-hint machinery. Two fields dominate the decision:

  • retryable: false — the hint says a retry cannot help. --force overrides it; don't, unless you can name what changed.
  • needs_change — the evidence says the input must change before a retry is meaningful. --ack-change asserts you changed it.

A hint also carries the strategy it recommends and the node to start from. Read it before overriding it:

cat .mini-ork/runs/<run-id>/retry-hint.json

board retry

mini-ork board retry <run-id> [--dry-run] [--ack-change] [--force] [--lane <lane>]

board retry re-dispatches one failed run. It is gated:

  • --dry-run prints what would be dispatched and why, without dispatching;
  • a needs_change hint requires --ack-change;
  • retryable: false requires --force;
  • --lane pins the lane for this retry.

Prefer board retry --dry-run before any retry you cannot fully explain.

Checkpoints and restore

mini-ork board checkpoints <run-id>          # list recorded checkpoints
mini-ork board restore <run-id> <node>       # restore a node's checkpoint

Restore brings a node's recorded state back so it can be reused. It does not rewind the target repository. If the tree already contains later commits, decide deliberately whether that is what you want.

What recovery will not do

Recovery is not a time machine for the target tree

Checkpoint reuse assumes the tree a node would run against still matches the tree it ran against. When the base has moved — someone pushed, a previous attempt committed, or a worktree was reset — a forced replay can compare against a different ancestor and produce a false result. Start a fresh run, or apply the intended patch explicitly with --carry-patch. Do not force a replay whose base you cannot name.

Other hard rules:

  • Never git reset --hard on shared main, and never revert a merged commit to make a replay pass.
  • Never git stash inside a worktree — it sweeps other sessions' files.
  • Do not re-run a failing step unchanged. Two identical attempts with identical evidence are one attempt plus noise.
  • Do not clear a cost sentinel instead of raising the cap.
  • A blocked descendant is a signal, not a chore. Fix the parent; do not tick the consumer by hand.

After recovery

  1. Confirm the run's verdict and the evidence it cites.
  2. If the change was published, verify the target's base is the one the run measured against.
  3. If the same failure recurs, escalate per Failure triage instead of retrying again.