Skip to content

Learning loop

Every run teaches the next one. mini-ork records what each dispatch actually did, turns those traces into policy at end-of-run, and — when the evidence is thick enough — routes the next run's nodes onto the lanes that measured better. When the evidence is thin, routing falls back to the recipe's configured lane.

That asymmetry is the design: learning can only add an override once it has earned the samples, so turning the loop on by default can never make a cold system worse.

Write half, read half

flowchart LR
    subgraph RUN_N["Run N — write"]
        D[_dispatch_node] -->|chosen lane| T[execution_traces<br/>agent_version_id + status]
        T --> PRM[PRM scorer<br/>process_reward 0..1]
        PRM --> EOR[end-of-run writers]
        EOR --> GRPO[(agent_performance_memory<br/>relative_advantage)]
        EOR --> COND[(conductor_decisions<br/>outcome + realized_score)]
        RHO[reflect / conductor] --> WIN[(prompt_win_rates<br/>wins / losses / ties)]
    end
    subgraph RUN_N1["Run N+1 — read"]
        R[learning_governed<br/>routing seam] -->|strict AND-gate| GRPO
        R -->|strict AND-gate| WIN
        R -->|override or fallback| D2[_dispatch_node]
    end
    GRPO -.-> R
    WIN -.-> R

The critical join is lane identity. The lane a node actually ran on is stamped onto its trace (agent_version_id), and GRPO scores exactly that column. If the writer keyed on anything else, every function would exist and the loop would still be open.

The five signals

Signal Table Written Captures
PRM — process reward execution_traces.process_reward Inline, per node Heuristic 0–1 quality of one node's work
GRPO — group-relative advantage agent_performance_memory.relative_advantage End of run Which lane beats its peers on a task class
RHO — prompt win rates prompt_win_rates Reflect / conductor Which prompt version wins per task class
Conductor writeback conductor_decisions.outcome / realized_score End of run Whether the chosen topology or recipe paid off
Grounded rejections grounded_rejections Review / oracle gates Refuted claims plus evidence — an anti-reward channel

PRM: scoring a node's work

PRM is a heuristic v1, additive and capped at 1.0. It covers the obvious degenerate cases — no tool calls, no files, a vacuous status — before GRPO consumes the score.

Signal Points
status = success +0.40
tool_calls non-empty (the agent did work) +0.20
files_written or files_read non-empty +0.10
reviewer_verdict in {APPROVE, pass, success, ok} +0.15
duration_ms in [1000, 600000] (not too fast or slow) +0.10
cost_usd > 0 (a real call, not a stub) +0.05

Why a heuristic, not a learned reward model

The PRM literature uses a trained reward model. mini-ork ships the heuristic first because it is auditable and free: an operator can recompute the score from a trace row by hand. It is deliberately coarse — its job is to keep "the node did nothing" from looking as good as "the node did the work".

GRPO: which lane wins

At end of run, traces are grouped by (node_type, task_class). For each group, GRPO computes the mean and standard deviation of the reward, then writes each lane's mean z-score:

relative_advantage = mean over the lane's traces of (reward - group_mean) / group_std

Rows upsert on (agent_version_id, task_class). A lane that consistently scores above its task-class peers gets relative_advantage > 0 — the exact predicate routing gates on.

This is group-relative because it compares lanes within a task class rather than against an absolute threshold. A task class where everything scores 0.5 still has a winner; a task class where everything scores 0.9 still has a loser.

Conductor writeback

Every conductor decision is recorded as pending when the topology is chosen. At end of run, writeback resolves rows whose epic has reached a terminal state:

Epic outcome outcome realized_score
done success 1.0
escalated or anything else failure 0.0

So the conductor's choices accumulate their own track record, independent of lane-level routing.

Grounded rejections

When a reviewer or oracle refutes a claim, the claim and the evidence that refuted it are stored. This is the anti-reward channel: it is what stops a plausible but wrong approach from being reinforced simply because it appeared often. The table and schema are part of the learning store; wiring every reviewer and oracle gate to emit rows is incremental work, and the channel is most valuable exactly where those gates are strongest.

Read half: learning_governed

The default routing policy is learning_governed (MO_ROUTING_POLICY). It computes a static-hybrid baseline first — cheap implementer, stronger reviewer — then tries to improve it from measured data with a strict AND-gate:

flowchart TD
    A[node type + task_class] --> B{prompt_win_rates row<br/>sample_size >= min_samples?}
    B -- no --> F[static-hybrid fallback lane]
    B -- yes --> C{agent_performance_memory row<br/>relative_advantage > 0<br/>runs_count >= min_samples?}
    C -- no --> F
    C -- yes --> W[learned winning lane<br/>override]

Both conditions must hold:

  1. a prompt_win_rates row exists for the task class with sample_size >= MO_LEARNING_MIN_SAMPLES (default 3), and
  2. an agent_performance_memory row with relative_advantage > 0 and runs_count >= min_samples.

If either misses, the recipe's configured lane is used. A cold store and a vacuous store are treated the same: no override.

The other policies

MO_ROUTING_POLICY Behaviour
learning_governed Static baseline, overridden only on earned evidence (default)
workflow_default Use the recipe lane verbatim; no learning
frontier_only Always the strongest lane
cheap_only Always the cheapest lane
static_hybrid Cheap implementer, frontier reviewer
trace_governed Escalate after a prefix failure

Routes can also be registered at runtime via mini_ork.dispatch.routing.register_policy(name, fn) — see Extension points.

Defaults

The loop is on for every real run; each gate has an opt-out:

Variable Default Effect Opt-out
MO_PRM_SCORE 1 Score every trace =0
MO_ROUTING_POLICY learning_governed Read learnings to route workflow_default
MO_LEARNING_WRITEBACK 1 GRPO + conductor writeback =0
MO_LEARNING_MIN_SAMPLES 3 Evidence floor before an override Raise to demand more

Run-scoped lane overrides still win over learned routing: an explicit lane on a node is a decision, not a prior.

Validating the loop

Two read-only checks exercise the loop without dispatching model calls:

bash scripts/learning-loop-closure-gate.sh   # source defaults + live schema
bash scripts/smoke-learning-loops.sh         # end-to-end on a throwaway DB copy

The closure gate proves two classes of evidence, because neither alone is sufficient:

  • [CODE] — the policy, PRM scoring and writeback fire by default, not by opt-in.
  • [LIVE] — the database actually has the schema and rows to store and read an outcome.

A loop is closed only when both are green. Otherwise you have machinery that could learn but is switched off, or machinery switched on against a database that cannot record the result.

Reading the store

sqlite3 .mini-ork/state.db \
  "SELECT agent_version_id, task_class, relative_advantage, runs_count
     FROM agent_performance_memory
    ORDER BY relative_advantage DESC LIMIT 10;"

sqlite3 .mini-ork/state.db \
  "SELECT task_class, prompt_version, wins, losses, ties, sample_size
     FROM prompt_win_rates ORDER BY sample_size DESC LIMIT 10;"

sqlite3 .mini-ork/state.db \
  "SELECT node_type, agent_version_id, process_reward, status
     FROM execution_traces ORDER BY created_at DESC LIMIT 20;"

What you should see as the store warms up

  • execution_traces fills first — one row per dispatched node, each with a process_reward.
  • prompt_win_rates fills as reflect and conductor comparisons land.
  • agent_performance_memory grows more slowly, because GRPO needs several runs per (node_type, task_class) group before the advantage is meaningful. An empty or near-empty table on a young install is expected, not a bug.
  • Historical traces written before lane stamping cannot be re-attributed; the advantage table fills forward only.

Honest limits

  • Correlation, not causation. A lane that wins on a task class may be benefiting from easier tasks routed to it. The AND-gate reduces movement on noise, but it cannot identify a confound.
  • The sample floor is small. Three samples is enough to stop obvious thrashing, not enough to be statistically confident. Raise MO_LEARNING_MIN_SAMPLES when routing decisions are expensive to get wrong.
  • Learned routing is not yet equal correctness. On held-out work, learned routing has spent materially less than a fixed strong lane while solving fewer tasks. The loop's promise is lower cost over time; cost parity at equal correctness is not established. See Results.