Skip to content

Testing

The suite is pure Python and runs with plain pytest. This page covers the layout, the conventions that keep tests isolated, how CI shards the suite, and the failure modes that have cost the most time in practice.

Quick start

python3 -m pytest -q

pyproject.toml sets pythonpath = ["."] and testpaths = ["tests"], so the command works from the repository root without installation tricks. The bash test layers were removed in the 2026-07 bash-removal; everything here is Python.

Useful narrower runs:

python3 -m pytest tests/unit -q
python3 -m pytest tests/test_otel_export.py -v
python3 -m pytest -q -k "gates and not live"

Layout

Path Contents
tests/unit/ Unit tests for mini_ork/* modules
tests/ (root) Integration-style suites: dispatch, recovery, web smoke, run artifacts
tests/integration/ Cross-component integration tests
tests/e2e/ End-to-end flows
tests/optimize/ GEPA and optimizer tests
tests/live/ Live-provider tests, skipped without credentials

Unit tests should not shell out. Root and integration tests may drive a CLI — prefer python -m <module> subprocesses or an in-process main(argv) call over a shell pipeline.

Conventions that keep the suite trustworthy

Build the database, do not copy one

DB-backed tests bootstrap the schema in about a second:

from mini_ork.stores.migrate import init_db

def test_run_row_is_visible(tmp_path):
    db = tmp_path / "state.db"
    init_db(str(db), root=REPO)
    ...

Use a tmp_path database per test. A shared database makes failures depend on execution order, and cleanup bugs turn into cross-test coupling.

Never leak environment or cwd

conftest.py has an autouse fixture that snapshots and restores os.environ and the working directory. Do not defeat it. A past leak of .mini-ork/state.db into the suite cwd poisoned later tests, and the symptom appeared in unrelated modules.

If a test needs an environment variable, set it inside the test and let the fixture restore it:

def test_dry_run_env(monkeypatch, tmp_path):
    monkeypatch.setenv("MO_OTEL", "1")
    ...
    monkeypatch.delenv("MO_OTEL")

Prefer monkeypatch over manual os.environ writes. It restores on both pass and failure.

Do not use real credentials

Tests must not read a developer's real provider keys and must not write to a real secrets file. A test that clobbers the user's secrets is a bug even when it passes. Stub the dispatch layer, or skip when credentials are absent.

Registry fixtures must not delete real entries

A fixture that calls .pop() on a registry can remove a real backend registration and leave later tests with a missing provider. Save and restore:

@pytest.fixture
def temp_backend(monkeypatch):
    from mini_ork.dispatch import providers
    monkeypatch.setitem(providers.PROVIDER_KIND_BUILDERS, "test_kind", builder)

monkeypatch.setitem / setattr restore precisely; a bare pop does not.

Writing a test for a change

A useful shape for engine code:

  1. Arrange a tmp_path home and database.
  2. Call the function or module under test directly.
  3. Assert on structured output — returned status, rows written, files created — not on log text.
def test_resume_skips_succeeded_nodes(tmp_path):
    home = tmp_path / ".mini-ork"
    db = home / "state.db"
    init_db(str(db), root=REPO)
    # seed a run with one succeeded and one failed node
    rc = resume(str(db), run_id="r1", force=False)
    assert rc == 0
    assert ...  # succeeded node was not re-dispatched

For CLI behaviour, assert the exit code and a machine-readable line rather than a human sentence. Exit codes are part of the contract; wording is not.

For verifier or gate changes, assert all three outcomes (pass, fail, defer) and the aggregation into a verdict. A gate test that only checks fail will miss a change that turns unknown gates into passes.

CI

.github/workflows/ci.yml runs:

Job What it does
python-lint ruff check mini_ork/ tests/ (blocking: F + E9)
advisory lint ruff --select E,W,I,UP,B as a non-blocking ratchet
python-tests pytest on Python 3.11 and 3.12, sharded three ways
UI typecheck Type-checks the IDE surface
web smoke Exercises the local web/API surface

The three pytest shards are unit-a, unit-b and rest. Unit files are split by line parity rather than by hand, so a new unit file lands in a shard automatically:

# what unit-a runs
ls tests/unit/*.py | awk 'NR % 2 == 1' | xargs python -m pytest -q

# what unit-b runs
ls tests/unit/*.py | awk 'NR % 2 == 0' | xargs python -m pytest -q

# what rest runs
python -m pytest tests/ -q --ignore=tests/unit

A shard that dies at collection reports as a whole-shard failure, not as one failing test. Read the collection error before assuming an assertion broke.

Lint tiers

make lint             # blocking: F + E9 — must stay green
make lint-advisory    # E, W, I, UP, B — statistics only, non-blocking
make test             # the suite

The blocking tier is deliberately small: real errors (undefined names, syntax-level problems), not style. Advisory findings are a ratchet, not a gate; do not "fix" them with bulk rewrites that make the diff hard to review.

Traps worth knowing

Overlapping test paths and pytest-asyncio

Passing a file and a directory that contains it, or two globs that overlap, makes pytest collect the same module twice. With pytest-asyncio loaded this has produced a RecursionError during collection. Pass each path once:

python3 -m pytest tests/unit tests/test_cli.py -q     # not tests/ + tests/unit/ + globs

When a test run itself is the subject, disable the plugin explicitly:

python3 -m pytest -p no:asyncio tests/test_kickoff.py -q

Relative time windows

A test that asserts "created within the last hour" passes in the morning and fails after a pause, or fails on a machine with a skewed clock. Seed absolute timestamps relative to a captured now, not to the wall clock at assertion time. If a test can expire while sitting in a queue, it is a time bomb.

Pipelines hide the exit code

python -m pytest ... | tail returns tail's status, not pytest's. In CI and scripts, set pipefail or check ${PIPESTATUS[0]}. A green pipeline over a red test suite is worse than a red one.

Scope is not CI

A scoped green gate is evidence for the files it ran, not for the repository. Before a merge that touches shared modules, run the unit layer, not only the one test file you edited.

Flaky under load

The full suite spawns subprocesses and can time out when several worktrees run their gates concurrently. If the gate fails only under parallel load, re-run serially before concluding the change is broken.

Where to run a broad regression

Run broad suites in a worktree, not the primary checkout. The worktree has its own files and its own .mini-ork; a long run cannot disturb the tree you are editing. See Development worktrees.

Debugging a failure

  1. Re-run the single test with -x -vv to see the first failure.
  2. Check the collection phase — most "whole suite fails" reports are a duplicate-path or import error.
  3. Reproduce in a clean environment (env -i plus PATH) to catch a key your shell exports.
  4. Read the changed code's callers before changing the assertion.

Do not weaken an assertion to make a real regression green. If the behavior is intended to change, the test's name and its stated contract should change with it.

Next