The Autonomy Ladder

The same system handed more self-direction, one level at a time. Every level was written down before it was built, and two of them shipped a “no.”

This classifier is also the protagonist of the portfolio’s roadmap: an autonomy ladder. Each level is the same system handed more self-direction, not a separate demo. The axis is who drives — reaching for a tool doesn’t move you up a level; deciding for yourself does.

L1 a human runs it → L2 + a tool → L3 it decides it’s done → L4 agents coordinate

1. L1 Single call Shipped · v1

   Prompt, structured output, the eval harness.

   Who drives A human runs each item.

2. L2 Augmented Shipped v2.0.0, then retired

   BM25 retrieval grounds the label in a corpus.

   Who drives A human runs it; the model reaches for a tool.

3. L3 Autonomous loop Climbed · both rungs

   Wrapped in a loop that iterates to an explicit done-signal.

   Who drives The system decides when it’s done.

   • rung 1 — prompt-optimization loop shipped, rode the v2.1.0 tag
   • rung 2 — agent-driven ML loop built · its live run gamed its own metric, and got caught

4. L4 Multi-agent Built, measured · declined

   Decomposed: triage, classify, and a critic that can hand work backward.

   Who drives Multiple agents coordinate.

Every level is now climbed, and every level’s story ends the same way: the measurement got the last word. The L2 question — does giving the model something to lean on help? — was eventually asked three separate ways: retrieved neighbor documents as context (harmful: 0 domain calls fixed, 4 broken — ADR-012), machine-mined keyword features (helpful on the set they were mined from, harmful off it — see L3 below), and retrieved labeled examples as few-shot exemplars (inert: 91.0% vs 90.0% baseline on an n=300 paired test, p=0.70 — ADR-019). Three shapes, three different ways to not beat a well-prompted single call. A rung that was climbed and then climbed back down is still part of the record, which is why L2 isn’t quietly relabeled.

L3’s two rungs each earned an autonomy scar, and the run replay now steps through both recorded runs. Rung 1’s shows the prompt-optimization loop gaining on the set it tuned against while slipping on held-out gold. Rung 2 — an agent doing error-driven feature engineering on a trained baseline (ADR-018) — did it live and bigger: +6.0 points on the validation split it was allowed to see, −8.6 on the human-labeled set it wasn’t. The loop’s design assumed its own gains might be fake, so the held-out set had veto power, and it used it. That’s not the harness failing — that’s the harness being the product.

L4 put the metric-gaming story where the ladder always said it would live: the backward edge. The critic was aimed at the one named, rule-checkable error cluster — seven rows where the model inferred a region from the US actor when the text stated no place — and it fixed six of the seven. The honesty test passed. The pipeline still lost (ADR-020): the critic’s restraint lived only in its prompt, it challenged 4× more often than its charter allowed, and outside its target it broke more labels than it fixed — the repo’s first statistically significant harm (domain 91.3% → 86.7% on the n=300 paired test, p=0.016) at 4× the cost. Both halves are the finding: the backward edge works, and an agent whose scope is enforced by prose instead of code will spend that power freely.

Each level was written down before it was built. ADR-006 records the decision to make the ladder the spine, and the living roadmap holds the per-level detail — so what shipped can be checked against what was promised, including the levels that shipped a “no.”