← Back to the classifier

The Autonomy Ladder

The same classifier, more self-direction one level at a time.

Each level is the same classifier with more self-direction. The axis is who drives. A reach for a tool does not move a level. A decision the system makes for itself does.

Plate 01 L1 shipped, L2 retired, L3 shipped, L4 declined. THE CLIMB L1 single call SHIPPED L2 augmented RETIRED L3 loop SHIPPED L4 multi-agent DECLINED

The measurements sit in the rungs below.

L1 a human runs it → L2 + a tool → L3 it decides it is done → L4 agents coordinate

  1. L1 Single call Shipped · v1

    Prompt, structured output, the eval harness.

    Who drives A human runs each item.

  2. L2 Augmented Shipped v2.0.0, then retired

    BM25 retrieval grounds the label in a corpus.

    Who drives A human runs it; the model reaches for a tool.

  3. L3 Autonomous loop Climbed · both rungs

    Wrapped in a loop that iterates to an explicit done-signal.

    Who drives The system decides when it’s done.

    • rung 1 — prompt-optimization loop shipped, rode the v2.1.0 tag
    • rung 2 — agent-driven ML loop built · its live run gamed its own metric, and got caught

  4. L4 Multi-agent Built, measured · declined

    Decomposed: triage, classify, and a critic that can hand work backward.

    Who drives Multiple agents coordinate.

    Triage Classify Critic hands work backward

L2 asks one question. Does something to lean on help the model? I asked it three ways.

Strip 02
One L2 question, asked three ways, returned three different verdicts. The neighbor documents fixed 0 domain calls and broke 4 (ADR-012). The few-shot exemplars scored 91.0% against a 90.0% baseline on an n=300 paired test, and p=0.70 (ADR-019). L3 below measures the keyword features. The strip holds no number; every number sits in the caption above and in the two linked records. It does not show the keyword-feature measurement. L3 below carries it.

L2 stays on the record as climbed, then retired.

L3’s two rungs each have a recorded run on the run replay. Rung 1 is the prompt-optimization loop. It gained on the set it tuned against, and it slipped on held-out gold. Rung 2 is an agent that does error-driven feature engineering on a trained baseline (ADR-018). It gained 6.0 macro-F1 points on the split that selected it. It lost 8.6 macro-F1 points on the human-labeled gold set that no decision read. The loop’s design gave the held-out set veto power, and it used it.

L4’s honesty test is the backward edge. I aimed the critic at the one named, rule-checkable error cluster. That cluster is seven of the 54 gold rows. In each row the model inferred a region from the US actor, and the text stated no place. The critic fixed six of the seven. The honesty test passed.

The pipeline still lost (ADR-020). The critic’s restraint lived only in its prompt. It challenged 4× more often than its charter allowed. Outside its target it broke more labels than it fixed. That is the repo’s first statistically significant harm: domain 91.3% → 86.7% on the n=300 paired test, p=0.016, at 4× the cost.

Both halves are the finding. The backward edge works. An agent whose scope rests on prose instead of code will spend that power freely.

ADR-006 records the decision to make the ladder the spine. The living roadmap holds the per-level detail. A reader can check what shipped against what I promised.