← Back to the classifier

Loop replay

A recorded-replay viewer for the autonomous loop’s run logs — both rungs of the L3 step of the classifier’s autonomy ladder: the prompt-optimization loop and the agent-driven ML loop.

The loop optimizes against one set and is graded on two it never gets to touch. That separation is the whole point: this page replays a run so the gap between “improved on A” and “improved on C” is something you can see, not just read as a footnote.

A drives the next edit re-scored every iteration B stops the run C reported only

Only A’s arrow comes back. B is held out of that feedback loop so stop-detection stays honest, and C — real gold text — never enters the agent’s context at all.

Real run claude-sonnet-5, 1,524,937 tokens, stopped on the threshold done-signal. Numbers below are a measured result, not a demo.

Run summary

Score progression

Category macro-F1 per split, per iteration. The shaded band is the gap between A and C — the wider it opens, the more the loop improved on the set it was tuned against without improving on real gold text.

A · training (optimized on) B · held-back (stops the loop) C · gold held-out (the honest number)

Iteration replay

Iteration 0 / 0

Agent rationale

Edit summary

Prompt diff

 
 
 
 
The decisions
 
 
Decision
 
Replay iteration-by-iteration, not just a summary chart
 
A single before/after number can hide exactly the thing that matters
        here: when A and C diverge, and what the prompt edit was that
        caused it. So this page steps through every iteration’s scores,
        diff, and rationale, with the chart and the detail panel moving
        together.
 
Why: the loop spec’s honesty
        claim is about a trajectory, not an endpoint — a viewer that only
        shows the final delta can’t show a run that overfit early and
        recovered, or one that looked fine on B right up until C caught it.
Tradeoff: more UI than a static chart would need; kept
        to plain HTML/CSS/JS (no chart library) to match the rest of the
        site’s stack.
 
 
 
Decision
 
Built against a labeled mock, then swapped for the real run
 
This viewer shipped first against a --dry-run mock —
        a deterministic offline backend, zero API calls, every rationale field
        prefixed [dry-run] — because the viewer was the
        deliverable and a real run costs real money. The mock was labeled as a
        mock in a banner that named it, rather than quietly presented as a
        result.
        The real run landed the next day, and the page now loads it: 1.5M
        tokens, threshold stop at iteration 2.
 
Why: a viewer that can only be built
        once real data exists can’t be reviewed today — but a mock
        left in place after real data arrives is just a stale claim. The design
        that made both true is the single swappable data file.
Tradeoff: the swap took a CSS fix the original plan
        missed — the “real run” state had a class but no style,
        so the banner would have kept its warning colour while claiming a
        measured result. Cheap to fix, and a good argument for actually
        exercising a swap path before calling it done.
 
 
How this reads a run log
 

The run log is append-only
      JSONL: a run_metadata line, one iteration
line per step, and a trailing run_summary. This page ships
      the run log as a plain JS string
      (projects/data/loop-replay-real.js)
      rather than fetching a .jsonl file, so it renders the same
      way whether it’s opened straight from disk or served from GitHub
      Pages — a fetch() of a local file hits browser CORS
      restrictions that a plain <script src> doesn’t.
      Rung 2’s run ships the same way in a second file
      (loop-replay-run2-real.js)
      — its source log is gitignored in the classifier repo, so the embed
      here is the published record. The two rungs’ logs speak slightly
      different dialects (rung 1 tags records with type and ships
      unified prompt diffs; rung 2 tags with record and ships
      one-line experiment diffs), and the viewer normalizes both at parse time
      rather than editing either log — a recorded run is a record.
      Adding another run is: drop the file in projects/data/,
      regenerate a template literal from its raw text, and register it in
loop-replay.js’s run list. Each run carries its own
      dry-run flag so a mock can never be shown without its warning banner.

 

What is deliberately not here: L4’s multi-agent audit
      trail. That log is event-shaped — triage evidence, a challenge, a
      backward bounce, a verdict, per row — not an iteration-by-iteration
      score curve, and replaying it through this player’s chart would
      flatten exactly the structure that makes it interesting. Its measured
      result lives in the classifier repo
      (evals/l4_eval.txt,
ADR-020);
      if it earns a viewer, it earns its own.

 
Stack
 

Vanilla HTML/CSS/JS, matching the rest of this site — no chart
      library, no build step. The chart is a hand-built SVG (same
createElementNS pattern as the system
      diagram), and the run log is parsed as plain JSONL with
JSON.parse per line.