← Back to the classifier

Loop replay

This page is a recorded-replay viewer for the autonomous loop’s run logs. It covers both rungs of the L3 step of the classifier’s autonomy ladder. The two rungs are the prompt-optimization loop and the agent-driven ML loop.

The loop optimizes against one set. Two other sets score the loop, and the loop never touches them. This page replays a run, so the chart shows the gap between “improved on A” and “improved on C”.

Plate 01 A drives the next edit re-scored every iteration B stops the run C reported only

Only A’s arrow comes back. B is held out of that feedback loop so stop-detection stays honest, and C — real gold text — never enters the agent’s context at all.

Real run claude-sonnet-5, 1,524,937 tokens, stopped on the threshold done-signal. Numbers below are a measured result, not a demo.

Run summary

Contents

Score progression

Category macro-F1 per split, per iteration. The shaded band is the gap between A and C at each iteration. The band shows a difference of levels at that iteration. The band does not show how much a split improved. The overfitting gap above the chart reports the change from the baseline.

A · training (optimized on) B · held-back (stops the loop) C · gold held-out (the honest number)
Plot 02

Contents

Iteration replay

Iteration 0 / 0

Agent rationale

Edit summary

Prompt diff

 
 
 
 
Contents
 
The decisions
 
 
Decision
 
Replay iteration-by-iteration, not just a summary chart
 
A single before/after number hides when A and C
        diverge, and which prompt edit caused it. This page steps through every
        iteration’s scores, diff, and rationale. The chart and the detail
        panel move together.
 
Why: the loop spec is about a
        trajectory, not an endpoint. A viewer that shows only the final delta
        cannot show a run that overfit early and then recovered. It cannot show
        a run that looked correct on B until C found it.
Tradeoff: this viewer needs more UI than a static
        chart. I kept it to plain HTML/CSS/JS with no chart library, to match
        the rest of the site’s stack.
 
 
 
Decision
 
Build against a labeled mock, then swap in the real run
 
This viewer shipped first against a --dry-run mock,
        because the viewer was the deliverable and a real run costs real money.
        The mock is a deterministic offline backend with zero API calls, and it
        prefixes every rationale field [dry-run]. A banner labeled
        the mock as a mock, and named it. The real run finished the next day,
        and the page now loads it: 1.5M tokens, threshold stop at
        iteration 2.
 
Why: nobody can review a viewer that
        needs real data before anyone can build it. A mock left in place after
        real data arrives is a stale claim. The single swappable data file made
        both statements true.
Tradeoff: the swap needed a CSS fix that the original
        plan missed. The “real run” state had a class but no style.
        The banner would therefore have kept its warning colour and claimed a
        measured result at the same time. The fix was cheap once I exercised
        the swap.
 
 
Contents
 
How this reads a run log
  
 Plate 03        prompt loop tags: type  ML loop tags: record     normalize at parse time neither log is edited   the viewer  
 Two run logs in different dialects reach one
          viewer. Rung 1 tags records with type and writes
          unified prompt diffs. Rung 2 tags with record and writes
          one-line experiment diffs. The plate does not show the log shape or the
          transport. The paragraph below keeps both, and it keeps the CORS
          reason. 
 
 

The run log is append-only
      JSONL: a run_metadata line, one iteration
line per step, and a trailing run_summary. This page ships
      the run log as a plain JS string
      (projects/data/loop-replay-real.js).
      It does not fetch a .jsonl file. The page renders the same
      way from disk and from GitHub Pages. A fetch() of a local
      file hits browser CORS restrictions that a plain
<script src> does not.

 

Rung 2’s run ships the same way in a second file
      (loop-replay-run2-real.js).
      The classifier repo gitignores its source log, so the embed here is the
      published record. Each run carries its own dry-run flag, so the page
      cannot show a mock without its warning banner.

 
To add another run:
 
     
  1. Drop the file in projects/data/.
    2.  
  2. Regenerate a template literal from its raw text.
    3.  
  3. Register it in loop-replay.js’s run list.
    4.  
 

This viewer does not cover L4’s multi-agent audit trail. That log is
      event-shaped: triage evidence, a challenge, a backward bounce, and a
      verdict, per row. It is not an iteration-by-iteration score curve. A
      replay through this player’s chart would flatten the structure the
      log exists to keep. Its measured result lives in the classifier repo
      (evals/l4_eval.txt,
ADR-020).
      If it gets a viewer, that viewer is a separate page.

 
Contents
 
Stack
 

The page uses vanilla HTML/CSS/JS, and it matches the rest of this site.
      It uses no chart library and no build step. The chart is a hand-built SVG,
      with the same createElementNS pattern as the
system diagram. The page parses the run log
      as plain JSONL, with JSON.parse per line.