Start here
A small AI system, built by directing Claude and run like a program: every change planned, built, reviewed, and iterated, with an eval gating the merge. If you have one minute, take one of these doors.
The measurement that lied to me: the Judge. The numbers, and how they were made honest: the Classifier. Six of my own checks reporting success for work that never ran: False Green. The program view: Product & Program.
The system
Three services wired into one loop: an API, an LLM classifier, and a RAG agent.
Notes API BackgroundTask Classifier Tags Knowledge base KB Agent Grounded answer
The decision log
Reversed. BM25 retiredEVAL · BM25
I added retrieval grounding, measured it, and cut it
I added BM25 lexical retrieval to ground each call, expecting a clear win. The first measurement said it barely helped. Then I found the measurement itself was unfair.
The result: I had been scoring the grounded arm against a stale baseline built on an older prompt. Rebuilt so both arms ran the same prompt, grounding fixed a domain call zero times and broke four across 162 classifications. It was retired outright, not left on as an option.
An unfair baseline flatters whatever you were hoping for, and comparing two things means changing exactly one of them. The retrieval code stays in the repo, dormant and reproducible, because the built-it-measured-it-cut-it trail is the artifact. Full eval →
Reversed. Model gated on evidenceSYS-002 · ADR-013
Gate model choice on evidence, not vibes
Sonnet by default across the system. Escalate to Opus only where an eval shows the quality gain pays for itself. I built the upgrade, measured it, and declined it.
The result: routing the uncertain cases to the premium model moved zero rows on both axes at about 1.97× the cost per article. Sending everything to the premium model scored identically, so there was no headroom for any router to capture.
A targeted prompt fix had already taken the ground routing was aimed at, so the cheapest fix won and that is now measured rather than asserted. The harness stays in the repo as the record. Full verdict →
Eval rebuiltEVAL · v1 → v3
I rebuilt the eval until it could tell me I was wrong
v1 graded the model on 300 snippets it generated itself, which measures consistency, not correctness. v2 replaced that with 54 hand-labeled real snippets, cross-checked by an Opus judge, and it has been catching me ever since.
The result: domain accuracy fell from 97.3% to 88.9% on real text, and that lower number is the trustworthy one.
Category accuracy actually rose, because real vocabulary separates more cleanly; domain fell because the synthetic set was trivially easy to grade. Full eval →
Eight more, in one line each. The full telling of every one is on the page it links to.
- Decoupled the repos with frozen HTTP contracts — then audited the seam and found it broken with both repos' CI green, because each side asserted against its own copy of the shape. Two unit tests that happen to agree are not a contract test. The System →
- Audited my own gates and found six of them reporting success for work that had not happened — a review lane that went green in 36 seconds having posted nothing, and the counter built to catch that reading a key that does not exist. False Green →
- Reversed. Built hybrid lexical retrieval to rescue one specific miss, measured it, and kept the dense-only default — MRR identical to four decimals, and the miss it was built for turned out not to be a lexical failure at all. Retrieval →
- Nearly published a dramatic premium-vs-cheap model gap that was my own bug — a
max_tokenscap truncated the cheap tier before its verdict on 20% of claims, and my scoring counted each as a disagreement. Judge → - Called it a ceiling, then went looking anyway — the misses clustered on one boundary, and two rubric clauses moved category 90.7% → 94.4% with nothing regressing. A prompt problem wearing a ceiling's clothes. Classifier →
- Reversed. A prompt that read better to a human and measured worse — category 79.0% → 76.7%, industry recall 0.217 → 0.100. Reverted. Classifier →
- Reversed. Built the enrichment loop on Kafka to learn it properly, then cut it — a single-user idempotent workload never needed a distributed log. Now an in-process background task. The System →
- Forced output to a tool schema instead of parsing prose, so an invalid label is rejected at the API layer — one out-of-enum response in 300, handled by a re-sample. Classifier →
Classifier evidence
Where the classifier stands now, on 54 hand-labeled real snippets.
- 92.6%Category
- 92.6%Domain
- 87.0%Region
Category and domain land on the same figure here. Two axes, scored separately, not a copied cell.
Those are the numbers after the eval got harder, not before. The full trajectory — synthetic self-graded, then real hand-labeled, a model migration, a prompt fix and a third axis, with the ceiling I called wrongly and the misses that clustered on one boundary — is the table on the classifier page. See the full eval →
The classical baseline that lost the case for using an LLM at all is vendored into this site and runs in your browser, on the same rows. Run it yourself →
Field notes
Notes I keep while learning, in plain language, on the techniques behind all of this. Read them →
- Structured output via tool-use
- Eval-driven development
- Reading the numbers
- Tool use, the general idea
- The agentic tool-use loop
- RAG: answering from your own docs
- Embeddings and vector stores
- Synthetic data and circular eval
- Tiered model routing
- Secrets and env
- Reproducible environments with uv
- Checkpoint, resume, and retries
A selection; the full, growing set is on the notes site.
About, honestly
For most of my career, what to build was decided above me. I was close to the work: running systems, shipping pieces of them, living with the seams. I had not owned a full-loop system, so I built one myself, in the open, to get that experience — decisions recorded, failed attempts included. It’s a small system, but I took the process seriously.
I did not get here alone. Most of what I know came from people who were patient when I was the one who did not know yet, and generous with hard-won expertise. I try to pass that forward where I can.
It is a solo build, so the program layer is simulated — I lay out what that means on the Product & Program page.
Outside the work: traveling, hiking, a bit of photography, and a Scottish Fold named Sango who supervises. The good frames live in the gallery.
