Selected work
Four views of the work: how I structured the system, measured its decisions, tested a judge, and managed the program. Deeper investigations remain linked throughout the record.
- The SystemArchitecture, interfaces, and the integration loop
- ClassifierEvaluation, reversals, and evidence-based tradeoffs
- Faithfulness JudgeCan an LLM reliably catch a made-up claim?
- Product & ProgramThe work read through a TPM lens
The instrument
telltale
Five coding-agent CLIs, one brief, five columns — and gauges that render nothing when nothing was measured.
I ran a month-long evaluation of the AI coding agents that build this record. I wrote the test down before the evidence existed. The measurement changed my operating policy. telltale is the instrument that reads what five agent CLIs leave on disk.
v0.3.0 · 171 golden files · 2191 test functions
The system
I wired three services into one loop: an API, an LLM classifier, and a RAG agent.
Notes API BackgroundTask Classifier Tags Knowledge base KB Agent Grounded answer
The decision log
Three decisions, one row each. The full log is on Work. The decision log →
- Reversed. I added BM25 retrieval grounding, rebuilt the baseline so both arms ran one prompt, and cut it. It fixed 0 domain calls and broke 4 across 162 classifications. Classifier →
- Reversed. I built the premium-model upgrade, measured it, and declined it. It moved 0 rows at 1.97× the cost per article, so the default model stays Sonnet. Classifier →
- I rebuilt the eval on 54 hand-labeled real snippets, in place of 300 the model wrote itself. Domain accuracy fell 97.3% → 88.9%, and the lower number is the trustworthy one. Classifier →
Classifier evidence
Where the classifier stands now, on 54 hand-labeled real snippets.
- 94.4%Category
- 98.1%Domain
- 94.4%Region
I score category and region separately. The two figures are equal here by coincidence.
Classifier v3.2.1 on the 54-snippet human-labeled gold set. Full eval →
The classical baseline lost the case for an LLM. I vendored it here, and it runs in the browser on the same rows. Run it yourself →
Field notes on the techniques behind this work sit on the work index. The field notes →
About
For the short version of how I came to this work and how I approach it, read About →
I run this record in the open. A measurement changed what I shipped, and a reversed decision carries a doubled rule on the page that recorded it. A figure caption states what the figure does not show, and the build fails on an empty one. The operating layer keeps its incident log in the open, postmortem-first.
It is a solo build, so I simulate the program layer. The Product & Program page states what that means.
I set the direction and the bar. Claude does most of the typing. The evals and postmortems are the proof. The colophon states the method in full.
A second record sits off this system: a physical lab that makes a factory-blank router provision itself. Zero-touch provisioning →
Outside the work I hike, take pictures, and live with a Scottish Fold named Sango. The rest is on About and in the gallery.
