Why an LLM at All
The boring baseline, built so the foundational spend has a number under it instead of an assumption.
Everything above measures something layered on top of a language model: a sharper prompt, retrieval, a bigger model for the hard rows. None of it asks the question a skeptical reader asks in the first thirty seconds. Does this task need an LLM? For most of this project’s life there was no answer, which is a conspicuous hole in a repo whose whole argument is measure before you spend.
So I built the boring baseline: TF-IDF features into logistic regression, one classifier per axis, trained on 300 real snippets graded by the Opus judge (the scaled set described further down, not v1’s synthetic one) and scored once against the same 54 hand-labeled rows as everything else on this page. Deliberately standard, not tuned. A clever baseline invites “you rigged it,” and a hobbled one invites the same accusation in reverse.
| Against human gold, n=54 | Classical baseline | The LLM |
|---|---|---|
| Category accuracy | 72.2% | 92.6% |
| Domain accuracy | 66.7% | 92.6% |
| Cost per article | $0.00 | one API call |
| All 54 rows, wall-clock | 4.8 ms | network-bound, seconds |
The LLM wins by 20 points on category and 26 on domain. Both arms ran on identical rows, so that gap can be tested rather than eyeballed: a paired exact McNemar returns p=0.013 on category and p=0.0005 on domain. Counted as disagreements, the baseline alone was wrong 14 times to the LLM’s 3 on category, and 15 to 1 on domain. The foundational spend now has a number under it instead of an assumption (ADR-017, full report in baseline_eval.txt).
I built it expecting either answer to be worth publishing. A baseline that landed close would have been the strongest negative result here by a wide margin, and I’d have had to say so on this page. It didn’t, and the reason it didn’t is more useful than the margin.
Where the baseline actually broke
The misses aren’t spread evenly. The baseline collapses toward
whichever class is lexically loud: five of six policy
stories came back as operations, and six of eleven
land rows came back as air. Those are the
labels that depend on what kind of claim an article is making
rather than which words it contains. A treaty and a deployment share a
vocabulary; only one of them is policy.
What that means for the spend: the LLM isn’t buying accuracy in general, it’s buying the boundaries that need reading comprehension. That is a much more specific claim than “the LLM is better,” and it’s the one the numbers actually support.
Two handicaps I’d rather name than have found
The training labels came from the Opus judge, not from people, so the
baseline was learning to imitate a grader that itself disagrees with
humans about 5–6% of the time, and then it was tested against those
humans. Separately, industry had exactly
one training row, which makes that label structurally
unlearnable rather than merely hard. The baseline scored 0.000 on it,
as it had to.
Why they stay in the writeup: both cut against the baseline, which means both flatter the conclusion I was hoping for. That is the exact failure the grounding experiment above already taught me once, when an unfair frozen baseline inflated a lift that turned out not to exist. A margin this size survives the caveats; the caveats still get stated.
Run the loser
Everything above is a recorded number. This is the same fit, running in your tab: the model exported as data, about 208 KB over the wire, no server and no key. It is the arm that lost the bake-off, which is exactly why it can run here at all — and watching the wrong class light up is a faster route to the claim above than the confusion matrix that produced it.
A row from the gold set, or your own sentence
Pick an example, or type a sentence.
Baseline, live in this tab
category
—
operational_domain
—
region: not covered. The shipped classifier assigns a third axis. The baseline never had region labels to train on, so there is nothing to draw here — which is a truer answer than a bar you would have to know to ignore.
Recorded, not run
The model is vendored from
the
classifier repo at commit 7b61315 — the export
and its inference module travel as a pair, and the repo’s parity
gate asserts this JavaScript reproduces scikit-learn’s own
decision function to 1e-6. The gold and LLM columns are read out of
committed artifacts
(data/gold/gold.csv,
evals/gold_predictions_v3.csv).
No API is called from this page.