← Back to the classifier

Why an LLM at All

TF-IDF plus logistic regression. I scored it on the same gold as the LLM.

The classifier writeup measures upgrades on top of a language model: a sharper prompt, retrieval, a bigger model for the hard rows. It does not answer whether the task needs an LLM.

So I built the boring baseline. It puts TF-IDF features into logistic regression, one classifier per axis. I trained it on 300 real snippets that the Opus judge graded, which is the scaled set and not v1’s synthetic one. I scored it once against the same 54 hand-labeled rows. Standard, not tuned.

Plate 01 Decision fork for the classifier bake-off. One defense-news snippet goes to a classical TF-IDF logistic-regression baseline and to the LLM. Both face the same 54-row human gold set. INPUT snippet one row ARMS TF-IDF + LR classical LLM forced tool GOLD n=54 human same rows both arms · paired McNemar

One snippet, two arms, one human gold set. The table below is the score; the fork is why the comparison is fair.

Against human gold, n=54Classical baselineThe LLM
Category accuracy72.2%92.6%
Domain accuracy66.7%92.6%
Cost per article$0.00one API call
All 54 rows, wall-clock4.8 msnetwork-bound, seconds

The LLM wins by 20 points on category and 26 on domain. Both arms ran on identical rows, so a test can measure the gap. A paired exact McNemar returns p=0.013 on category and p=0.0005 on domain. I also counted the disagreements. The baseline alone was wrong 14 times to the LLM’s 3 on category, and 15 to 1 on domain (ADR-017, full report in baseline_eval.txt).

Where the baseline broke

The misses do not spread evenly. The baseline collapses toward whichever class is lexically loud.

Strip 02
  • five of six policy stories came back as operations
  • six of eleven land rows came back as air
Both collapses run in one direction. The baseline assigns the lexically louder class, and the quieter class loses its rows to it. ADR-017 and baseline_eval.txt record the bake-off, and the paragraph above links both. The strip shows two boundaries, not the whole confusion matrix. It does not show why these two boundaries are hard. The two sentences below carry that reason.

Those are the labels that depend on what kind of claim an article makes, not which words it contains. A treaty and a deployment share a vocabulary; only one of them is policy.

What that means for the spend: the LLM buys the boundaries that need reading comprehension.

Two handicaps

The training labels came from the Opus judge, not from people. So the baseline learned to imitate a grader that itself disagrees with humans about 5–6% of the time. Then I tested it against those humans. Separately, industry had exactly one training row, which makes that label structurally unlearnable. The baseline scored 0.000 on it, as it had to.

Why they stay: both cut against the baseline. The same class of unfairness inflated the BM25 lift until I rebuilt the baseline. A margin this size survives the caveats.

Run the loser

This is the same fit, and it runs in your tab. I exported the model as data, about 208 KB over the wire, with no server and no key. It is the arm that lost the bake-off. Here you watch the wrong class light up. That is faster than a read of the confusion matrix.

A row from the gold set, or your own sentence

Pick an example, or type a sentence.

Baseline, live in this tab

category

—

operational_domain

—

region: not covered. The shipped classifier assigns a third axis. The baseline never had region labels to train on, so there is nothing to show here.

Recorded, not run

Strip 03

The model is vendored from the classifier repo at commit 7b61315 — the export and its inference module travel as a pair, and the repo’s parity gate asserts this JavaScript reproduces scikit-learn’s own decision function to 1e-6. The gold and LLM columns are read out of committed artifacts (data/gold/gold.csv, evals/gold_predictions_v3.csv). No API is called from this page.