Defense News Classifier

Case study summary

• Result and evidence: The shipped single-call classifier scores 94.4% category, 98.1% domain, and 94.4% region accuracy on the current gold. The gate declined three more complex approaches, and the simpler path stayed.
• Problem: Classify public defense news across category, operational domain, and region. A convincing demo is not proof of quality.
• Ownership: I built the classifier and the eval harness. I labeled the gold set, defined the merge gate, and made the ship-or-cut decisions.
• Constraints: The initial 300-row synthetic set graded model-written examples. The replacement human-labeled set had only 54 rows. Small apparent gains could therefore be noise.
• Actions: I replaced the synthetic eval with real text and I tested a classical baseline. I scaled a corroborating run to 300 examples. I measured retrieval, premium routing, and multi-agent review against held-out evidence.

The ceiling that wasn’t

A per-cell confusion breakdown showed that the remaining misses were not scattered. They clustered on one boundary. The model read technology as operations whenever a new system was tested or demoed in an operational setting. It also over-assigned land. Two targeted rubric clauses moved category from 90.7% to 94.4% and domain from 90.7% to 92.6%. The 90.7% start is the post-rubric eval snapshot the classifier recorded on 2026-07-17. It sits one row above the table’s 88.9% category figure on the same 54 rows. technology recall reached 1.000. Domain ended at 92.6%, one row below the 94.4% column.

The lesson: the misses were a prompt problem. A stronger model did not move that boundary.

Reversed. The prompt that read better and performed worse

I sharpened the system prompt to separate procurement from industry. procurement is a purchase from the buyer's view, and industry is a company's own business news. Category accuracy fell from 79.0% to 76.7%, and industry recall dropped from 0.217 to 0.100. The sharper rule gave the model a cleaner excuse to move borderline company stories into procurement. I reverted it.

The lesson: the prompt read cleaner. The numbers did not. Reverted.

Reversed. The grounding that did not pay

I added BM25 lexical retrieval to ground each classification in similar documents. I expected a clear win. The measured lift was +1.9% category accuracy, a net of one correct call in 54. On domain the lift was +0.0%: grounding fixed three calls and broke three. Lexical retrieval does not earn the cost of a move to embeddings here.

The comparison itself had a bug. I scored the new-prompt grounded arm against an old-prompt frozen baseline, and that inflated the lift. I rebuilt the baseline so both arms ran the same prompt, and the advantage disappeared. A three-pass confirm settled it. Across 162 grounded classifications, grounding fixed a domain call zero times and broke four.

So I retired it (ADR-012). It came out of the shipped path and out of the CI gate. The retrieval code, the corpus, and the measurement scripts stay in the repo, dormant and reproducible. The verdict is that BM25 does not justify itself.

The lesson: the first measurement said skip embeddings. A fair baseline said cut BM25. The comparison only holds if exactly one thing changes.

The scaled eval

A couple of lucky rows can look like progress on n=54. So I scaled to 300 real snippets, and the Opus judge graded them. That roughly halves each interval: category to 93.3%, domain to 90.3%. Those corroborate the hand-graded numbers.

Plot 02

95% Wilson intervals on an 80–100% axis. Wide bar: the n=54 human-graded gold the scaled run was checked against. Category [84.9, 98.1], domain [82.5, 97.1]. Narrow bar: the n=300 judge-graded run. Category [89.9, 95.6], domain [86.5, 93.2]. The point estimates barely move, and the interval is what halves.

The caveat: the judge is the answer key here. This measures agreement with a model, and it inherits the judge's roughly 5–6% disagreement with humans. The source wire is also operations-heavy, which makes the category macro-F1 uninformative. The report states this.

Reversed. The upgrade that bought nothing

The roadmap called for tiered routing: escalate the uncertain cases to the more expensive model. I built the trigger, measured it, and shipped the negative result. Routing moved +0 rows on both axes at about 1.97× the cost per article. Of nine escalations on the gold set it fixed zero and broke one. I sent everything to the premium model, and it scored the same 94.4% category. So no router had any headroom.

Why it shipped as a no: the earlier prompt fix already took the ground routing was aimed at. One more finding stayed in the record: a new field in the schema flipped one benign snippet into a model refusal. The measurement apparatus changed the behavior it measured. The full verdict is in ADR-013. A separate project measured how reliable an LLM judge is, and it landed on the same verdict. Three independent measurements now state that escalation barely pays.

Decision · ADR-002

Force structured output with tool-use, not prompt parsing

The model is called with a forced tool schema (all three fields required, strict enums), so an invalid label is rejected at the API layer before it reaches application code. When v3 added region, it was one more required strict enum. The schema is where a new axis goes.

Why: the schema becomes the contract, and the app code stays free of defensive parsing. Tradeoff: tool-use costs a few extra tokens and a more verbose call signature than plain chat. Out-of-enum responses were rare (once in 300) and a single re-sample handled them.

Decision

One model call, not a multi-step pipeline

Classification is a single call, not a generate-reasoning-then-classify chain.

Why: a single call is a cleaner baseline for the eval. Tradeoff: gives up any lift a reasoning step might add, which the eval can revisit later.

Decision · ADR-004

Implement the metrics by hand

Precision, recall, F1, and confusion matrices are computed directly with pandas and arithmetic, no ML framework.

Why: hold off on ML tooling until it earns its place. It keeps the dependency surface to anthropic plus pandas. Tradeoff: about 40 lines that sklearn.metrics would replace, accepted for clarity.

Decision · SYS-005 (system-level)

In-process, idempotent writeback to the notes API

As a component of the larger system, the classifier is invoked by a FastAPI BackgroundTask in notes-api and writes labels back as namespaced tags with replace semantics, so reprocessing the same note converges instead of piling up duplicate tags.

Why: replace semantics make reprocessing safe by design, so idempotency is a requirement. Tradeoff: the background task carries real retry and error-handling logic. See The System for the full loop.