Defense News Classifier

Case study summary

• Problem: Classify public defense news across category, operational domain, and region without treating a convincing demo as proof of quality.
• Ownership: I built the classifier and evaluation harness, labeled the gold set, defined the merge gate, and made the ship-or-cut decisions.
• Constraints: The initial 300-row synthetic set graded model-written examples; the replacement human-labeled set had only 54 rows, so small apparent gains could be noise.
• Actions: I replaced synthetic evaluation with real text, tested a classical baseline, scaled a corroborating run to 300 examples, and measured retrieval, premium routing, and multi-agent review against held-out evidence.
• Result and evidence: The shipped single-call classifier scores 94.4% category, 98.1% domain, and 94.4% region accuracy on the current gold. The gate rejected three more complex approaches, preserving the simpler path.

The ceiling that wasn’t

A per-cell confusion breakdown showed the remaining misses were not scattered. They clustered on one boundary: technology read as operations whenever a new system was tested or demoed in an operational setting, plus a habit of over-assigning land. Two targeted rubric clauses moved category from 90.7% to 94.4% and domain from 90.7% to 92.6%, with technology recall reaching 1.000 and nothing regressing.

The lesson: the misses were a prompt problem. A stronger model did not move that boundary.

Reversed. The prompt that read better and performed worse

I sharpened the system prompt to draw a cleaner line between procurement (a purchase, the buyer's view) and industry (a company's own business news). Category accuracy fell from 79.0% to 76.7%, and industry recall dropped from 0.217 to 0.100. The sharper rule gave the model a cleaner excuse to dump borderline company stories into procurement. I reverted it.

The lesson: the prompt read cleaner. The numbers did not. Reverted.

Reversed. The grounding that did not pay

I added BM25 lexical retrieval to ground each classification in similar documents, expecting a clear win. Measured lift: +1.9% category accuracy (a net of one correct call in 54) and +0.0% on domain, where grounding fixed three calls and broke three. Lexical retrieval does not earn the cost of moving to embeddings here.

The comparison itself had a bug: I was scoring the new-prompt grounded arm against an old-prompt frozen baseline, which inflated the lift. Rebuilding the baseline so both arms ran the same prompt killed the advantage. A three-pass confirm settled it: across 162 grounded classifications, grounding fixed a domain call zero times and broke four.

So I retired it (ADR-012): out of the shipped path, out of the CI gate. The retrieval code, the corpus, and the measurement scripts stay in the repo, dormant and reproducible. The verdict: BM25 does not justify itself.

The lesson: the first measurement said skip embeddings. A fair baseline said cut BM25. The comparison only holds if exactly one thing changes.

Scaling the eval

A couple of lucky rows can look like progress on n=54. So I scaled to 300 real snippets graded by the Opus judge, which roughly halves each interval — category to 93.3%, domain to 90.3%. Those corroborate the hand-graded numbers.

95% Wilson intervals on an 80–100% axis. Wide bar: the n=54 human-graded gold the scaled run was checked against — category [84.9, 98.1], domain [82.5, 97.1]. Narrow bar: the n=300 judge-graded run — category [89.9, 95.6], domain [86.5, 93.2]. The point estimates barely move; the interval is what halves.

The caveat: the judge is the answer key here, so this measures agreement with a model and inherits its roughly 5–6% disagreement with humans. The source wire is also operations-heavy, which makes the category macro-F1 uninformative. The report says so.

Reversed. The upgrade that bought nothing

The roadmap called for tiered routing: escalate the uncertain cases to the more expensive model. I built the trigger, measured it, and shipped the negative result. Routing moved +0 rows on both axes at about 1.97× the cost per article. Of nine escalations on the gold set it fixed zero and broke one, and even sending everything to the premium model scored the same 94.4% category, so there was no headroom for any router to capture.

Why it shipped as a no: the earlier prompt fix had already taken the ground routing was aimed at. One more finding stayed in the record: adding a field to the schema flipped one benign snippet into a model refusal, so the measurement apparatus itself changed the behavior being measured. Full verdict in ADR-013. A separate project, measuring how reliable an LLM judge is, later landed on the same verdict. Three independent measurements now say escalation barely pays.

Decision · ADR-002

Force structured output with tool-use, not prompt parsing

The model is called with a forced tool schema (all three fields required, strict enums), so an invalid label is rejected at the API layer before it reaches application code. When v3 added region, it was one more required strict enum. The schema is where a new axis goes.

Why: the schema becomes the contract, and the app code stays free of defensive parsing. Tradeoff: tool-use costs a few extra tokens and a more verbose call signature than plain chat. Out-of-enum responses were rare (once in 300) and a single re-sample handled them.

Decision

One model call, not a multi-step pipeline

Classification is a single call, not a generate-reasoning-then-classify chain.

Why: a single call is a cleaner baseline for the eval. Tradeoff: gives up any lift a reasoning step might add, which the eval can revisit later.

Decision · ADR-004

Implement the metrics by hand

Precision, recall, F1, and confusion matrices are computed directly with pandas and arithmetic, no ML framework.

Why: hold off on ML tooling until it earns its place. It keeps the dependency surface to anthropic plus pandas. Tradeoff: about 40 lines that sklearn.metrics would replace, accepted for clarity.

Decision · SYS-005 (system-level)

In-process, idempotent writeback to the notes API

As a component of the larger system, the classifier is invoked by a FastAPI BackgroundTask in notes-api and writes labels back as namespaced tags with replace semantics, so reprocessing the same note converges instead of piling up duplicate tags.

Why: replace semantics make reprocessing safe by design, so idempotency is a requirement. Tradeoff: the background task carries real retry and error-handling logic. See The System for the full loop.