Product & Program
The same system, framed as a product and run as a program. Solo build. The program layer is simulated.
Case study summary
- Problem: Reduce the time a defense-news analyst spends finding and sorting developments before analysis can begin.
- Ownership: I acted as product owner and TPM for a solo build, turning delivery into a roadmap, dependency map, success measures, and risk register.
- Constraints: There was no team or external design partner, so cross-team coordination and user-outcome claims could not be represented as real.
- Actions: I defined the job to be done, sequenced Now / Next / Later work, tied program artifacts to shipped work, and used eval results to manage quality, cost, and scope risk.
- Result and evidence: The asynchronous enrichment loop, tool-layer contract, and classifier CI gate shipped. Classification is measured; analyst outcome metrics are explicitly defined but not yet validated in real use.
This is a solo build. There is no team to coordinate, so the program layer is simulated. The workstreams, the dependency map, and the risk register are real work, done as if a team existed.
The problem (product framing)
Defense developments are scattered across many sources and arrive faster than one person can triage. An analyst spends a disproportionate share of time finding and sorting before any real analysis begins.
Firehose in; brief out. Triage and a queryable knowledge base sit in the middle so analysis time is not spent sorting.
Who it is for
Primary user: a defense-news analyst who must stay current across programs, contracts, geopolitics, and technology, and brief others on what matters. Defense is a deliberately chosen vertical, not a claim of domain expertise. Design partner: me, dogfooding, as the first user and the early validation loop.
Job to be done
When a stream of defense news comes in, help me quickly know what matters and where it fits, so I spend my time analyzing instead of sorting.
Success metrics
Two kinds. They answer different questions.
User-outcome metrics (does it help the analyst?)
- Auto-triage rate: share of incoming items correctly categorized automatically, so the analyst never hand-sorts them.
- Coverage (recall): share of genuinely relevant developments surfaced rather than missed.
- Answer usefulness: for a query, the agent returns a correct, cited answer, fast.
These outcome metrics are defined, not yet measured. Measuring them honestly needs the analyst in the loop over real use, and this is a solo build. What is measured today is classification quality, below.
AI-quality metrics (can it be measured, not just demoed?)
- Classification quality: per-field precision, recall, F1. On real, human-labeled text the current v3 classifier scores category 94.4% (macro-F1 0.930), operational domain 98.1% (0.982), and region 94.4% (0.975). Corroborated at n=300 with confidence intervals half as wide as the 54-snippet set allows.
- Cost per unit of quality: escalating to the premium model was built and measured, not assumed. It moved zero rows at about 1.97× the cost, so the shipped path stays single-model. That is the second of three independent measurements landing on the same verdict, alongside the retired retrieval layer and the judge-reliability study.
- Retrieval quality (RAG): recall@k, answer groundedness, citation correctness.
- Regression gate: an eval pass-rate threshold wired into CI, so a change that drops below it does not merge.
Roadmap (program framing)
Sequenced Now / Next / Later, harvested from delivery.
Solid is shipped. Hairline is queued. Dashed is deferred, including the things I will not build.
Now
- Async enrichment loop: create, classify, idempotent writeback
- Integration tests across both seams (pytest, fully offline)
- Agent tool-layer contract (SYS-003), accepted and implemented
- Evals-as-CI: the classifier’s gold evals gate every PR
- Product one-pager and this program view
Next
- Evals-as-CI for kb-agent: extend the gate to retrieval
- A weekly status cadence from real progress
Later
- Containerize and deploy the services
- OpenTelemetry observability across services
- Other verticals: an explicit non-goal, articulated not built
Risk register
| Risk | Severity | Mitigation |
|---|---|---|
| Enrichment lost if the background task fails or the process restarts (no broker redelivery) | Low | Tags are derived and reclassification is idempotent (namespaced-tag replace), so enrichment re-runs safely; note creation never fails on classifier errors |
| Classifier accuracy ceiling (residual label ambiguity, not the model) | Medium | Do not escalate the model — measured at +0 rows for ~1.97× cost. Refine the rubric instead: a confusion breakdown found the misses clustered on one boundary and two clauses lifted category 90.7% → 94.4% |
| Breadth creep (verticals without depth) | Medium | Deep on one vehicle; other verticals are an explicit non-goal |
| Planning theater (docs drift from delivery) | Medium | Keep artifacts thin and living, tied to real delivery only |
| Silent contract drift (separate repos) | High — this one happened | Frozen wire contracts, but the enforcement was never real: each repo's tests assert against its own copy of the shape, so when the classifier added a field the coordinated consumer update was skipped and both builds stayed green. Open. The fix is one shared artifact both sides check against |
| Simulated program (no real cross-team coordination) | Low (honesty) | Stated plainly; the reasoning and artifacts are real, the org is not |
What it demonstrates
The scale is personal. The program is simulated. The source documents live in the architecture repo: the product one-pager and the program view.