Product & Program
I frame the same system as a product, and I run it as a program. Solo build. The program layer is simulated.
Case study summary
- Result and evidence: The asynchronous enrichment loop, tool-layer contract, and classifier CI gate shipped. Classification is measured; analyst outcome metrics are explicitly defined but not yet validated in real use.
- Problem: Reduce the time a defense-news analyst spends finding and sorting developments before analysis can begin.
- Ownership: I acted as product owner and TPM for a solo build, turning delivery into a roadmap, dependency map, success measures, and risk register.
- Constraints: There was no team or external design partner, so cross-team coordination and user-outcome claims could not be represented as real.
- Actions: I defined the job to be done, sequenced Now / Next / Later work, tied program artifacts to shipped work, and used eval results to manage quality, cost, and scope risk.
This is a solo build. There is no team to coordinate, so the program layer is simulated. The dependency map and the risk register are real work. I did that work as if a team existed.
The problem (product framing)
Defense developments come from many sources. They arrive faster than one person can triage. An analyst spends a disproportionate share of the time to find them and to sort them. Real analysis begins after that.
Firehose in; brief out. Triage and a queryable knowledge base sit in the middle, so the analyst does not spend analysis time on the sort.
Who it is for
Primary user: a defense-news analyst. The analyst must stay current across programs, contracts, geopolitics, and technology, and must brief others on what matters. Defense is a deliberately chosen vertical, and not a claim of domain expertise. Design partner: me. I use the system myself, as the first user and the early validation loop.
Job to be done
When a stream of defense news comes in, help me quickly know what matters and where it fits, so I spend my time analyzing instead of sorting.
Success metrics
Two kinds. They answer different questions.
User-outcome metrics (does it help the analyst?)
- Auto-triage rate: share of incoming items correctly categorized automatically, so the analyst never hand-sorts them.
- Coverage (recall): the share of genuinely relevant developments the system finds rather than misses.
- Answer usefulness: for a query, the agent returns a correct, cited answer, fast.
I defined these outcome metrics. I did not measure them yet. An honest measurement needs the analyst in the loop over real use, and this is a solo build. What I measure today is classification quality, below.
AI-quality metrics (is there a measurement, or only a demo?)
- Classification quality: per-field precision, recall, F1. On real, human-labeled text the current v3 classifier scores category 94.4% (macro-F1 0.930). It scores operational domain 98.1% (0.982), and region 94.4% (0.975). An earlier two-axis run, category and domain only, scaled that eval to n=300. An Opus judge graded it against the two-axis gold: category 93.3%, domain 90.3%. That run corroborated the category and domain figures of its own version, and it halved the intervals of the 54-snippet set. It did not cover region, and it did not grade the v3 figures above.
- Cost per unit of quality: I built the premium-model escalation and I measured it. I did not assume it. It moved zero rows at about 1.97× the cost, so the shipped path stays single-model. It is the second of three independent measurements with the same verdict. The other two are the retired retrieval layer and the judge-reliability study.
- Retrieval quality (RAG): recall@k, answer groundedness, citation correctness.
- Regression gate: an eval pass-rate threshold wired into CI, so a change that drops below it does not merge.
Dependencies
kb-agent calls notes-api and the classifier as tools, so both services must work before the agent works as one system. SYS-003 sets that tool-layer contract, and SYS-004 and SYS-006 freeze the two wire seams. The dependency map in the architecture repo draws the graph.
Roadmap (program framing)
I sequenced the work Now / Next / Later. I harvested the sequence from delivery.
Solid is shipped. Hairline is queued. Dashed is deferred, and it includes the things I will not build.
Now
- Async enrichment loop: create, classify, idempotent writeback
- Durable SQLite outbox: a crash after 201 does not drop the job
- Integration tests across both seams (pytest, fully offline)
- Agent tool-layer contract (SYS-003), accepted and built
- Evals-as-CI: classifier gold evals and kb-agent retrieval both gate a PR
- Local compose and kind manifests, with /health on each service
- Weekly status cadence harvested from merged PRs
- Product one-pager and this program view
Next
- None. The 2026-09-09 wave closed the queued rows.
Later
- Operate-what-you-built beyond local compose and kind
- Other verticals: an explicit non-goal, articulated not built
Risk register
| Risk | Severity | Mitigation |
|---|---|---|
| Enrichment lost if the process restarts after create | Low | A SQLite outbox stores the job in the same commit as the note. Namespaced-tag replace keeps a re-run idempotent. Note creation never fails on classifier errors |
| Classifier accuracy ceiling (residual label ambiguity, not the model) | Medium | Do not escalate the model. I measured it at +0 rows for ~1.97× cost. Refine the rubric instead. A confusion breakdown found the misses clustered on one boundary, and two clauses lifted category 90.7% → 94.4%. The 90.7% start is the post-rubric eval snapshot of 2026-07-17, and the fix cost one domain row |
| Breadth creep (verticals without depth) | Medium | Deep on one vehicle; other verticals are an explicit non-goal |
| Planning theater (docs drift from delivery) | Medium | Keep the artifacts thin and current, and tie them to real delivery only |
| Silent contract drift (separate repos) | This is the one risk that happened. The callout below carries the severity, what happened, how I closed it, and what the fix still does not cover | |
| Simulated program (no real cross-team coordination) | Low (honesty) | I state it plainly. The reasoning and the artifacts are real. The org is not |
-
Risk
Silent contract drift (separate repos)
-
Severity
High. This one happened
-
What happened
The wire contracts were frozen, but the enforcement was never real. Each repo’s tests asserted against its own copy of the shape. The classifier added a field, and the coordinated consumer update never happened. Both builds stayed green.
-
Status
Closed on 2026-07-18, under SYS-018.
-
The fix
One shared artifact both sides check against. Each provider generates a schema from its live response model and commits it. Every consumer fetches that schema in its own CI and fails on divergence.
-
What it still misses
A consumer that cannot fetch the schema warns and exits 0, so a real provider outage reads as a pass. I took that trade so a GitHub outage cannot redden an unrelated build.
I pre-registered six axes for a capacity review, then measured that nothing wrote to them.
This record applies an evidence gate to a model choice. I applied the same gate to the build capacity behind the work. On a fixed date I wrote the review down. It named six evidence axes, a close date, and a rule that nothing changes until the close. Then I let the axes decide.
Two design choices did all the work. I wrote the axes before the evidence existed, and nothing changed until the close.
Those two choices are the whole method. I wrote the axes first, so nothing could pick its own success criterion after the fact. I froze every change until the close, so nothing could be cut on a bad week or kept on a good anecdote. The close came early, on a full month of evidence. One reason justifies an early close: the last stretch added no new evidence on any axis.
The results, by axis
This section reports by evidence axis. It never reports by lane and never by name. An axis is a question. A lane is a seat, and a seat maps to somebody’s product. A report by lane turns a one-operator measurement into a public claim about a third party. A measurement is not an opinion. A table around an opinion does not make it a measurement.
-
One axis produced repeated, measured value.
Independent review loops raised real findings. I confirmed and fixed most of them before merge, and the noise rate was nil. This is the one axis with demonstrated unique value.
-
The capacity axis produced the opposite of what I expected.
There were no friction events at all. The record goes the other way: dated directives in my own operating rules tell sessions to spend the headroom rather than to conserve it. Capacity was never the binding constraint.
-
The remaining axes produced no dated unique result.
The month gave them the chance to produce one.
The rulings, as principles
-
Capacity that was never consumed is unproven, not proven.
So the top tier steps down. The first month on the lower tier is itself the next measurement.
-
I declined a parked upgrade.
No recorded event showed that capacity ever blocked anything.
-
A seat is retained for demonstrated unique value.
Attachment to a tier is not evidence.
The limit on this reading, at the point of the claim: one month, one operator, one body of work, and no controlled comparison. It is a routing decision for me. It is not a verdict on anybody’s product.
The result: I pre-registered the six axes and nothing ever wrote to them. There was no friction log, no tally file, and no dead-rows list. So I had to reconstruct the evidence at the end out of dated records. The gate still returned a clear answer, because the answer was lopsided. A closer result would have been unrecoverable.
The lesson: pre-registration without an instrument is half a measurement. I built a gauge for this same capacity, and it does not write this log today. That is the open item. I name it here, because that is the only honest place to end.
telltale, the instrument for the same capacity →
What it demonstrates
The scale is personal. The program is simulated. The source documents live in the architecture repo: the product one-pager and the program view.