Glossary

Convention

ADR / SYS-NNN

I write one of these for every decision worth defending later: context, choice, tradeoff, alternatives considered. Numbered so I can cite it instead of re-arguing it.

Here: ADR-NNN are decisions scoped to one repo; SYS-NNN are the ones that cross repo boundaries — the contracts wiring notes-api, the classifier, and kb-agent together.

AI

Agent

Software that chains tool calls toward a goal instead of answering in one shot: pick a tool, read the result, decide the next step, repeat until done or stuck.

Here: kb-agent is mine — it searches the knowledge base and synthesizes a grounded answer instead of guessing from what the model already knows.

ARCH

BackgroundTask

A way to run a follow-up step right after a request without standing up a message queue: the caller gets a response immediately, and the extra work runs in the same process a moment later.

Here: notes-api uses FastAPI's BackgroundTask to call the classifier and write its labels back as tags. I first built this loop on Kafka, then right-sized it once the workload didn't earn a broker. Full decision →

EVAL

Baseline

The simplest credible alternative, measured first, so the fancier approach has a number to beat instead of a vibe.

Here: I built a TF-IDF + logistic-regression baseline and ran it against the LLM classifier on the same gold rows. The LLM won decisively — and the losing baseline now runs live in your browser on the demo page, so the comparison is inspectable rather than asserted. Full demo →

EVAL

BM25

A sparse keyword-retrieval method: it scores documents by how often and how distinctively they contain your query terms, no model required.

Here: I tested it as a lightweight grounding layer for the classifier. The measured lift (+1.9% category accuracy, a net of one call in 54, and +0.0% on domain) didn't justify the added complexity, so it didn't ship. Full decision →

AI

Classification

Sorting text into a fixed set of labels using a model, instead of hand-written rules.

Here: my classifier assigns two labels to each defense-news item — a category (procurement, operations, policy, technology, industry) and an operational domain (air, land, sea, cyber, space, multi). Full writeup →

SYSTEMS

Contract seam

The boundary where one service depends on another through a stable request/response shape. The point is to make integration explicit instead of relying on hidden assumptions.

Here: I froze three of these (SYS-004, SYS-005, SYS-006) and wrote tests on both sides of each. I described that as “contract tests in CI” and said a renamed field would fail the build. An audit of my own decision log in July 2026 showed that was wrong, and I’d rather correct it here than leave the tidier claim up.

What was actually true: each side asserted against its own copy of the shape. When the classifier added a third field, the provider’s test fixture was updated in the same commit, and the consumer’s test kept passing against a stub it wrote itself. Both suites stayed green through a breaking change. Two unit tests that happen to agree aren’t a contract test — a real one needs a single shared artifact both sides check against, so changing one copy fails the other’s build. Full decision →

EVAL

Eval / eval harness

A scored test set with known-correct answers, used to check whether a change actually helped — instead of trusting that it reads better.

Here: this is the discipline the whole site is built to demonstrate. Every prompt change, retrieval experiment, and model choice got measured against one, including the changes that failed and got reverted. Full decision →

EVAL

F1 / precision / recall

The numbers I use to score the classifier instead of trusting a skim. Precision: of everything I labeled X, how much was actually X. Recall: of everything that was actually X, how much I caught. F1: the balance of precision and recall, so neither one alone can make the model look better than it is.

Here: see the classifier evidence table for the real numbers, synthetic vs. hand-labeled. Full table →

EVAL

False green

A check that reports success without being able to fail — green whether the work happened or not. A check that cannot fail is indistinguishable from a check that passes.

Here: I audited my own repos and found six, including a mobile gate measuring unstyled pages and a counter logging zero denials off a key the log never carried. The diagnostic that catches them: what is this green measuring, and can it tell “passed” from “never ran”? Full writeup →

EVAL

Gold set

A hand-labeled evaluation set used as the reference answer key.

Here: mine is small (54 snippets) and pulled from real DoD news wire and SEC filings, not model-generated. Its job is to test label boundaries and eval discipline, not production-scale generalization. Full decision →

SYSTEMS

Idempotency

An operation that produces the same result whether it runs once or five times — safe to retry, no side effects pile up.

Here: it's why the classify-and-tag writeback didn't need Kafka's delivery guarantees. The tags I write are derived and idempotent, so re-running the step just re-arrives at the same state. Full decision →

EVAL

Inter-rater agreement (Cohen's κ)

How often two graders agree, after subtracting the agreement they’d reach by chance. Raw agreement flatters graders on unbalanced sets; κ doesn’t.

Here: it’s how I scored the faithfulness judge against my own labels — Opus at 0.751, Sonnet at 0.716, with overlapping intervals, so the data doesn’t crown either judge. Trusting a judge starts with measuring the judge. Full writeup →

EVAL

Judge model

A separate model used to grade another model's output against the gold set, instead of grading it by hand at scale.

Here: I cross-check the classifier's real-text eval with an Opus judge — useful for catching errors hand-review can't keep up with, as long as I still spot-check the judge itself. Full decision →

AI

RAG (retrieval-augmented generation)

Instead of asking a model to answer from memory, retrieve the relevant documents first, then hand only those to the model as grounding for its answer. Fixes the model's biggest weakness — it doesn't know your data — without retraining it.

Here: kb-agent is the RAG layer, retrieving from the knowledge base before it answers a query. Full system →

EVAL

Recall@k / MRR

Retrieval scores: recall@k asks whether the right document showed up in the top k; MRR asks how far down the list it sat. Together they say whether the retriever found it, and whether anyone would have seen it.

Here: kb-agent’s retrieval eval runs on these. Filtering boilerplate out of the index lifted recall@1 from 0.630 to 0.741; the same harness then scored the hybrid-retrieval experiment as a wash, so dense-only stayed the default. The harness gets to say no. Full writeup →

AI

Tool use / structured output

Forcing a model to respond through a defined schema — a function call with typed, required fields — instead of parsing free text and hoping the format holds.

Here: my classifier's tool schema makes an invalid label a rejected API call, not a runtime bug. Out-of-enum responses were rare: once in 300. Full decision →