Retrieval, Measured
kb-agent never measured its own retrieval. Three paired A/B changes later, hybrid stayed off the default.
kb-agent is the RAG agent in the system, and it answers questions about my own projects out of a small knowledge base. I planned its second milestone as a shared retrieval backbone: the classifier would call it over HTTP instead of its own vector store. That consumer shipped its own lexical retriever, so the backbone had nobody left to serve. I retired the backbone, and that left an untested assumption: kb-agent’s retrieval was good enough to be another project’s substrate. Nothing measured it, and my own kickoff note is blunt about that. Every quality claim about the search tool was a vibe.
So the milestone turned inward: measure retrieval first, then decide whether it needs to change at all.
One query, a ranked list, an objective path match. The tables below measure how often the right source file is in the top k. No LLM judge runs at this layer.
The baseline, and the two suspicions it named
27 queries, hand-labeled, weighted to the corpus rather than split evenly: 8 about projects, 5 about libraries, 10 about my plain-language notes, and 4 adversarial. The corpus is lopsided, so an even split would misrepresent what retrieval actually searches. Each label is a query with the source file or files that should come back. The harness records recall at 1, 3 and 5, plus MRR. This page prints recall@1, recall@5 and MRR. I chose k=5 to match the search tool’s own default.
There is no LLM judge at this layer, and the label is an objective path match. A judge would re-score a question a person already answered, and it would add variance and a validation burden. Retrieval quality stays separate from answer quality. A RAG system that conflates the two hides bad retrieval behind a capable model. If I measure answer quality later, that layer can have a judge. It stays off this one.
The tool takes an optional kind filter. The gap between a run
with it and a run without it is the largest split on this page.
| Baseline setting | recall@1 | recall@5 | MRR |
|---|---|---|---|
| Unfiltered | 0.630 | 0.889 | 0.744 |
| With the kind filter | 0.852 | 1.000 | 0.920 |
The retriever is good when it knows the kind. Real usage sat on the top row, because the model mostly did not supply one.
The first run also turned two vague suspicions into named misses. I left repo boilerplate and generated report files indexed as notes on purpose, to measure whether they hurt retrieval. They sat in the top five of the misses. I also assumed the model usually passed the kind filter. It did not.
Three changes, and an A/B test for each one
Every change ran through one paired comparison layer. I vendored that layer from the classifier’s eval. I did not re-derive it. A comparison that quietly shrinks its own denominator still prints a confident number.
Each pair scores one outcome, and each change names its own outcome. Change 1 and change 3 score recall@5: was the right file in the top five. Change 2 scores the model’s first tool call instead, because it changes no retrieval.
Change 1PR #71
Filter the boilerplate out of the notes ingest
The notes sweep globbed whole repositories for Markdown. It indexed
README files, CLAUDE.md files and generated reports
alongside real content. Those files crowded the real content out of the
top results. The filter excludes them from the notes sweep only. It
records what it skipped, and it never drops a file in silence. The index went
from 47 source files to 44.
The result: recall@5 rose from 88.9% to 92.6%, recall@1 from 0.630 to 0.741, and MRR from 0.744 to 0.807. The kind-filtered arm went from 0.852 to 0.963 at recall@1. At recall@5 the pairs are one win, zero losses and 26 ties. The claim is one recovered miss and no regressions, not the percentage.
Change 2PR #74
Steer the model to pass the kind filter
This is prompt and tool-description text, not a retrieval change. It aims at the gap in the table above, and it needed a different instrument. That harness makes one real model call per gold query, with the real system prompt and the real tool list. It then grades the first tool call. Did it reach for the search tool with a kind, and was the kind right. No tool executes and no loop runs, so the measurement is model behaviour rather than retrieval.
The result: the rate at which the model supplies a kind went from 0.370 to 0.889, and the correct-kind rate from 0.370 to 0.852. This model rejects the temperature parameter as deprecated, so I cannot pin the sampling. I ran each arm three times and the figures are medians. On the supplied kind the pairs are 15 wins, zero losses and 12 ties. On the correct kind they are 14 wins, zero losses and 13 ties. Each pair takes the majority of the three runs, so these counts do not divide out of the medians. The baseline spread was 0.333 to 0.407 and the candidate 0.889 to 0.926. Those ranges do not overlap, so the median of three is worth quoting.
This does not show that answers got better. It shows that the agent now picks the stronger setting more often. The retrieval eval already measured that setting as the stronger one.
The one I built and did not ship
A doubled rule marks a decision that was reversed.
MRR identical to four decimals. I built it to rescue one miss, and that miss stands.
Reversed. Change 3PR #73 · ADR-010
Hybrid lexical and dense retrieval
The whole milestone pointed at this change. The hypothesis was specific: four of the 27 queries exist for this decision. They are two pairs, and each pair sets an exact-jargon phrase against a paraphrase of the same need. I placed them where lexical and dense retrieval ought to disagree. One of them was still a miss after change 1, and it was the paraphrase half. That is the textbook dense failure a lexical leg should rescue.
I built the lexical leg against the same chunk corpus already in the vector store. I fused it with the dense ranking by Reciprocal Rank Fusion at k=60. I left k at the canonical value and did not tune it. A tune of k on the same 27 queries that judge the outcome fits the test set. Ranks fuse and scores do not: a cosine distance and a BM25 score share no scale. Any weighted blend needs a constant fitted on that same data.
| Unfiltered | recall@1 | recall@5 | MRR |
|---|---|---|---|
| Dense only | 0.741 | 0.926 | 0.8068 |
| Hybrid | 0.704 | 0.963 | 0.8068 |
Recall@5 went from 92.6% to 96.3%. That is the number I would have led with, and it is one query. The MRR column carries four decimals, because the identity below is the claim that column has to support.
I built hybrid to fix one thing, and it does not fix it. That paraphrase query is still a miss under hybrid. A run of the lexical leg on its own explains why. The paraphrase’s surface terms point at the notes file just as hard as the embedding does.
Both retrievers prefer the same wrong file. So the query was never a lexical-versus-dense failure. It is corpus crowding: a topically adjacent notes file out-competes a project stub. No fusion of two retrievers that agree can repair it. The other surviving miss reads the same way. That falsifies the premise I built the experiment on.
The result: the default stayed dense only. MRR is identical to four decimal places across both arms, and recall@1 drops. Under the kind filter, hybrid is strictly worse on every metric that moved, and that filter is the setting change 2 steers the model toward. Against that, the lexical leg is a second retriever, a second dependency, an in-memory index and a cache-invalidation surface on every search. A default should not change on a wash.
Both paths stay in the tree behind one constant. One flag re-runs the losing arm from the same checkout.
What the numbers actually support
n = 27, so a single query is 3.7 percentage points and the significance line carries more weight than the recall deltas. Two of the three changes rest on one discordant pair, at McNemar p = 1.0000. That p-value does not mean hybrid is worse. It means the experiment did not clear the bar to change a default. It also does not prove change 1. I therefore write the claim as “regressed nothing” rather than as a percentage.
One query moved → at n=27 that is 3.7 points.
decisions/ADR-010-hybrid-bm25-retrieval-measured-and-not-defaulted.md at c90dd9c. The plate shows only the pairs where the two
arms disagree. It leaves out every tied pair, because the test does not
read those. It shows the hybrid comparison alone, and it does not show
the recall deltas or the MRR, which the tables above carry. The kind-steering result is the largest effect on the page and the least comparable to the other two. It is a model-behaviour measurement over three unpinned runs against a corpus-independent instrument. It is not a retrieval measurement. I report the two separately.
What this does not establish
- This is a per-KB measurement. 260 chunks across 44 files, one embedding model, one corpus. Nothing here is a claim that hybrid retrieval is useless in general. The claim is that on this knowledge base at this size it did not earn a default.
- Retrieval, not answers. recall@k and MRR say the right chunk was retrievable. They say nothing about whether the agent’s final answer was good. There is no answer-quality eval in this repo. That split is the design, not a gap.
- One labeler, and I drafted the labels before I reviewed them. I drafted the queries and the proposed labels against the actual stub contents. I then reviewed every entry before merge, and that review is the hand-label pass. One label is arguably wrong: a question about a packaging tool is labeled as a note, where the library slice is defensible. That label is the residual miss on the kind-correct axis. A change to it after the result would tune the answer key to the outcome, so it stands as labeled.
- The remaining misses point somewhere I did not go. Both are corpus crowding. That is a question about chunking and about what the stubs say, not about which retriever runs. That is the measurement that would actually move them, and it is not built.
- The knowledge base drifts. I took the first baseline two weeks earlier, and it no longer reproduced when the A/B ran. So I re-measured both arms of every comparison on the same day, against the same index. Figures from different weeks on this project are not comparable, and the ones on this page are same-day pairs.
Stack
Python, and ChromaDB with the local all-MiniLM-L6-v2
embedding model, so retrieval needs no API key and no network.
rank-bm25 runs the lexical leg. A hand-written recall@k and
MRR harness measures both arms. I vendored the paired-comparison layer
from the classifier’s eval. I did not rewrite it. The repo is public:
github.com/sanlee-ys/kb-agent.
Everything behind every figure above is checkable. That is the
gold set, both eval harnesses, the decision record for the negative
result, and the per-query rank tables. Read
the gold set
and
ADR-010.
The three pull requests
(#71,
#74,
#73)
each carry their own numbers.
What it demonstrates
This page shows the same discipline as the classifier, applied to a different problem. Build the measurement before the change, then let it decide. I designed the third change around a specific hypothesis, and I built it to completion. I then reported it as a result that falsifies its own premise. The miss I aimed the milestone at was corpus crowding, not a lexical-versus-dense gap.