Faithfulness Judge
Measuring the measurer: can an LLM be trusted to tell you when an answer is making things up?
Every LLM system I build is graded by another LLM at some point. That raises a question the grading quietly depends on and rarely answers: how good is the grader? This project measures that directly. It takes answers generated over public defense text, breaks them into individual claims, has a person label each one against its cited source, and then asks two model tiers to do the same job blind. The deliverable is not a working product. It is a number for how far the judge can be trusted, and an explicit statement of where it can’t.
The result
| Judge | Binary κ | Raw agreement (95% CI) | Unsupported recall |
|---|---|---|---|
| Opus | 0.751 | 89.4% [84.2, 93.0] | 97.9% |
| Sonnet | 0.716 | 88.4% [83.0, 92.2] | 89.6% |
Both tiers land in what the statistics literature calls substantial agreement, which makes either usable as an automated faithfulness check. The tempting headline is that the premium tier wins. I didn’t write that headline, because the confidence intervals overlap heavily at this sample size, so the gap in κ isn’t resolvable from this data. Reporting it as a win would be a soft number dressed up as a solid one.
The tempting second headline is the fabrication class, catching claims the source doesn’t back, where recall is 97.9% versus 89.6%. It doesn’t survive either. Those percentages are 47 versus 43 catches out of 48, and only the claims where the judges disagree carry information. There are four, all in the premium judge’s favor. McNemar’s exact test on those pairs gives p = 0.125: the direction is consistent, with no reversals in 48 chances, but four pairs can’t size an effect. An earlier version of this page reported that gap as the reason to pay for the premium tier; the test is the correction.
So neither axis separates the tiers on this set. The cheap tier is good enough and escalation isn’t evidenced. Showing the recall gap, if it’s real, needs a larger fabrication class than 48.
The result I almost published instead
A truncation setting that looked like a finding
The first scoring run showed Opus at 0.70 and Sonnet at 0.43. A dramatic, clean tier gap, and a far more satisfying story than the one above: the expensive model is simply better at judging. It was one commit from shipping.
It was false. I had set max_tokens=10 to force a one-word
verdict out of the judge. Opus complied. Sonnet tended to open with a
short clause of reasoning first, so on 39 of the 191 claims
then scored,
a full 20%, it was cut off before it ever reached the verdict word. My
scoring counted every unparsed verdict as a disagreement, so a fifth of
Sonnet’s answers were scored as wrong when they were simply
missing. On the 152 claims it actually finished, Sonnet was already at
about 0.70, which is where it sits today.
What caught it was reading the misjudgment log before publishing rather than after. Sonnet’s failures didn’t look like judgment errors, which are scattered and arguable. They looked empty. Errors have a shape, and that shape was wrong.
The lesson: when a model scores unexpectedly badly, inspect the raw outputs before believing the metric. A number can be perfectly reproducible and still be measuring your own harness. The generalization I actually changed my practice on: constrain a critical output structurally, not with a token budget.
The fix took two tries, and the failed one is worth recording. Prefilling
the assistant’s reply, seeding the response so the model continues
from a fixed token, is the standard trick for forcing a format, and these
models reject it. What works is a forced tool call: a
record_verdict tool whose schema constrains the verdict to
the label set, with the tool choice pinned so the model has to use it. The
model can reason as long as it likes, and the verdict arrives in a
structured block that cannot be truncated into ambiguity. That is the same
pattern the classifier uses to guarantee valid labels, applied to a second
problem.
One scoring convention did the real work here, and it was tempting to change it. Counting unparsed verdicts as disagreements is harsh; dropping them would have been defensible and would have quietly discarded 20% of the set, hiding the truncation entirely and shipping the false gap. The strict rule is what made the bug visible. Full decision record in ADR-001.
Why this is a different problem from the classifier
The classifier is graded against cheap, objective labels: a story is about procurement or it isn’t, and two careful people will mostly agree. This project grades against expensive, subjective ones. Deciding whether a specific sentence is supported by a specific cited passage takes real reading, the boundaries are genuinely arguable, and a hedged half-claim is a judgment call rather than a lookup. It reuses the classifier’s machinery, the judge harness, the Wilson intervals, the gold-set discipline, and points it at the opposite ground-truth regime.
It also produced the third verdict in a row on the same question. The classifier retired BM25 grounding after measuring it, then declined tiered model routing after measuring that. Here the cheap tier is again close enough that escalation is hard to justify on overall agreement. Three independent measurements, all landing on escalation barely pays, is no longer a one-off result. It is the reason the default across this system is the cheaper model until an eval says otherwise.
What this number does not establish
The limits are on the repo’s front page rather than buried at the bottom, because a floor is only useful if you state where it gives out.
- One labeler. The gold set is my labels alone, with no inter-annotator agreement measured. That makes the ground truth one person’s consistent reading of a written rubric, not a validated consensus. A judge agreeing with this gold at 0.751 has not been shown to agree with humans in general at 0.751, and adding a second labeler is the single thing that would upgrade the claim.
- The gold set has been corrected once. Two claims labeled supported were filler with no factual assertion, an offer to help and a suggestion to check the source, which the rubric excludes. Re-labeling and re-scoring moved the numbers on this page slightly. Single-labeler ground truth is worth auditing, not just declaring.
- 189 scored claims out of 193, with four excluded as non-claims. The intervals are about nine points wide. Any difference smaller than that is noise, and that includes both the tier gap in κ and the fabrication-recall gap.
- Floor tier by design. One pass, no judge prompt tuning, no ensembles, no retrieval. This measures the ruler as built, not the best ruler obtainable.
- Short passages from one domain. Public defense reporting, skewed toward operations and procurement. Faithfulness judging over long documents or technical specifications is untested here.
- A floor I’ll say out loud: below about 0.6 I wouldn’t trust an automated verdict unreviewed. Both judges clear that. Neither clears the bar for treating a verdict as final without a human in the loop on anything consequential.
Stack
Python, the Anthropic SDK (Sonnet and Opus as the two judges under test), forced tool-use for the verdict schema, Cohen’s κ and Wilson intervals computed directly, pytest on mocked calls. Source text is public DVIDS reporting; nothing proprietary or non-public is used anywhere in the project. The repo is public: github.com/sanlee-ys/faithfulness-judge — the gold set, the labeling guide, the misjudgment log, and the ADR that documents the bug are all checkable.
What it demonstrates
Anyone can run an eval and report the number it prints. The transferable skill is knowing when the number is lying to you, and being willing to publish the boring true result after you’ve seen the exciting false one.