Faithfulness Judge
Can an LLM judge catch a made-up claim?
Another LLM grades every LLM system I build at some point. This project measures the grader. It takes 79 answers generated over public defense text and breaks them into individual claims. A person labels each claim against its cited source. Two model tiers then do the same job blind. The deliverable is a number for how far the judge can be trusted, and where it cannot.
One claim, one cited source, one binary label. The table below measures how often the model judge matches the human gold on that unit.
The result
| Judge | Binary κ | Raw agreement (95% CI) | Unsupported recall |
|---|---|---|---|
| Opus | 0.762 | 89.9% [84.8, 93.5] | 97.9% |
| Sonnet | 0.716 | 88.4% [83.0, 92.2] | 89.6% |
Both tiers sit in what the statistics literature calls substantial agreement. Either tier is usable as an automated faithfulness check. The premium tier edges the cheap one on κ. The confidence intervals overlap at this sample size. This data cannot resolve the gap.
I measured those figures against a gold set. I audited that gold set in August, blind, under a selection rule that I committed before I read a single label. 28 of the 30 claims it pulled came back unchanged. That consistency is the finding. The audit also moved Opus from 0.751 to 0.762.
Count only the corrections on claims that neither judge got wrong, and the Opus figure is 0.752, flat. The audit reached one claim that both judges failed. The removal of that claim lifts agreement arithmetically. I pre-registered the check, and the record publishes it both ways.
All four disagreeing pairs behind the gap are hedged claims, which the rubric itself calls the least crisp call a labeler makes. McNemar’s exact test on those four pairs gives p = 0.125. The direction is consistent, with no reversals in 48 chances. Four pairs cannot size an effect.
An earlier version of this page reported that gap as the reason to pay for the premium tier. That version called the gap recall on the fabrication class. The test and the restriction are the two corrections.
decisions/002-solid-tier-call.md at cee3310. The plate shows only the pairs where the two
tiers disagree. It leaves out every pair they agree on, because the
test does not read those. It also does not show the κ axis,
which the table above carries. So neither axis separates the tiers on this set. The cheap tier is good enough and escalation is not evidenced.
The result I almost published instead
A token cap cut the cheap tier before the verdict on 20% of the set. I counted each empty as a miss. The gap was my harness.
A truncation setting that looked like a finding
The first scoring run showed Opus at 0.70 and Sonnet at 0.43. That was a clean tier gap. One commit would have shipped it.
It was false. I set max_tokens=10 to force a one-word
verdict out of the judge. Opus complied. Sonnet tended to reason in a
short clause first. So the cap cut it off before it ever reached the
verdict word, on 39 of the 191 claims then scored, a
full 20%.
I counted every unparsed verdict as a disagreement. So my score marked a fifth of Sonnet’s answers wrong, and those answers were simply absent. On the 152 claims it actually finished, Sonnet was already at about 0.70. That is where it sits today.
I found it when I read the misjudgment log before I published. Sonnet’s misses did not look like judgment errors. A judgment error is scattered and arguable. They looked empty.
The lesson: inspect the raw outputs before you believe a surprising metric. A number can be reproducible and still measure the harness. Constrain a critical output structurally, not with a token budget.
The fix took two tries. A prefill of the assistant’s reply is the standard trick for a forced format. It seeds the response, so the model continues from a fixed token. These models reject it.
A forced tool call works. A record_verdict tool has a schema
that constrains the verdict to the label set. I pin the tool choice, so
the model has to use it. The model can reason as long as it likes. The
verdict arrives in a structured block that truncation cannot make
ambiguous. That is the same pattern the classifier uses to guarantee valid
labels.
The rule that counts an unparsed verdict as a disagreement is harsh. A rule that dropped those verdicts discards 20% of the set and hides the truncation. The strict rule made the bug visible. ADR-001 carries the full decision record.
What this grades against
I grade the classifier against cheap, objective labels. A story is about procurement or it is not, and two careful people will mostly agree. This project grades against expensive, subjective ones. A person must read the passage and the sentence closely to decide whether one supports the other. The boundaries are arguable, and a hedged half-claim is a judgment call.
This project reuses the classifier’s machinery: the judge harness, the Wilson intervals, and the gold-set discipline. It points that machinery at the opposite ground-truth regime.
The classifier measured BM25 grounding and retired it. It then measured tiered model routing and declined that. Here the cheap tier is again close enough that escalation is hard to justify on overall agreement. Three independent measurements reach the same verdict: escalation barely pays. That is why the default across this system is the cheaper model until an eval says otherwise.
What this number does not establish
The repo’s front page also states these limits:
- One labeler. The gold set is my labels alone, and I measured no inter-annotator agreement. So the ground truth is what one person consistently read out of a written rubric. It is not a validated consensus. Agreement with this gold at 0.762 does not show agreement with humans in general at 0.762. A second labeler is the single thing that would upgrade the claim. The August audit does not change this. The same person re-read his own labels, blind to the verdicts. That measures whether the gold is self-consistent, and it cannot measure whether the gold is right. A consistency audit is never inter-annotator agreement.
- I audited the gold set twice, and it moved both times. The first pass found two claims labeled supported that were filler with no factual assertion. The rubric excludes filler. The second pass was the blind one. It found a third instance that the first pass missed. It also reversed one of the first pass’s own calls. There the claim does assert that the source omits a detail, so it is a correct refusal and not filler. The second audit is also the evidence that the first was incomplete.
- The set holds 189 scored claims out of 193. I excluded four as non-claims. The intervals are about nine points wide. Any difference smaller than that is noise. That includes the tier gap in κ and the recall gap. The recall denominator is only 48.
- Each answer holds several claims, so the claims are not independent. These intervals are narrower than a clustered estimate. I did not compute one.
- Floor tier by design. The judge makes one pass. I tuned no judge prompt, built no ensemble, and added no retrieval. This measures the ruler as built, not the best ruler obtainable.
- Short passages from one domain. The source is public defense reporting, and it skews toward operations and procurement. This project does not measure a faithfulness judge over long documents or technical specifications.
- A floor: below about 0.6 I would not trust an automated verdict unreviewed. Both judges clear that floor. Neither judge clears the bar for a final verdict with no human in the loop on anything consequential.
Stack
The stack is Python and the Anthropic SDK, with Sonnet and Opus as the two judges under test. It uses forced tool-use for the verdict schema. It computes Cohen’s κ and the Wilson intervals directly. pytest runs on mocked calls.
The source text is public DVIDS reporting. The project uses nothing proprietary and nothing non-public. The repo is public: github.com/sanlee-ys/faithfulness-judge. The gold set, the labeling guide, the misjudgment log, and the ADR that records the bug are checkable.
What it demonstrates
The first scoring run printed a dramatic tier gap. The misjudgment log showed that a cap cut off the cheap tier. The published result is the one that survived that check: both judges are good enough, and escalation is not evidenced.