False Green
Six checks I built, wired into CI, and later caught reporting success for work that had not happened. What they have in common is not that they broke. It is that they passed.
Two ways to fail
The first agentic review lane I put into CI broke three times before it worked.
The first break was a missing OIDC permission. It went red in minutes, which is what a broken thing is supposed to do.
The second break was a missing tool grant. It went green in thirty-six
seconds, having spent $0.14 across seven turns. The agent had read the
diff, formed a review, attempted to post it six times, been refused each
time, and exited zero. The run recorded
permission_denials_count: 6. A tool denial is not a job
failure, so the job did not fail.
The third break was a pull request that edited the lane’s own workflow file. Green. The job never executed at all.
The shape: a deterministic tool fails loudly when it does nothing. An agentic one succeeds quietly — the same green appears whether the work happened, was denied, crashed, or was never attempted. That was the lesson I wrote down, and I thought it was a lesson about agents.
It was not. The same shape was already sitting in gates I had written by hand, and it took me another four months to find them.
The check that measured the wrong thing
Reversed. The gate that tested an invisible page
Mobile overflow, measured on unstyled pages
The mobile QA gate renders every page at four widths and fails on
horizontal overflow. It opened the built files directly, over
file: URLs. The layout emits root-absolute asset paths, and
those do not resolve over file:, so every page it measured
rendered unstyled.
The result: an unstyled page does not overflow. The gate ran, produced numbers for sixteen pages at four widths, and the numbers described a page no visitor would ever see. It now serves the build over HTTP.
Reversed. The proof that proved something else
“Fonts embedded: Geist”
The résumé PDF script printed that line and I believed it
for months. It read the value from document.fonts, which
proves a woff2 loaded into the browser and says nothing about
the bytes written to the PDF.
The result: it read as evidence. It cost me a round of investigating a font defect that did not exist. It now asserts against the raw bytes of the file it just wrote.
Both of these are worse than a check that crashes, because both produced output. There is no moment where you look at a passing gate and wonder what it measured.
The check that could not see what it was looking for
Reversed. Every deep link, unverified
The link checker split the fragment off first
One line: split the URL on #, split on ?, then
ask whether the file exists. So a link to a real page with a dead anchor
was a good link, and the reader lands at the top of the page with no sign
anything went wrong.
The result: it had done this since the checker was written, so every deep link on the site had been reported valid without the anchor ever being confirmed to exist. It mattered the moment I started moving sections between pages, which is to say for the three pull requests immediately before I found it.
Reversed. A face the gate cannot see
The font gate’s regular expression
The coverage gate matches font declarations with
@font-face\s*{([^}]*)}. A comment sitting
between @font-face and its opening brace hides the block
from \s*. A closing brace inside a comment truncates the
body early.
The result: either way the face falls out of the scan
through a silent continue, and the count of declared faces
the gate expects is standing by to turn that gap into a quiet pass.
Against the fixture I wrote afterward, the old pattern sees one face out
of three.
What each one was actually measuring
| The check | Appeared to measure | Actually measured |
|---|---|---|
| Mobile overflow | Every page at four widths | Unstyled pages, which cannot overflow |
| Résumé fonts | Fonts embedded in the PDF | Fonts loaded into the browser |
| Link checker | Every link resolves | Every link’s page exists |
| Font coverage | Every declared face | The faces its pattern could see |
| Review lane | A code review happened | The job exited zero |
| Denial counter | Tool denials in the run | A key that does not exist |
The check watching the check
Reversed. The instrument that read a missing field
A counter that was structurally always zero
The review workflow was supposed to warn me about exactly the failure that started this page. If a run hit tool denials, the classify step would flag it, so a green check with no comment could never again be mistaken for a clean review.
It read permission_denials_count from the saved execution
log. That key does not exist. The log is the raw message stream, and its
result message carries the array permission_denials.
Reading an absent key returns undefined, and the code defaulted it to
zero.
On one pull request the step logged denials=0 while
reporting turns as 8 and cost as 0.1447, both
matching the run summary to the digit. The same result object, one field
silently misread.
The result: that review agent held four tools, none of which could open a file. On a diff whose meaning lived in the markup it burned five denied reads and posted nothing. The instrument that existed to surface those denials was reading a field that was never there. The two defects concealed each other.
The lane’s earlier configuration was worse in a way that is almost funny in hindsight: for ten days it had no publish channel at all, and posted zero comments on zero pull requests, which looked exactly like a quiet stretch with nothing to say.
The question
None of these were subtle. Each is one line, and each survived because a passing check and a check that cannot fail look identical from the outside. The only thing that ever found one was asking a specific question of a specific gate.
What is this green measuring → and can it tell “passed” from “never ran”?
The cheapest form of that question is to reconcile the guard’s own reported count against a raw grep, once. Every gate on this site now asserts a non-zero count of the thing it walked, because a gate pointed at an empty directory finds nothing and reports that as success.
One of them did not, and this page said so first. The metrics guard skipped its parity check on Markdown, so a Markdown marker that parsed to nothing was indistinguishable from a file making no claims — and marking a number wrong was therefore quieter than not marking it at all. It was named here while it was still live, deliberately, because a list of solved problems is a weaker claim than an open one.
It is closed now. The Markdown branch counts its own markers, and a suite asserts the counter reports a gap for each shape the value pattern cannot read. That second half is the part that matters: a parity counter that returns zero and a parity counter that is correct produce identical output on a clean tree, so the fix was unfalsifiable until something made it fail on purpose. Reverting the one-line exemption turns exactly three of those tests red.