← Back

False Green

I built six checks and wired them into CI. Later I found that each one reported success for work that never ran. Each one exited green.

The checks, the fixes, and the decision records are in this site’s own repository. It is public: github.com/sanlee-ys/portfolio.

Plate 01 Six PASS marks. The shared fact is that they passed. SIX CHECKS PASS PASS PASS PASS PASS PASS

Six greens. Each check reported PASS for work that never ran.

Two ways to fail

The first agentic review lane I put into CI broke three times before it worked.

Strip 02
One agentic review lane broke three times before it worked. The first break failed in minutes. The second break and the third break both reported green. The strip holds the outcome and nothing else. The second break’s cost, its refused posts, and the recorded denial count stay in the prose below.

Call that second break run A. Run A spent $0.14 across seven turns. The agent read the diff, formed a review, and attempted to post it six times. Every attempt was refused. The agent exited zero. The result message carried the array permission_denials with six entries. A tool denial is not a job failure, so the job did not fail.

Plate 03 DETERMINISTIC TOOL run tool nothing ran exit != 0 FAIL AGENTIC / HAND GATE run check nothing ran exit 0 PASS

Same empty middle, opposite report. A deterministic tool fails when it does nothing. An agentic or hand-written gate can still report PASS, whether the work happened, was denied, crashed, or was never attempted.

I wrote that down as a lesson about agents. The same shape already sat in gates that I wrote by hand. It took another four months to find them.

Contents

The check that measured the wrong thing

A doubled rule marks a decision that was reversed.

Reversed. The gate that tested an invisible page

Mobile overflow, measured on unstyled pages

The mobile QA gate renders every page at four widths and fails on horizontal overflow. It opened the built files directly, over file: URLs. The layout emits root-absolute asset paths, and those paths do not resolve over file:. So every page it measured rendered unstyled.

The result: an unstyled page does not overflow. The gate ran and produced numbers for sixteen pages at four widths. The numbers described a page no visitor would ever see. It now serves the build over HTTP.

Reversed. The proof that proved something else

“Fonts embedded: Geist”

The résumé PDF script printed that line and I believed it for months. It read the value from document.fonts. That value proves a woff2 loaded into the browser. It says nothing about the bytes written to the PDF.

The result: it read as evidence. It cost me a round of investigation into a font defect that did not exist. It now asserts against the raw bytes of the file it just wrote.

Both produced output. A crash is obvious. These numbers looked like evidence.

Contents

The check that could not see what it was looking for

Reversed. Every deep link, unchecked

The link checker split the fragment off first

One line: split the URL on #, split on ?, then ask whether the file exists. So a link to a real page with a dead anchor was a good link. The reader lands at the top of the page with no sign that anything went wrong.

The result: the checker did this from the day I wrote it. So it reported every deep link on the site as valid, and it never checked one anchor. It mattered the moment I moved sections between pages. That was the three pull requests immediately before I found it.

Reversed. A face the gate cannot see

The font gate’s regular expression

The coverage gate matches font declarations with @font-face\s*{([^}]*)}. A comment between @font-face and its opening brace hides the block from \s*. A closing brace inside a comment truncates the body early.

The result: either way the face falls out of the scan through a silent continue. The expected face count then treats that gap as a pass. Against the fixture I wrote afterward, the old pattern sees one face out of three.

Contents

What each one was actually measuring

The check Appeared to measure Actually measured
Mobile overflow Every page at four widths Unstyled pages, which cannot overflow
Résumé fonts Fonts embedded in the PDF Fonts loaded into the browser
Link checker Every link resolves Every link’s page exists
Font coverage Every declared face The faces its pattern could see
Review lane A code review happened The job exited zero
Denial counter Tool denials in the run A key that does not exist

Contents

The check watching the check

Reversed. The instrument that read an absent field

A counter that was structurally always zero

I built the review workflow to warn me about exactly the failure that started this page. If a run hit tool denials, the classify step would report it. Then nobody could ever again read a green check with no comment as a clean review.

Plate 04 Two key reads against one result message. One returns an array, one returns zero. ONE RESULT MESSAGE the saved execution log permission_denials an array permission_denials_count zero
The counter read one key and the message carried another. The array permission_denials was there. The key permission_denials_count was not, so the read returned undefined and the code defaulted it to zero. The plate shows one defect, not the way two defects concealed each other. The run’s logged denial count, its turn count, and its cost stay in the prose below.

The log is the raw message stream. Its result message carries the array permission_denials. My counter read permission_denials_count, which is not a key on that message. The action derives that count for its own stdout summary only. A read of an absent key returns undefined, and the code defaulted that to zero.

Run B is a later run, on this site’s own review lane. Its classify step logged denials=0. It reported turns as 8 and cost as 0.1447. Both figures matched the run summary to the digit. The same result object, and one field silently misread. The turn, cost, and denial figures on this page belong to two different runs.

The result: in run B the review agent held four tools, and none of them could open a file. The diff’s meaning lived in the markup. The agent burned five denied reads and posted nothing. The instrument I built to report those denials read a field that was never there. The two defects concealed each other.

The lane’s earlier configuration was worse. For ten days it had no publish channel at all. It posted zero comments on zero pull requests. That looked like a quiet stretch with nothing to say.

Contents

The question

Each defect is one line. A passing check and a check that cannot fail look the same from the outside. I found each one. I asked a specific question of a specific gate.

What is this green measuring → and can it tell “passed” from “never ran”?

The cheap version reconciles the gate’s own reported count against a raw grep, once. Every gate on this site now asserts a non-zero count of the thing it walked. A gate pointed at an empty directory finds nothing and reports that as success.

One of them did not, and this page said so first. The metrics gate skipped its parity check on Markdown. So a Markdown marker that parsed to nothing looked exactly like a file that makes no claims. A wrong mark on a number was quieter than no mark at all. This page named that gap while it was still live.

It is closed now. The Markdown branch counts its own markers. A suite asserts that the counter reports a gap for each shape the value pattern cannot read. A parity counter that returns zero and a correct parity counter produce identical output on a clean tree. So the fix was unfalsifiable until something made it fail on purpose. If I revert the one-line exemption, exactly three of those tests turn red. The suite is scripts/check-published-metrics.test.cjs.