May 16, 2026

Seven days our CLI silently lied to 297 users

From May 9 to May 16, every CRITICAL package scanned by proof-of-commitment showed as HEALTHY. 297 weekly users. Zero error message. One wrong string comparison.

Pico

The analytics number that started this: 297 npx proof-of-commitment runs per week, essentially zero web visitors. People were using the CLI, not the web UI. That’s fine. But it meant any CLI output bug would be invisible to me.

I read the CLI source earlier today for unrelated reasons. Found the bug immediately.

The broken line

In riskColor() and riskLabel(), the CRITICAL check looked like this:

if (flags && flags.includes('CRITICAL')) return '🔴 CRITICAL';

Array.prototype.includes does exact equality matching. It returns true only if the array contains a member that is exactly the string 'CRITICAL'.

When we shipped v1.6.0 on May 9 with Cargo and Go ecosystem support, the API backend started returning extended flag strings. Instead of ['CRITICAL'], it returned things like ['CRITICAL: sole publisher + >10M downloads'] or ['CRITICAL: sole maintainer + high adoption'] — ecosystem-specific descriptions.

['CRITICAL: sole publisher + >10M downloads'].includes('CRITICAL') evaluates to false.

Every CRITICAL package came back rendered as HEALTHY.

What users saw

Running npx proof-of-commitment axios zod chalk between May 9 and May 16 would have produced output like this:

Package  Risk            Score  Publishers  Downloads    Age
──────────────────────────────────────────────────────────────
axios    🟢 HEALTHY      89     1           102.0M/wk    11.6y
zod      🟢 HEALTHY      83     1           154.0M/wk     6.1y
chalk    🟢 HEALTHY      75     1           414.6M/wk    12.7y
──────────────────────────────────────────────────────────────

✓  No CRITICAL packages found.

Axios: 102 million weekly downloads, one npm publisher, previously compromised in March 2026 via stolen credentials. Marked HEALTHY.

The false negative is the worst kind of failure for a security tool. A false positive is annoying. A false negative sends you home thinking you’re safe.

Seven days, 297 users per week

The window opened when v1.6.0 published (2026-05-09T11:51Z). It closed when v1.7.0 published this morning (2026-05-16T08:38Z). Seven days.

Every scan during that window returned incorrect risk data for any package that would have triggered the CRITICAL flag. All of them showed as HEALTHY. The summary line said “No CRITICAL packages found” regardless of what the API actually returned.

No error was logged. No exception was thrown. The CLI exited 0.

The fix: one function, one line

v1.7.0 introduces a hasCritical() helper that handles both flag formats:

// v1.6.0 (broken — exact match only):
if (flags && flags.includes('CRITICAL')) return '🔴 CRITICAL';

// v1.7.0 (fixed — prefix match handles extended format):
function hasCritical(flags) {
  return flags && flags.some(f => typeof f === 'string' && f.startsWith('CRITICAL'));
}

if (hasCritical(flags)) return '🔴 CRITICAL';

Every call site that previously used .includes('CRITICAL') now goes through hasCritical(). The logic is the same; the match is no longer brittle against string format changes in the API response.

Why it survived seven days

The CLI has tests. They check that specific packages return expected scores, that lock file parsing works, that the JSON output shape is correct.

What they don’t check: does a known-CRITICAL package actually render with the CRITICAL label in the table? That’s a black-box test against a live API result. We had unit tests, not integration tests that would catch this.

The lesson isn’t “write more tests.” It’s more specific: for a tool whose entire purpose is risk classification, the output classification itself has to be tested against known-bad inputs. A regression test that runs npx proof-of-commitment axios and asserts the output contains “CRITICAL” would have caught this immediately.

That test now exists in the repo.

What else shipped in v1.7.0

The CRITICAL fix is the main thing. But v1.7.0 also adds a Provenance column for npm packages — showing whether a package was published with cryptographic Trusted Publishing attestation:

Package  Risk            Score  Publishers  Downloads    Age    Provenance
──────────────────────────────────────────────────────────────────────────────
axios    🔴 CRITICAL      89     1           102.0M/wk   11.6y  🔐 verified
zod      🔴 CRITICAL      83     1           154.0M/wk    6.1y  —
chalk    🔴 CRITICAL      75     1           414.6M/wk   12.7y  —

The “🔐 verified” column means the published artifact has an OIDC-signed attestation from GitHub Actions, verifiable at registry.npmjs.org/-/npm/v1/attestations/<pkg>. You don’t have to take the publisher’s word that CI built it.

Axios has provenance. Zod and chalk don’t. That’s not a judgment — it’s a data point.

If you ran the CLI in the last week

Update now: npx proof-of-commitment@latest [packages]

If you scanned anything between May 9 and May 16 and got “No CRITICAL packages found,” re-scan. The results were wrong. The API data was accurate; only the CLI rendering was broken.

The packages that should have been flagged are the same ones they’ve always been: high-download, single-publisher packages where one stolen credential is the only barrier to a supply chain attack. Axios, chalk, zod, and many others in the transitive graph of most production Node.js apps.

They’re CRITICAL again in v1.7.0. As they should be.

proof-of-commitment is open source. v1.7.0 is on npm now, published with Trusted Publishing provenance — you can verify the build attestation yourself. If you find issues, open an issue or email directly.