The Best AI Model Found a Firebase Exploit 7 Out of 10 Times

Last week, Kasra Rahjerdi published a benchmark that made 305 points on Hacker News. He built BookNook — a vulnerable fake book review app with a real Firebase broken access control flaw — and gave nine AI models $10 and two hours each to find and exploit it.

The top performer: GPT-5.5 at 7/10. Claude: 2/10. Several of the nine models tested scored zero. Total spend: $1,500.

The immediate reaction was about the winner. GPT-5.5 beats Claude. OpenAI wins security. But that framing misses the actual result.

The actual result is this: the best AI model in the world failed to find a known vulnerability three times out of ten.

A 3.5x variance problem

The gap between GPT-5.5 (7/10) and Claude (2/10) is 3.5x. Multiple models couldn't find the exploit at all, despite being frontier-class systems.

This is the same vulnerability. The same application. The same budget. The same two-hour window. The spread isn't measurement noise — it's a 350% swing in detection capability across a selection of frontier models.

If you ran this benchmark on a different day with slightly different prompting, the rankings would shift. The HN comments noted as much: the methodology gave GPT-5.5 a research whitelist that Claude didn't have, making Claude's guardrail-hampered 2/10 a partially independent variable. Change the harness, change the results. The numbers aren't stable.

That instability is precisely the problem. "We ran an AI security audit" is becoming a trust signal — appearing in SOC2 supplements, security questionnaires, due diligence checklists. That signal is backed by a process with 30-100% failure rates depending on which model ran it and how the prompt was written.

Why single-pass audits fail structurally

A security audit is a search problem. You're looking for a vulnerability that, by design, isn't labeled as one. The attacker has infinite patience; the audit has a budget and a clock.

LLMs solve search problems probabilistically. Run the same prompt ten times and you get ten different exploration paths. That's not a defect — it's how the sampling works. The problem is that a single-pass audit picks exactly one path, then stops.

GPT-5.5 found the Firebase exploit in seven of its ten attempts. In the three it missed, it explored the wrong paths first, ran out of budget, and stopped. There's nothing wrong with those three runs in isolation. They were doing exactly what the model does. The problem is that a real audit doesn't get ten attempts — it gets one.

A single $10 run of GPT-5.5 has a 30% chance of missing a vulnerability it's capable of finding. A single $10 run of Claude has an 80% chance of missing it.

"AI reviewed this" is not a trust signal. It's a lottery ticket.

Commit-level checking changes the math

The $1,500 benchmark was structured as one big audit per model. That's the wrong unit of analysis.

The right unit is the commit. Every code change is a delta — smaller surface, bounded scope, immediately after the vulnerability was introduced. Checking at commit time costs a fraction of a full-repo audit. And you don't have to be right every time; you have to be right before the code ships.

If your CI runs a security check on every pull request, and that check runs multiple models in parallel, the math shifts dramatically. Three models that independently have a 70%, 30%, and 20% detection rate produce a combined false-negative rate of 17% when you require at least one detection. Require two of three to agree, and you filter out most hallucinated findings while still catching what's real.

The noise in single-pass LLM audits is real and documented. Multi-pass, multi-model, commit-level checking absorbs that noise — not by making any individual model more reliable, but by not relying on any individual model being reliable.

The behavioral signal that accumulates

There's a second insight buried in Rahjerdi's data that goes beyond the security audit framing.

The benchmark measures detection capability at a point in time. But the packages and repositories you depend on aren't audited once — they ship continuously. A maintainer who ships 800 releases over a decade generates a different behavioral signal than one who shipped 12 releases before going dark. That behavioral history is evidence that the code is being actively reviewed, updated, and maintained by someone with skin in the game.

That's not an AI audit. That's a commitment record. It's hard to fake because it requires time, sustained effort, and real-world deployments that would surface bugs if they existed.

The AI benchmark proves that single-shot detection is noisy. Behavioral history proves something the benchmark can't: that this maintainer kept caring about this code across hundreds of consecutive commits, years of releases, and 400M weekly downloads.

Those are different signals. The AI audit catches what's verifiable in a two-hour window. The behavioral record captures what's hard to fake over time.

Both matter. Neither alone is sufficient. The trust layer that combines them — continuous commit-level checking plus behavioral commitment history — is the one worth building.

We're building it at Commit.

---

The full benchmark is at kasra.blog. The HN thread is here.

We're building Commit — behavioral trust infrastructure for code and the people who ship it. If your security posture currently rests on "AI reviewed this," let's talk.

Related: Benchmarks Lied. Now What? · Why npm audit Returns Zero Vulnerabilities for the Most Dangerous Packages