AI's strongest bug-finding results so far came out of defensive research

The bug Big Sleep found in SQLite

In early October 2024, an AI agent called Big Sleep read the source code of SQLite, the database engine that ships inside nearly every phone and browser. It found a flaw no human had reported: a stack buffer underflow in the code that plans how to answer a query. The bug fired on one edge case, a query constraint on a column the engine marks with the placeholder −1, which the function mishandled into a write past the start of an array. Project Zero and DeepMind, who built the agent and reported the find, called it the first public case of an AI agent finding a previously unknown, exploitable memory-safety bug in widely used software.¹ The flaw was caught in the development version and fixed the same day, so no shipped copy of SQLite was exposed.¹

Two things sit inside that result. The team that scored the find also built the tool, so the significance claim comes from an interested party. And the write-up says plainly that a target-specific fuzzer, a standard automated bug-finder, would likely have found the same bug at least as fast.¹ What Big Sleep established is narrow and real: a system found one genuine bug, once.

The worry it speaks to is older than the tool. In February 2018, twenty-six researchers from Oxford, Cambridge, OpenAI and elsewhere published The Malicious Use of Artificial Intelligence, the report that gave the modern argument its shape, and their case was careful. AI would widen who could attack, how fast, and how many targets at once, and automating vulnerability discovery and the writing of exploit code was one named scenario.² They put it as a trajectory, not a present fact: AI hacking tools "may exhibit much better performance" than what has been possible, and beat humans "ultimately (though perhaps not for some time)."²

Seven years on, that worry is usually stated as one claim. It is really three, resting on different evidence. One: AI can find real, previously unknown vulnerabilities. Two: AI can run an intrusion end to end, finding the way in, writing the exploit, and moving through a hardened target on its own, at scale. Three: this hands attackers a decisive advantage, because attack gets cheaper faster than defense can adapt. The evidence shows the first, gestures at the second, and says little about the third. Keeping them apart is what most cyber-uplift claims fail to do.

Eighteen real flaws in 54 million lines

Big Sleep was one bug. The clearest sign this is a real capability came ten months later. In August 2025, DARPA ran the finals of its AI Cyber Challenge at DEF CON. Seven finalist systems were pointed at roughly 54 million lines of real open-source code. On a synthetic test set seeded with known bugs they found 54 of 63, about 86%, and wrote working patches for 43. In the real code, they surfaced 18 genuine, previously unknown vulnerabilities, six in C and twelve in Java, and patched eleven. Every team found at least one. They worked at a pace no human review matches: about 45 minutes per task, at a machine cost near $152.³

The challenge was defensive. Scoring rewarded finding a bug and shipping a fix, and DARPA's program manager said it changed what is possible in automatically finding, and more importantly fixing, vulnerabilities.³ Both of the strongest demonstrations of AI finding real bugs, Big Sleep and this challenge, came from teams trying to protect software rather than break into it.

Where the tests hit a ceiling

Finding a bug is one step. Running an intrusion is many: finding the way in, writing an exploit that beats the target's defenses, and moving through unmapped ground without being caught. This is the same long, multi-step autonomy that models have struggled with elsewhere in this course. The published tests of that fuller capability all stop well short of it.

Cybench is an academic benchmark of 40 professional capture-the-flag tasks: security puzzles where solving means prying a hidden token out of a deliberately vulnerable system. It scores agents against the time human teams first took to solve each. Working unguided, with no hints toward the sub-steps, the best agent, Claude 3.5 Sonnet, solved 17.5%, seven of the forty. GPT-4o solved five; Claude 3 Opus and o1-preview four each. No unguided agent solved a single task that had taken a human team longer than eleven minutes, and the hardest task in the set took human experts nearly 25 hours.⁴ The systems clear the quick puzzles and stall on everything that takes real time.

DeepMind ran a similar test on its own Gemini models and reported the same shape. Its best agent, Gemini 1.0 Ultra, solved 3 of 13 easy-to-medium challenges the lab built in-house and none of 13 harder Hack The Box puzzles. Reading the number as a lab grading its own homework, the authors concluded they had found no strong dangerous capability, only early warning signs, and set the bar they were testing against as "poorly protected" assets, not hardened ones.⁵

The one paper that reports a high success rate at autonomous exploitation carries its own correction. Fang and colleagues at Illinois gave a GPT-4 agent 15 already-disclosed vulnerabilities, each with its public write-up. It exploited 13 of 15, 87%, where every other model and scanner scored zero. Withhold the write-up and the same agent exploited one of fifteen.⁶ The system was executing a known bug's public description, not discovering anything. The authors built the agent themselves. Even their best case describes exploiting disclosed bugs, the ordinary work of the lag between a patch and its installation, not the novel-intrusion capability the second claim needs.

The most eye-catching present-day figure comes from a vendor selling the capability. In mid-2025, an autonomous penetration-testing system called XBOW reached number one on HackerOne's US bug-bounty leaderboard, filing more than a thousand vulnerability reports in a few months. The targets were public web applications that bug-bounty programs invite anyone to probe, not hardened systems. And the reports were not fully unattended: to satisfy HackerOne's rules on AI tools, human staff reviewed the findings before submission.⁷ It is a real capability against soft targets with a person still in the loop, not an autonomous operation against a defended one.

Who gets the advantage

The third claim is the one people usually mean: not that AI can find a bug, but that it tips the whole balance toward attackers. Here the evidence is thinnest, and it is contested by the very demonstrations offered for it.

Start with what a national agency will say on the record. In January 2024 the UK's National Cyber Security Centre judged that AI will almost certainly increase the volume and impact of cyber attacks over the next two years. But it placed that uplift almost entirely in reconnaissance and social engineering: sharper phishing, with the grammar tells gone. It rated the uplift to malware and exploit development only a "realistic possibility."⁸

That judgment collides with the origin of the argument. Brundage's 2018 report worried that AI would widen the set of actors who could attack, lifting the floor so less-skilled people could do more.² The NCSC picture is closer to the opposite. At the hard end, AI concentrates advantage among those already expert. It lifts novices mainly in low-skill work, phishing and gaining access, where the task was never the barrier.⁸ Both cannot be the whole story, and no source here reconciles them. The benchmark ceilings point the same way: models help most where a skilled human already could.⁴

Then there is the symmetry the doom version skips. Every capability that finds a bug for an attacker also finds it for a defender, and the strongest evidence on the table was produced by defenders. Big Sleep's flaw was fixed before it shipped.¹ The DARPA challenge rewarded patches, and its director described the tools as giving defenders an edge in finding and fixing flaws at speed and scale.⁹ AI also shortens the window between a bug becoming public and being exploited, which cuts the other way and which NCSC flags.⁸ The balance could tip toward either side. No result on record shows it tipping decisively toward attack.

So the two loud versions of this argument each overreach, in opposite directions, and each is missing a specific thing. The doom version, that AI collapses the cost of attack faster than defense can adapt and breaks it, has no demonstrated result behind it. NCSC's measured "almost certainly more volume" is the strongest sourced claim, and it sits a long way from a decisive edge. The dismissal version, that this is just a chatbot regurgitating known bugs, cannot explain Big Sleep or the eighteen novel flaws the DARPA systems found. The evidence sits between the poles, and neither gets to borrow its confidence from the other.