I Watched AI Agents Try to Hack My Vibe-Coded Website

A few weeks ago, I watched a small team of artificial intelligence agents spend roughly 10 minutes trying to hack into my brand new vibe-coded website.

The AI agents, developed by startup RunSybil, worked together to probe my poor site to identify weak spots. An orchestrator agent, called Sybil, oversees several more specialized agents all powered by a combination of custom language models and off-the-shelf APIs.

Whereas conventional vulnerability scanners probe for specific known problems, Sybil is able to operate at a higher level, using artificial intuition to figure out weaknesses. It might, for example, work out that a guest user has privileged access—something a regular scanner might miss—and use this to build an attack.

Ariel Herbert-Voss, CEO and cofounder of RunSybil, says that increasingly capable AI models are likely to revolutionize both offensive and defensive cybersecurity. “I would argue that we’re definitely on the cusp of a technology explosion in terms of capabilities that both bad and good actors can take advantage of,” Herbert-Voss told me. “Our mission is to build the next generation of offensive security testing just to help everybody keep up.”

The website targeted by Sybil was one I created recently using Claude Code to help me sort through new AI research papers. The site, which I call Arxiv Slurper consists of a backend server that accesses the Arxiv—where most AI research is posted—along with a few other resources, combing through paper abstracts for words like “novel”, “first”, “surprising” as well as some technical terms I’m interested in. It’s a work in progress, but I was impressed with how easy it was to cobble together something potentially useful, even if I had to fix a few bugs and configuration issues by hand.

A key problem with this kind of vibe-coded site, however, is that it’s hard to know what kinds of security vulnerabilities you may have introduced. So when I spoke to Herbert-Voss about Sybil, I decided to ask if it could test my new site for weaknesses. Thankfully, and only because my site is so incredibly basic, Sybil did not find any vulnerabilities.

Herbert-Voss says most vulnerabilities tend to be the result of more complex functionality like forms, plugins, and cryptographic features. We watched as the same agents tried probing a dummy ecommerce website with known vulnerabilities owned by Herbert-Voss. Sybil built a map of the application and how it is accessed, probed for weak spots by manipulating parameters and testing edge cases, and then chained together findings, testing hypotheses, and escalating until it breaks something meaningful. In this case, it did identify ways to hack the site. Unlike a human, Herbert-Voss says Sybil runs thousands of these processes in parallel, doesn’t miss details, and doesn’t stop. “The result is something that behaves like a seasoned attacker but operates with machine precision and scale,” he says.

“AI-powered pen testing is a promising direction that can have significant benefits for defending systems,” says Lujo Bauer, a computer scientist at Carnegie Mellon University (CMU) who specializes in AI and computer security. Bauer recently coauthored a study with others from CMU and a researcher from AI company Anthropic that explores the promise of AI penetration testing. The researchers found that the most advanced commercial models could not perform network attacks but developed a system that set high-level objectives like scanning a network or infecting a host, which enabled them to perform penetration tests.