Online bot traffic will exceed human traffic by 2027, Cloudflare CEO says

Bots are taking over the web, according to Cloudflare CEO Matthew Prince. In an interview at the SXSW conference in Austin this week, he said that with the speed at which artificial intelligence is growing, AI bot traffic will exceed the amount of human traffic that’s online by 2027.

Prince explained that bots’ web usage has been increasing alongside the growth of generative AI technology because bots are capable of visiting far more sites to get answers for users’ chatbot queries.

“If a human were doing a task — let’s say you were shopping for a digital camera — and you might go to five websites. Your agent or the bot that’s doing that will often go to 1,000 times the number of sites that an actual human would visit,” Prince said. “So it might go to 5,000 sites. And that’s real traffic, and that’s real load, which everyone is having to deal with and take into account.”

Before the generative AI era, the internet was only about 20% bot traffic, with Google’s web crawler being the largest, according to Prince, whose infrastructure and security company is used by one-fifth of all websites. But beyond some other reputable crawlers, the only other bots were those used by scammers and bad actors.

“With the rise of generative AI, and its just insatiable need for data, we’re seeing a rise where we suspect that, in 2027, the amount of bot traffic online will exceed the amount of human traffic that’s online,” Prince said.

The executive also noted that this change to the web would require the development of new technologies, like sandboxes for AI agents that can be spun up on the fly and then torn down when their task has finished. These could come into play when consumers ask AI agents to perform certain tasks on their behalf, like planning a vacation.

“What we’re trying to think about is, how do we actually build that underlying infrastructure where you can — as easily as you open a new tab in your browser — you can actually spin up new code, which can then run and service the agents that are out there,” Prince said.

He imagines there will soon be a time when millions of these “sandboxes” for agents would be created every second.

Of course, bots’ use of the internet at this scale would require physical infrastructure in the form of data centers and servers. Prince pointed out that, during Covid, internet traffic increased so quickly, particularly among video streamers like YouTube, Disney, and Netflix, that some parts of the internet were nearly buckling under the strain.

“This [growth] is more gradual, but unlike Covid, where it spiked over two weeks and then it kind of plateaued at the new high, we’re seeing internet traffic grow and grow and grow, and we don’t see anything that’s going to slow it down or stop it,” Prince added.

All these concerns about overload are great marketing for Cloudflare, a company whose services focus on helping websites stay highly available, load quickly, and remain safe from attacks. Among its offerings is a content delivery network, a series of security and DDoS protections, and an “Always Online” technology that serves cached versions of websites when the main server fails or goes offline. It also provides businesses with tools to block the AI bot traffic they don’t want.

Still, Cloudflare’s scale gives it the advantage of being able to view the internet’s ongoing evolution and the quickly arising challenges facing the generative AI era.

“I think the thing that people don’t appreciate about AI is it’s a platform shift,” Prince said, recalling the web’s earlier platform shifts, like the move from the desktop to mobile. “AI is another platform shift…the way that you’re going to consume information is completely different.”