Cloudflare: bots now account for the majority of web traffic

On Cloudflare's network, traffic to websites is now more than half automated, and the number of daily requests from AI agents has grown by more than 1,700 percent over the past year. In its September 30 summary marking the company's sixteenth anniversary, Cloudflare described software that browses and performs tasks on people's behalf as the internet's "second audience." According to the company, it now handles an average of 115 million HTTP requests per second, up from 63 million at the end of 2024. Automated traffic, however, is a broader category than AI agents alone: it also includes traditional search-engine crawlers and other bots.

The shift also affects the logic that has so far financed the internet. Search engines used to crawl websites and then direct visitors to them, who would see ads, subscribe or make purchases. AI services, by contrast, often use information found on websites to formulate answers without the user ever reaching the original source. According to Cloudflare, in some of the most heavily crawled categories, including retail, software and financial services, the number of human visits has dropped by as much as 40 percent in under a year. Machine requests continue to consume resources in the meantime, but they don't necessarily generate revenue for the site's operator. Monetization Gateway is meant to address this: in its closed beta, eligible U.S. Cloudflare customers can set fees for resources accessible through the network—websites, datasets, programming interfaces or tools usable by AI agents. Built on the open x402 protocol, the system sends an HTTP 402 response indicating a payment requirement whenever a request matches a billing rule, allowing the agent to pay the seller directly. Following the summer announcement, thousands signed up for the waitlist, and the company says there has been recurring demand for machine visitors to pay while access for human audiences remains unchanged.

Pay Per Use is aimed primarily at publishers who, due to their size or bargaining position, cannot negotiate separate licensing agreements with AI companies. Under this model, payment is tied to actual use of the content, not merely to the fact that a bot downloaded the page. Participating buyers operate as verified web crawlers, define what counts as usage and how much they pay for it, and publishers can decide whether to accept the offer. Both payment solutions are currently in beta. Control over content is also served by the Disallow AI Training feature, introduced on September 15, which allows a site to keep appearing in search results while signaling that it does not permit its content to be used for training models. According to Cloudflare, Apple, Google and Microsoft have committed to respecting this signal. Since it relies on signals supported by those particular providers, it allows search presence and training use to be regulated separately, rather than providing technical copy protection enforceable against all internet actors.

Among the security announcements of the anniversary week is the expansion of free access to the Threat Events Platform to all Cloudflare accounts, as well as the launch of Threat Signals. The latter processes publicly available security reports: it summarizes them, identifies technical traces linked to attacks—such as malicious domains and IP addresses—and then organizes the information into tagged events. The free tier offers processing of one chosen RSS feed and storage of the resulting data for up to thirty days; enterprise plans add more sources, additional threat databases and broader configuration options. Application Profiles learns the typical structure and format of requests to web applications and then flags traffic that deviates from it. This can help identify suspicious requests that don't match a known attack pattern but, for example, contain text where the application expects a number. The flag alone does not block the request: after reviewing the data, the operator can set up blocking rules. According to Cloudflare, this is especially important because AI also makes it easier for attackers to quickly vary their attack methods. The feature is already available to API Security customers, while other invited enterprise customers can try it in closed beta.

Cloudflare co-founders Matthew Prince and Michelle Zatlyn wrote in their annual letter that they had previously expected automated traffic to overtake human traffic in the second half of 2027, but the shift actually occurred as early as May 2026. If current trends continue, they say machine traffic could grow to a thousand times human traffic within five years. This is a forecast based on the explosive growth in agent activity: to answer a single user query, software may visit hundreds or even thousands of pages. The central question raised by these announcements is therefore how websites that provide the content and services behind such answers can share in the resulting revenue, even as users no longer necessarily visit them directly.

Ugar