For years, publishers have watched AI companies quietly access their content to train large language models, power generative tools and build products, often without consent, credit or compensation.
This week, a new chapter begins.
Cloudflare, a major internet infrastructure provider, has decided to block AI bots from accessing websites unless publishers explicitly allow them to do so.
“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Cloudflare CEO Matthew Prince.
Cloudflare, which manages about 20% of global web traffic, became the first major internet infrastructure provider to block AI crawlers—such as OpenAI, Google and Anthropic—by default.. The decision, made on July 1, is a direct response to growing concerns over AI companies scraping copyrighted content without permission or compensation.
Unlike robots.txt (which bots often ignore), these new network-level controls give publishers real power to keep AI scrapers out—unless they say otherwise.
Cloudflare’s controls are a big step up, but they aren’t foolproof. Bad actors may still find ways around them. Publishers can fight back by updating their terms of service (TOS), using CAPTCHAs and watermarking to better shield their content from AI scraping.
By making bot-blocking the default, Cloudflare is handing power back to publishers, allowing them to decide who can access their content and under what terms.
“Pay Per Crawl” and Monetization Opportunities
Cloudflare is also opening new doors with its “pay per crawl” model, which is currently in private beta. This approach lets publishers set terms and potentially charge AI companies for access to their content.
Publishers can dig into their Cloudflare settings, decide which AI bots (if any) to let in and consider testing this “pay per crawl” beta for potential revenue. By auditing bot access and tracking policy shifts, they can stay protected and get paid.
As referral traffic from traditional search engines declines, creating bot paywalls could become a vital new source of revenue. Publishers gain the ability to whitelist specific bots, negotiate direct deals and see exactly how their content platforms use their content—whether for training, inference or search.
It’s a significant step toward a more transparent and sustainable economic model for AI.
Cloudflare’s move to a permission-first model could reset the rules.
As more players follow suit, the power could swing back to content creators, setting the stage for fairer deals with AI vendors and a more sustainable digital ecosystem.
Publisher Perspective: Control, Transparency and Value
Major publishers like DMGT, Associated Press, Condé Nast, The Atlantic and TIME have voiced strong support for Cloudflare’s initiative.
Rich Caccappolo,Vice Chairman of dmg media, summed up the industry’s sentiment: “We support any innovation that creates a structured and transparent relationship between content creators and AI platforms and respects fundamental property rights.”
Paul Bannister, Chief Strategy Officer at Raptive, added that Cloudflare’s “pay per crawl” model is an experimental step toward building a content marketplace.
“While it’s unclear if it’s the ideal solution, testing it will generate valuable data to inform better models,” said Bannister. “We’ve been part of the publisher group in discussions with Cloudflare and are eager to see how it plays out.”
Cloudfare’s new model aligns closely with the themes we’ve explored at AdMonsters, where the focus has long been on empowering publishers with the tools and strategies needed to protect their intellectual property, demand fair value and adapt to digital media’s evolution.
With Cloudflare’s efforts, publishers will have one more tool to control how their content is accessed and monetized with AI.
The message for publishers is clear: Take advantage of these new controls, explore emerging monetization models and continue to advocate for fair compensation and transparency.