With little data and no clear standards, publishers are experimenting with AI crawl controls to probe a still-unanswered question: What’s the real value of letting AI in?
When Cloudflare rolled out its Pay Per Crawl and AI Crawl Control tools last year, it gave publishers what they’d long been asking for: the power to choose who can mind their content.
But power isn’t a playbook.
For publishers already juggling monetization pressure and traffic volatility, AI crawling introduced a new unknown that didn’t fit neatly into their existing ad ops and SEO frameworks. Blocking AI bots felt risky, allowing everything felt reckless, and, with little data to guide them, they were forced to guess.
“[It] was exciting when Cloudflare made the announcement,” said Amanda Martin, VP of Publisher Growth and Strategy at Mediavine, which represents publishers and helps them monetize. “But we weren’t sure what that meant from an execution standpoint. Having the optionality is one thing. Knowing how to use it responsibly is another.”
Mediavine has partnered closely with BigScoots to help bridge that gap. Through tailored hosting plans and technical collaboration, BigScoots integrates Cloudflare’s advanced controls directly into its platform for Mediavine publishers, providing actionable data and technical support that align with publishers’ monetization needs.
That uncertainty is why BigScoots, a managed hosting provider for WordPress publishers, started deploying Cloudflare’s AI Crawl Control as part of its hosting infrastructure. The goal is to give publishers clearer visibility into how AI agents interact with their sites.
Visibility Before Strategy
For much of the past year, publisher sentiment about AI crawling has swung between extremes. AI was either framed as an existential threat siphoning value from content or as the inevitable future of content discovery that publishers had no choice but to accommodate.
The reality, as Martin describes it, is messier.
Discovery no longer maps cleanly to monetization. AI agents might surface content without driving traffic, train once before disappearing or continually scrape to generate answers. Without insight into these behaviors—or how often they’re happening—publishers are left guessing about the value exchange.
“AI is changing what discovery actually looks like,” Martin said. “It’s not a one-to-one transaction anymore. So publishers have to ask: Is this agent serving me through traffic, through discovery or not at all?”
Although early data, according to Martin, suggests that AI referrals are still a small percentage of overall publisher traffic, blocking a crawler could reduce exposure in AI-generated answers, Martin said. But allowing bots could increase infrastructure costs without a clear return on investment.
The missing piece has been visibility: who’s crawling, how frequently and for what purpose. An infrastructure provider like BigScoots connects publishers to Cloudflare’s enterprise-level controls without having to worry about the technical details, Martin said.
“You can start to think agent by agent,” she said. “Are they actually using your site? Are they coming back repeatedly? That context changes the conversation.”
Why This Decision Sits Outside the Ad Stack
But even with better visibility into AI crawling, many publishers still face a practical question: Who actually owns this decision?
AI crawl controls sit outside of familiar ad tech and CMS tools. They’re enforced at the hosting and Content Delivery Network (CDN) level where traffic is filtered before it ever reaches analytics, ad delivery or monetization systems. That setup helps explain why this topic often feels abstract to ad ops teams, even though the downstream effects eventually land on their desks.
A hosting platform like BigScoots doesn’t determine which AI agents publishers should block or allow. Its role is more mechanical than strategic, making Cloudflare’s enterprise-grade controls accessible to publishers.
That separation is intentional, according to Martin.
“We sit in ad management and monetization,” she said. “Hosting goes beyond that. There are controls that have nothing to do with monetization, and those need to stay in the publisher’s hands.”
The distinction matters because AI crawling introduces costs and risks before it introduces revenue. Nonhuman traffic affects infrastructure load and page performance regardless of whether it generates referral traffic or eventually leads to licensing discussions. Those trade-offs vary widely by site.
Many Mediavine publishers rely on managed hosting precisely to avoid operational complexity. Still, Martin argues AI crawling isn’t an area where centralized rules make sense.
“Sites are exposed to this in very different ways,” she said. “How dependent you are on search, how much AI traffic you’re seeing, how often agents come back all change the calculus.”
Experimentation Without Guarantees
There’s not that much data yet about AI crawling. Trends are directional, not definitive, and Martin is careful not to oversell what publishers can expect from asserting more control.
But, regardless, it’s always important to make the distinction between AI models that train once and those that return repeatedly. The latter could eventually lead to licensing deals.
“If someone is coming back to your site over and over again, that’s a different value proposition,” Martin said. “Knowing how relevant your site is to those agents is key to deciding whether the juice is worth the squeeze.”
In that context, blocking is more about leverage than protection.
When publishers assert themselves and set boundaries, it can prompt interest in licensing, even for smaller sites. But there’s a catch. Publishers must consider the cost. The nonhuman traffic generated by AI crawlers still consumes server resources, and smaller or less-efficient agents can become especially expensive.
Martin cautions against taking sweeping action. There’s no single threshold that defines when to block, allow or negotiate.
“I wouldn’t advocate for blocking everything,” she said. “It’s not a light-switch decision. You have to understand what traffic you’re getting, how you show up in those environments and whether this is actually impacting your business.”
No Playbook Yet
Publishers have been here before.
Ads.txt, MFA scrutiny and supply-path optimization all began as incomplete solutions that evolved through experimentation and industry pressure. Cloudflare’s AI Crawl Control may follow a similar path, but the timeline and outcome are far from clear.
Standards don’t yet exist, regulation remains unsettled and AI’s impact on publisher economics varies widely depending on category, scale and audience behavior.
“There’s no right or wrong answer yet,” Martin said. “It’s your comfort level and your threat level. If you’re seeing real traffic decline tied to AI overviews, this may be a priority. If you’re not, it may not be.”
AI Crawl Control hands publishers a decision to make, but not a road map to follow. And in a world where the rules are still being written, having a choice is better than nothing at all.