Publishers are rallying against what feels like an existential threat: Generative AI and large language models (LLMs) repurposing premium content without permission or payment.
Rather than handing their content over to the machines without a fight, publishers want to ensure control, consent, credit and compensation are baked into any AI licensing deal. The IAB Tech Lab launched the LLM Content Ingest API Initiative in June to establish a framework for protecting these publisher requirements before AI companies can freely ingest their work.
Some publishers are considering shutting out AI companies like OpenAI, Anthropic and Perplexity completely in the hope that taking a hard line will bring these companies to the bargaining table.
Here’s what publishers who are participating in the Tech Lab’s initiative have to say about the viability of collaborating with AI firms.
Drafting a Framework to Block Unpaid AI Scraping
Last week in New York, more than 80 media leaders, platforms and tech vendors gathered as part of an IAB Tech Lab working group to discuss AI platforms scraping their content without permission and compensation.
AI companies were noticeably absent from the meeting. As IAB Tech Lab CEO Anthony Katsur put it, “Publishers are still very much concerned about whether those companies are even going to participate, which is why the consensus in the room was we have to block them.”
The meeting aimed to draft a technical framework that would let publishers control who accesses their content and under what terms. While the plan is still to develop such a framework, the consensus among publishers seems to be that negotiations have to start by cutting off AI vendors’ access.
The sell side of the industry is “coming together to think through ways to put friction into the system to make it harder for the AI systems to steal everybody’s content,” said Paul Bannister, chief strategy officer at publisher ad network Raptive, who attended the meeting.
The biggest question in publishers’ minds remains whether AI companies will voluntarily comply with publisher demands as outlined in the Tech Lab’s proposed framework. Thus far, AI companies have seen fit to ignore publisher attempts to restrict access to crawlers that scrape sites to train LLMs.
The framework details hybrid monetization models for AI licensing deals, from pay-per-crawl agreements to a revenue share awarded to publishers each time their content feeds AI-generated answers. But no single model is likely to suffice, and which AI companies will pay remains uncertain.
Some of these initiatives are already in action, however, and could give publishers a better idea of AI companies’ willingness to play ball.
For example, Cloudflare’s “pay to crawl” program, launched in July, could be a potential solution. By acting as gatekeepers at the network edge, companies like Cloudflare help publishers meter and monetize AI bot traffic. Human and TollBit are also rolling out tools to incentivize AI agents to compensate publishers financially for access.
Sustainable AI Revenue
But ensuring fair payment for crawling their content solves just part of the problem for publishers. Publishers also seek steady partnerships to bring predictability—and recurring revenue—to an unstable ecosystem.
Some publishers are pushing for deals that offer “real money, real transparency and — perhaps most elusive—a sense of control,” according to a recent report by Digiday. One commercial executive summarized the tension around AI licensing deals bluntly: “You have to assume the [upfront check] is the last time you’ll see that money.”
The Tech Lab initiative might just be the first line of defense for protecting publisher interests in the long term. The Tech Lab aims to publish a draft of its framework by Fall 2025.
Although there is widespread skepticism about AI companies agreeing to self regulate in partnership with publishers, there is also some optimism around the Tech Lab initiative.
As one anonymous publishing executive optimistically told Digiday, “If we do this right, everyone who creates information can take a share in a growing market.”