Why AI content marketplaces are the future

A US body is formulating a content marketplace framework in which AI companies can pay a fair price for fair use.

by Rob Corbidge

Published: 15:06, 21 August 2025

One of the central commercial discrepancies affecting the publishing industry in the current era is the simple fact that big players are able to strike deals directly with AI companies, while all-important small- and medium-sized publishers generally remain prey to whatever content harvesting crawlers head their way to feast on their data for free.

The reasons behind this current awful state of affairs can be reasonably encapsulated by the word "disruption". The speed at which LLM systems advanced and pushed into widespread public usage after ChatGPT emerged less than three years ago has been remarkable to witness, even for veterans of other disruptions. 

Publishing was akin to a medieval baggage train set upon by an invading army, such was the extent of the carnage and carrying off of valuables by those waving the AI banner, a banner which, for a time, protected them from accusations of theft.

Flat fee access deals were struck with these raiders by larger publishers, but even then those were largely of the kind of deals you strike in a hurry with raiders, which are never good deals.

If we take the AI companies at face value - not even the biggest ones, but also some you've never heard of or are never likely to - many of them enjoy inflated valuations which even a large, profitable publisher will only reach in a fever dream. Given that, even the better deals struck with larger publishers still look like pennies thrown to beggars.

As it becomes clearer by the day that website referrals from AI Answer Engines are not going to be a traffic driver of any reasonable kind for our industry, it also becomes apparent that a market value must start to emerge for the content used to train the LLMs driving the disruption. 

This apparency is seeded in facts such as that the AI Answer Engines that are, by all accounts, destined to become the Search of the immediate future, require quality content to consume in order to have any use, and hence, popularity.

So then, a virtuous commercial circle seems possible between those who produce content and those who require said content to make their LLMs work better than their rivals. Or if not quite virtuous, a better state than what is happening right now.

To this end, the non-profit Interactive Advertising Bureau in the United States is driving an initiative to "Set AI-Era Publisher Monetization Standards" through a working party set up under the IAB Tech Lab umbrella. As they say themselves: "It's time to build the technical plumbing for a better path forward."

Locking the door to robots

The AI Content Monetization Protocols (CoMP) Working Group is considering three central components in order to achieve this "better path forward". The first is a reliable and secure way of blocking bot traffic and prevent unwanted scraping of content. The IAB Tech Lab describe this as a "door with a lock". It's clear the honourable but ignorable robots.txt protocol no longer cuts the mustard in a more dishonourable and ignorant age, so this is an obvious and needed requirement to meet.

Second comes a requirement for "LLM-Friendly Discovery". Essentially, this points to the pressing need for a marketplace, where publishers can provide content to those seeking to utilise it for AI applications. The seekers can find the right content, and the publishers can get paid. Content can be sampled and compared.

Finally, and importantly, there's a requirement for what IAB Labs term as an "LLM Ingest API", an industry standard mechanism for the purposes of content ingestion by AI systems with monetisation built in. You take it, you pay for it. This has been termed, with possible irony, the "easy button" by the IAB working group.

In short, this is a serious attempt to set the technical standards required to have a functioning content marketplace and to be a logical step if we are to move on from the plunder era to something more equitable.

There is a debate about what monetisation would look like of course and the IAB Labs group isn't foolish about it. They accept that a number of different models need to exist in such a content marketplace, with everything from an all-you-can-eat website buffet open 24/7, to a much more exclusive menu of content options, both depending on the client and the publisher. 

Then there are the payment models themselves, with three main types identified:

• Pay Per Crawl
• Aggregation or Pay Per Use
• Outcome or attribution-based payment

Naturally I'm in the Pay Per Crawl or nothing camp, which is why I'm not on the working group. Among many other reasons. 

As much as I would like to see commercial vengeance weighed on the AI industry, it's not the path forward. These three options outlined by IAB Labs are not exclusive and form a framework in which both parties in any deal can find a model that works for them.

It even brings to mind the possibility of a co-operative venture in which publisher and AI business share both risk and reward. Imagine such a thing!

Of course, there's no guarantee that the IAB attempt to "build the technical plumbing" for a Content Marketplace will prove a success. What counts in its favour is timing being right for such a thing to emerge, they have some serious people onboard, and they are an American organisation, and the US leads the AI field.

The situation in which only the big publishers can get paid isn't good for anyone. After all, the big publishers were little publishers once.