The digital locks to make AI botflies pay

Technology is responding to the era of content harvesting and AI companies will have to pay for their base materials

by Rob Corbidge

Published: 15:02, 11 September 2025

With the end of the internet's age of relative honour over content use coming to a close, and the rules of copyright and fair use in flux and question, placing our content under digital lock and key to ward off the parasitic digital mosquitos sucking up AI training data for free is now the order of the day.

We've written about the potential future of content marketplaces recently, and even as we discover work on the technical protocols to make such a thing possible is being done in one place, we find out it's being done in several others. The requirement for protection is meeting solutions, regardless of what is occurring in the legal sphere.

A couple of new examples of this effort have sailed into our field of view in the past few days. The Really Simple Licensing (RSL) standard is a very promising project that already has outfits such as  Reddit, Quora and Yahoo onboard.

"The RSL Standard gives publishers and platforms a clear, scalable way to set licensing terms in the AI era," said Steve Huffman, CEO of Reddit in an RSL press release. "The RSL Collective offers a path to do it together. Reddit supports both as important steps toward protecting the open web and the communities that make it thrive."

One of the key personnel behind RSL is Eckart Walther, co-creator of the RSS syndication standard. If adoption is a true indicator of success, he clearly has a solid track record.

RSL is designed to be added to the robots.txt protocol, with machine-readable licensing and royalty terms essentially setting out the terms and conditions for the website content it sits in front of.

The organisation themselves do a better job of explaining it with brevity, so here you go:

RSL is an open, XML-based document format for defining machine-readable licensing terms for digital assets, including websites, web pages, books, videos, images, and proprietary datasets. It enables publishers, authors, and application developers to:

• Define licensing and compensation terms, including free, pay-per-crawl, and pay-per-inference, to use digital assets for AI training, web search, and other applications
• Create public, standardized catalogs and licensing terms for digital assets
• Enable clients to automate licensing and paying for legal access to digital assets
• Define and implement standardised licensing and royalty agreements

Essentially, RSL puts a solid door with terms of access clearly spelled out on it between content and crawler. Importantly, with RSL alone, that door can still be breached by bad actors, however the RSL team is working with Fastly, the content delivery network specialists, to create a technical lock on that door. The lock would only admit AI bots if the licensing conditions in the RSL file had been agreed to.

Arguably, if such a standard as RSL saw proper widespread adoption, then any such breach would become a matter of collective concern for everyone using it, as Doug Leeds, former CEO of IAC Publishing and Ask.com, who now sits on the RSL Publisher Advisory Council told The Verge: "All participants in the collective rights organisation participate in the enforcement of any infringement."

Apes together strong, indeed. The lock might not be needed, or I am being naive?

The other side needs to buy in, of course. By the other side, we mean the cash-engorged strip miners who've been busy "transforming" the sweat of our publishing brows into outrageous promises that investors are still, for the most part, finding attractive.

As Matthew Prince of Cloudflare pointed out recently, you only need someone big to accept they must pay for content, and then everyone remotely reputable will follow suit.

Another interesting solution in the stop-stealing-our-content movement comes from ProRata. The most interesting part of their offering is providing content attribution for AI-produced answers. That is to say, the sources from which the answer is drawn can be identified. They have built a proof-of-concept answer engine of their own, https://chat.gist.ai/, and are working with a number of larger publishers in order to bring their technology to bear and help restore some order to the pirates' playground enjoyed currently by AI bots.

Bit by bit, solutions to the situation we have found ourselves in are becoming known and becoming usable. Make no mistake, this is simply a restoration of the natural order of things, whereby ownership and creative rights mean something again and judgement is not based on some Will-o'-the-Wisp promises about AGI, and never mind what gets wrecked on the illusive path to it.