How to confound AI crawlers? Feed them on a diet of their own work

Cloudflare's CEO has unveiled an ingenious tool to help publishers keep their sites AI crawler-free

by Rob Corbidge

Published: 15:05, 26 June 2025

An unlikely ally is coming to the aid of publishing. Unlikely, because it has felt for ages that much of the consumer-facing tech industry has succeeded at the cost of our own publishing industry, and has not much concern for us, even while enabling easily-reined smaller content creators to thrive under its dubious umbrella.

Enter Matthew Prince, CEO of Cloudflare. It was Mr Prince who recently supplied us with the interesting metric of pages scraped by a search engine versus the number of visitors that the site scraped would receive in return. Last week Prince updated us with the latest such data and take a deep breath, because it's rough. 

Six months ago the ratio of pages crawled by OpenAI to visitors it sent a publisher was 250:1. We thought that was bad. A little more than 24 weeks later and the figure is 1,500:1.

Twenty-four weeks.

It's not going to get better until OpenAI et al hear the slurping sound at the bottom of publishing's cup is it? And it's not just them. Being so large makes them answerable, but once you get past the foolish and arrogant "we are the future" waffle there are countless smaller GenAI actors in operation too and in darker places.

Prince has now announced that Cloudflare have produced a tool that will confound the most determined of AI crawlers, and the way it operates is actually quite delicious. Rather than blocking a crawler -  an action which alerts it to the block and allows it to recalibrate its attack - the Cloudflare tool instead leads the crawler into an endless maze of (and here's the delicious part) GenAI content. Meaningless GenAI content. A technological moron mirror, if you will. Hoist with their own petard.

It's also dual use, acting as a honeypot for malign actors, as Cloudflare themselves point out: "No real human would go four links deep into a maze of AI-generated nonsense. Any visitor that does is very likely to be a bot, so this gives us a brand-new tool to identify and fingerprint bad bots."

It's hard not to sound like an advert for Cloudflare writing this, yet Prince's intervention is not entirely altruistic of course. A successful business such as Cloudflare wants a healthy, thriving web in which to transact more business. It's likely that Prince sees a future in which the living, creative web has been shrivelled on the vine by the ravage of AIGen aphids.

He's also suitably pugnacious about the matter, in a way in which few in publishing are willing or able. He didn't quite say "I'm going to clock those AI cowboys so hard they'll be saying IA for a month!" (No offence to our Spanish readers! - Ed) but he kinda did in saying, as reported by Axios, that "I go to war every single day with the Chinese government, the Russian government, the Iranians, the North Koreans, probably Americans, the Israelis, all of them who are trying to hack into our customer sites. And you're telling me, I can't stop some nerd with a C-corporation in Palo Alto?"

It's only a single statement of defiance, but it's indubitably defiance, and from someone who can do something about it. More defiance please. Stop the malign nerds.

It's not the first good idea Cloudflare have presented in reaction to the AI-scraping age. The company also still aims to create a marketplace for site owners to sell AI model providers access to their site’s content and Prince may have been alluding to this when he mentioned another tool Cloudflare is working that will stop content scraping.

"That's the easy step, and that's coming very, very soon, and every publisher you have ever heard of is on board," he said at Cannes.

We're likely all reading some of the many SEO takes on getting your brand and content featured in AI search results at the moment. Every single one I read, no matter how skilled and insightful the SEO practitioner is, seems to me to be an account of how to thread a beautiful strand of content through the eye of a particularly jagged and uncooperative needle. Who would want to do that?

It's begging for crumbs, and begging for crumbs will make you a beggar.

A rebalancing is required if the web which has grown so wonderfully in our lifetimes is not to become creatively barren. Tools such as that built by Cloudflare supply some of the treatment required. The law, and specifically copyright law, must do much of the rest. 

We'll likely see more of these kinds of "AI baffling" tools become available in the near future. Prince can't be alone in seeing the grim prospect ahead and wanting to stop it. 

Nothing is inevitable.