Sick of AI Crawlers? Cloudflare Can Block Them or Make Them Pay You.

It sounds like a dystopian future, where sentient robotic bugs roam the Earth’s surface like a bunch of DSLR-toting tourists swarming the streets of Venice. AI crawlers aren’t quite that exciting, though they might be more infuriating.

You don’t have to look far to see complaints that AI bots have been seeking out, copying, and using people’s content—written, photographic, or anything, really—in their own AI models, without permission and without payment. From both individuals and large companies, it’s become a common refrain in just the last couple of years. That’s AI crawling.

What if you could make them pay? Or block them entirely? Content delivery network Cloudflare is testing out a new tool that can do both automatically.

three choices

“After hundreds of conversations with news organizations, publishers, and large-scale social media platforms, we heard a consistent desire for a third path: They’d like to allow AI crawlers to access their content, but they’d like to get compensated,” says Cloudflare in a blog post announcing the introduction of what they’re calling Pay Per Crawl.

“Currently, that requires knowing the right individual and striking a one-off deal, which is an insurmountable challenge if you don’t have scale and leverage.”

You don’t have to enable blocks on AI crawlers if you don’t want to. For whatever reason, from thinking AI crawlers are the best thing since sliced bread to being desperate for adulation, even from soulless robots, you can set the AI crawler permissions to free if you want to allow them unfettered access to your content.

You can also go the opposite way and block them, shutting them down with a hard no. Or, if you’re the enterprising sort, you can set your permissions to “charge.” That’ll block the AI crawlers, but let them know that if they pay a fee, they’ll be allowed access to crawl your content.

Cloudflare has revived an old error code not seen much on websites, what they call “a mostly forgotten piece of the web”: HTTP Status Code 402. The Mozilla Foundation calls it the HTTP 402 Payment Required client error response status code, which is more self-explanatory. Or at least it is until it begins turning into a run-on sentence.

Cloudflare gets all into the nitty gritty in its blog announcement, if you like to tinker under the hood of websites and know your computer lingo.

For now, Cloudflare’s 402 Code is only available for people involved in its private beta testing. If you want a possible invite to Pay Per Crawl, you can reach out to them to request it through this form.