AI VS The Open Web

I run a sandwich shop. Every so often a man, let's call him George Oogle, comes to my sandwich shop and I let him have a sandwich. It costs me money, sure, but it's worth it because thousands of people a day ask George where to get a sandwich, and he tells them about my shop.

The system works.

But, all of a sudden Mr Oogle has started behaving differently. He still comes and has a free sandwich, but now when people ask him for a sandwich he produces his own from his pocket, using bits of my sandwich, bits of another place's sandwich, and for some reason a bit of a Caesar Salad recipe, for reasons I can't be sure of.

Sure, if you really interrogate him he'll reluctantly tell you where he got all the bits of his weird Frankenstein sandwich, but he would rather you took HIS sandwich, sandwiches he has an infinite supply of, as they're made of other peoples hard work.

Anyway, people tell me this is good and that I should be thankful to George, and I should just keep letting him eat my sandwiches before vomiting them into the hands of people who ask where they can get one.

I hope this clever analogy hasn't been too obscure for you!!!

This post is a response to people blaming Cloudflare for killing the open web with their default AI blocking - something that I don't want to call victim blaming, but something that is definitely akin to that.

Don't get me wrong, I love the open web - you can find thousands of words on this very website written by me defending RSS, and I believe the internet should be an open, community based piece of infrastructure that serves to enrich the lives of everyone.

But let's be honest, the open web has been dying for a long time in various ways, some fast, some slow. Every single app on your phone wants your permanent and undivided attention, whether that's through a never ending stream of videos and posts selected by an algorithm that makes it hard to see what your friends and family are doing and actively punishes including links, push notifications giving you 5% off if you make an order RIGHT NOW, AI overviews that try to keep you from clicking away from Google, or ads that follow you around the web trying to get you to buy anything and everything you've ever had a passing interest in.

Your concentration is a resource that the hyperscalers are all trying to mine; the technology behind the web may remain somewhat open, but the companies who owe their existence to that tech have been desperate to make you their prisoner for years now.

And let's talk about the technology behind the open web - the idea being that the web is not a maze of paywalls, logins and barriers, but an eminently accessible and ever growing resource; a repository of human knowledge, an enormous library that contains all of human endeavour.

Does that sound like the web you know? Everyone wants you to login via Google nowadays so they can send you emails. News sites have paywalls because it's the only feasible model. Podcasts and videos and pictures and games exist behind Patreon subscriptions; the open web dies a little more every day that someone else finds a new way to monetise it.

Which brings me on to LLMs. LLMs realised they could monetise the web by just crawling all of it, chewing it into a big amorphous white bread blob, then flobbing bits of it into the hands of whoever wants to pay $200/mo for an unlimited subscription, thus starving the websites who rely on traffic and ad impressions to generate money - and weirdly, right, some of those websites didn't like it, so Cloudflare came up with an answer - the ability to block AI bots by default.

Now, I've already yammed on at length about why that isn't actually anything new, Cloudflare have been blocking dodgy looking traffic to websites for years; it's part of their core business model (I work for a SaaS crawler, believe me I know all about it), this is merely cloudflare saying that unless you tell them otherwise, they will treat this traffic as detrimental to your business, or at least ask the crawlers to pay for it in order to redress the balance of power in the exchange you're making with them.

And this is the point, right: The open web was well and good when we could reasonably assume that Google would link to us in the SERPs. The open web was great when a single technology would DDOS sites in the name of glomming every bit of information it possibly could in order to make itself more powerful. The power dynamic is fucked, and as content creators and site owners we are expected to work to create "great content", only for it to then be used by genAI or the usual suspects in whatever manner they so please, either in algorithmic feeds or GenAI responses. Hell, even Claude struggles to argue for the ethics of its own existence: enter image description here

Essentially, the open web was great until big business made it so totally unfair and then told us we should be grateful for it. Google has been sticking toes over the line for years, but they at least offer some level of control over how and when they crawl, and try to do so somewhat elegantly - meanwhile Claudebot is routinely so aggressive it DDOS's websites - who get what exactly in response?

So, fuck them.

A ton of people with "AI" in their job title will tell you that blocking LLMs is commercial suicide - this is the future! Don't appear inside LLM clients and you're stuck in the past - but unless you sell a physical product, and make money by clicking on your shop and buying, the referral traffic from AI is pretty much worthless -and even if you are selling an actual product, I'm yet to see any real data that says that tiny trickle of traffic is better than old fashioned search clicks in terms of conversion. We're currently in a place where Boosterism says "AI will be the future, these sites need to suck it up and deal with the fact that this is how things work now" and cool, ok but, if the sites that Claude is slurping the information from cease to exist, where is Claude going to slurp the information from in the future?

The other day Sundar Pichai said “There’ll be a marketplace in the future, I think—there’ll be creators who will create for AI [...] People will figure it out.” - Well guess what, people have figured it out already, but I get the feeling they've figured it out in a way that most of the big GenAI providers won't particularly like.

You might say that this competes with Robots.txt; that Robots works on the assumption that everything is fair game unless specified otherwise, but this new Cloudflare change assumes everything is to be locked away and that having both means we have competing standards, but I'll just show you this:

enter image description here

The open web, to all intents and purpose is already dead. Everything is monetised. Everything is paywalled. Everything is rate limited, every bit of free technology the open web was built on has had gates put in front of it. What Cloudflare are doing isn't killing the open web, they're trying to keep the web open for those who are willing to operate as a society.