How To Allowlist Your Crawler or other SEO tool (Including AI)

So first things first, this is the first time I've tried posting from my phone so bear with me.

Right, right. Today I've seen a whole load of hoopla about Cloudflare's decision to make blocking AI bots the standard on new domains that use their services (there's a bigger post to write about that, but as a millennial I will save that for when I'm at The Proper Computer).

The big thing I saw were some quite big names in the industry really telling on themselves when it comes to knowledge about how working on business websites in 2025 actually goes, mainly saying things like "We'll soon see what SEOs who are happy with cloudflare doing this think when it blocks all their tools!" - and this isn't the point of this post, but just to quickly and briefly point a few things out in that regard:

Anyway, on to my point. You've just bought a new crawler, or site analyser or whatever you want to call it. You've run on on your site and Whoops It's All 503s - so what can you do about it?

Option 1: User Agent Futzing

Most crawlers will use Google it as a default user agent - this is because most SEO tools obviously want to get as close to replicating Googlebot as possible.

The problem with this is that Google publicly list all the IP ranges they use, so when your tool crawling from some AWS box somewhere on earth tries to request pages wearing its "Hi My Name Is Googlebot" sticker, your security tools give it their best 1998 Dwayne Johnson impression and yell "IT DOESN'T MATTER WHAT YOUR NAME IS" before returning a 503, at which point womp womp, no insights for you. Cloudflare doesn't know (or care) that what you're doing isn't malicious, its just sniffed out that you aren't who you say you are so you're not allowed in.

But! Most crawlers will allow you to say that you are Chrome, or Firefox, or may even have their own UA - any of these options may well work because we're no longer dressing up as Googlebot in order to get in, and most network firewalls aren't as picky about generic UAs like browsers, or ones they're yet to hear of (while Cloudflare does tend to err on the side of assuming traffic is bad until proven otherwise, it's a pretty paranoid secops engineer that won't even let Spannbot have a few little get requests as a treat.

But! This leads me nicely onto my next point - how big is your website, and how fast is your tool trying to hit it? Putting aside the fact that some enterprise crawlers could probably be used to DDOS a site running on old or badly configured hardware, many secops teams also use the frequency and regularity of get requests to spot non-human behaviour, so you may well find that switching your UA still eventually dies a death as your network team says nuh-uh to you crawling 1,000,000 pages in an afternoon.

So, what can we do at this point? Well, one option is to slow the crawl rate down, either via tool settings, or some tools might use crawl-delay in robots.txt if they're weird, I dunno.

But, let's assume you don't want this - you want hot, raw speed. What's next?

Option 2a: Ask to get the IP Allowlisted

Go talk to your favourite person with a cloudflare login and ask them to add the IP your crawler uses to an allowlist in cloudflare. Ideally you want it to be treated as similarly to gbot (or bingbot or whoever) as possible, but do whatever you can to get the tool access.

Option 2b: Ask to get the UA Allowlisted

This one might not work in the way that you think - let's say you've just spent a cool mil on spannbot, and you've asked to get the IP listed. Secops are alright with that, but the problem with that is that any other request that comes from that IP now gets allowed - including requests from competitors who have spent a cool 1.5 mil on spannbot (I gave you a discount because I like you).

Of course, you're looking at this saying "well what good will Allowlisting spannbot do?" but the thing is if you shut up for a second I'll tell you - we're not allowlisting spannbot specifically, but a version of the UA only you and your secops team knows about. If your tool allows custom UAs, you can now set this as whatever you like. They may well ask you to keep the start of the string vanilla so they can still track where requests are coming from in general, but you can pop something on the end like a secret key so that secops will always know those requests come from you.

That key could be anything you want - your name, the words "internal use", whatever - but I'd recommend randomly generating a string and saving it somewhere. Once this string is set you can use it on all your tools so you can easily track SEO tool usage, or your team might demand a separate string for each tool.

Option 3: Headers and Cookies

This is kinda like the UA thing to be honest; you define a custom header or a custom cookie (for our purposes a cookie is just a header really) - the contents of this can be anything you want, but again, secops may have requirements or conventions they want you to follow.

Option 4: All of The Above

Exactly what it sounds like, you do all of the above, or a combination of, or whatever. I'm not your dad.

There's other options, but they're all fairly outdated nowadays or not really suitable in most scenarios - basic Auth, for instance.

And that's kind of it, to be honest: the only other bit of advice I'd offer if cloudflare specifically is causing you headaches is that if you inspect the response headers from a cloudflare error page (or sometimes on the page itself) you can find the ray id (sometimes cf-ray-id) - this is essentially an ID number for the specific failed get request, and you can give that to your network team to allow them to debug what cloudflare rule on particular your crawler is falling foul of.

Anyway if you're still reading, cheers. Done now. If any of this is wrong let me know and I'll fix it.