Are AI crawlers blocked on your site? How to tell, and how to let them in without opening the door to everyone.
A block on AI crawlers almost never lives in robots.txt. It lives in a CDN toggle somebody flipped during a security scare, or in a hosting firewall nobody can see. Here are the crawlers that decide whether ChatGPT, Claude, Perplexity, and Google's AI answers can cite you, a ten-minute check, our own crawl logs showing what a block looks like, and the fixes that keep the bad traffic out.
Where the block livesThree layers
Sites that turn AI crawlers awayPer 100 scanned
One week on growthr.comOct 13 to 19The short answer
Request your homepage with an AI crawler's user-agent and see what comes back. A 200 with your real HTML means that layer is open. A 403, a challenge page, or an empty 429 means something between the crawler and your pages said no, and the something is usually not robots.txt.
Three layers can refuse a crawler, and they fail differently. robots.txt is a request that well-behaved crawlers honour; it blocks nothing by itself. CDN bot management (Cloudflare, Akamai, Vercel's firewall) is a wall that matches on user-agent or on a fingerprint of the client, and several products ship with an "AI bots" switch that was on by default for a while. The origin host's firewall is a second wall behind the first, with its own IP-reputation list, and it is the one nobody checks because it is not in any dashboard you log into.
When we scanned 640 company sites in September, 100 of the 591 we could measure turned away a request naming an AI crawler, and only four of those had written the decision down in robots.txt. The rest were defaults nobody chose. The write-up is in Which industries can AI read?; this page is the fix.
The crawlers that matter, and which answer each one controls
Each AI company runs more than one bot, and they do different jobs. Blocking the training crawler and blocking the search crawler are different decisions with different consequences, so the first step is knowing which name does what.
| User-agent | Who | What it does | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT's search features | Your site is not shown in ChatGPT search answers, per OpenAI's docs |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT something that needs it | OpenAI says robots.txt rules may not apply, because a person triggered the fetch; a firewall block still stops it |
| GPTBot | OpenAI | Collects content for training foundation models | No effect on whether you appear in ChatGPT search; it is the training opt-out |
| Claude-SearchBot | Anthropic | Indexes for Claude's search quality | Reduced visibility and accuracy in Claude's search results, per Anthropic |
| Claude-User | Anthropic | Fetches a page a Claude user asked about | Claude cannot read your page for that user |
| ClaudeBot | Anthropic | Collects content for model development | Training opt-out; Anthropic says its bots honour robots.txt and publishes its IPs at claude.com/crawling/bots.json |
| PerplexityBot | Perplexity | Indexes pages to surface and link in Perplexity answers; not used for training | You stop appearing as a Perplexity source |
| Perplexity-User | Perplexity | Fetches a page a user asked about | Perplexity says it generally ignores robots.txt for user-requested fetches; a firewall block still stops it |
| Googlebot | Indexes for Search, which feeds AI Overviews and AI Mode | You are out of Google, overviews included. Google-Extended is the separate Gemini training control | |
| Applebot, Amazonbot, Meta-ExternalAgent, Bytespider, CCBot | Apple, Amazon, Meta, ByteDance, Common Crawl | Siri and Apple Intelligence, Alexa, Meta AI, ByteDance's models, the Common Crawl dataset | Each is a separate decision; Common Crawl is what many smaller models train on |
Sources: OpenAI's crawler documentation, Anthropic's crawler article, Perplexity's bot guide, all read October 19, 2026. Each publishes the IP ranges its bots use; a request with the right user-agent from an IP outside the list is somebody else.
The ones that decide whether you get cited are the search and user-fetch bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, and Googlebot. The training crawlers are a policy choice with no effect on citations. Most "block AI" switches do not make that distinction, which is how a site ends up invisible to ChatGPT search while believing it only opted out of training.
How to tell, in ten minutes
Test the behaviour, not the file that describes it. A robots.txt that allows GPTBot means nothing if the CDN returns a 403 before the request reaches your server.
In this order0 / 6- Run the free scan. willaiseeme.com requests your homepage as GPTBot and ClaudeBot from a datacenter IP and reports the status code. Two minutes, no signup.
- Repeat it yourself with curl, once from your laptop and once from a server or a cloud shell. A 200 at home and a 403 from the cloud means the block is on the IP's reputation, not the user-agent.
- Read robots.txt by group, not by eye. A list of twenty AI user-agents stacked above
User-agent: *with no blank line is one group sharing the*rules; without aDisallow: /it blocks nothing. Squarespace ships exactly this file, and it is misread as a blocklist constantly. - Open your CDN's bot settings. On Cloudflare it is Security, then Settings, then "Configure AI bot policies". Three rows: Search, Agent, Training. Anything on Block is a wall.
- Pull a week of edge logs filtered by user-agent. Group by status. A bot with 2xx on cache hits and 403 on cache misses is being blocked by the origin, behind the CDN.
- Verify the IPs before you act. Compare the source addresses against the published lists. Scanners borrow Googlebot's and GPTBot's names to probe for
.envandwp-config.php; those should stay blocked.
The curl version of step two, with the user-agent strings as the vendors publish them:
Terminalfor ua in "GPTBot" "OAI-SearchBot" "ChatGPT-User" "ClaudeBot" "Claude-User" "PerplexityBot" "Googlebot"; do
printf "%-16s " "$ua"
curl -s -o /dev/null -w "%{http_code} %{size_download}B\n" \
-A "Mozilla/5.0 (compatible; $ua/1.0; +https://example.com/bot)" \
https://yourdomain.com/
done
Read the size as well as the status. A 200 with 1,400 bytes is a challenge page or an empty shell, which fails the crawler just as surely as a 403. Then run the same loop from a cloud shell. Bot-management products fingerprint the client through its TLS handshake and header order, so curl can be challenged on a site that serves real browsers fine; if every user-agent fails from every network, you are looking at client fingerprinting rather than an AI-crawler rule, and the vendor may still allowlist the real bots by IP in a way you cannot see from outside. Your own edge logs settle it in a minute.
What a block looks like in the logs: our own
We found two of these on growthr.com this year, and neither was in robots.txt. The logs below are a week of Cloudflare edge requests, grouped by user-agent and status, with scanner probes under bot names removed.
Three things in that chart are worth knowing how to read.
- PerplexityBot is at 36 of 38 served, and in September it was 24 of 56 refused. The cause was the hosting provider's IP-reputation firewall, which returned 403 to Perplexity's AWS addresses whenever Cloudflare's cached copy had expired. Same bot, 200 on a cache hit and 403 a minute later. robots.txt allowed it; Cloudflare allowed it; the server behind both did not. The fix was adding the 18 addresses from perplexity.com/perplexitybot.json to the host's allow list on September 4, and the 403s went to zero.
- Amazonbot is still losing 9 of 42 to the same firewall. That is the next allow-list entry. A 403 whose cache status reads
missorexpired, with no matching event in the CDN's security log, is the signature of an origin block. - Meta-ExternalAgent's 50 rate limits are mostly not Meta. Twenty-seven of them were requests for
/compose.yaml, which no crawler needs. Something is probing the site under Meta's user-agent, and the rate limit is doing its job. This is why step six, verifying IPs, comes before any fix: a bot name in a log is a claim, not an identity.
The other block we found this year was Cloudflare's own. In August, every AI crawler on the list got a plain-text 403 from the edge: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Amazonbot, Bytespider, all of them, including on /llms.txt itself. Googlebot and plain curl got 200. That is the fingerprint of the "Block AI bots" toggle, which matches on user-agent and ignores robots.txt entirely. It had been turned on during a DDoS in July along with everything else. Nothing in the site's files changed; the GEO work we had done was unreachable by the engines it was for.
How to let them in without opening the door
Allow the named, verified crawlers by category. Keep the rate limit, the managed WAF rules, and the scanner blocks exactly as they are. The two are separate controls, and allowing AI bots does not weaken any of the rest.
On Cloudflare
Security, then Settings, then search for "AI bot". The panel has three rows. Search covers crawlers that index your content to answer questions later (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Agent covers bots acting in real time for a person (ChatGPT-User, Claude-User, Perplexity-User). Training covers crawlers collecting content for models. Set Search and Agent to Allow if you want to be cited; set Training to whatever your policy is. Cloudflare labels Allow as the recommended option on all three.
Two dates matter here. Cloudflare's documentation says that since September 15, 2026, new domains get updated defaults: Training and Agent bots blocked on pages that display ads, Search allowed. If your zone was created after that date and you run ads, Agent is blocked out of the box, which means ChatGPT-User and Claude-User cannot read the page a user asked about. The same note says mixed-purpose crawlers are now blocked under every configuration meant to block training, so a "training only" block catches more than it used to. Check the panel rather than assuming.
At the origin, behind the CDN
Shared hosting and managed WordPress hosts run their own IP-reputation firewalls, and they are not in the control panel you normally use. The tell is in the logs: 403s that only occur on cache misses, with no event in the CDN's security log. The fix is an allow list of the crawler's published addresses. On Hostinger it is the IP Manager under Advanced; the field takes one IPv4 address at a time and no CIDR ranges, so the 18 Perplexity addresses were 18 entries. The vendors rotate those lists occasionally, so when 403s reappear, re-read the JSON and diff it.
Some hosts answer a crawler's user-agent with a 429 on every uncached request regardless of IP. We found that with GPTBot on the same server: real GPTBot traffic was mostly served from Cloudflare's cache and never saw it, but every cache miss did. A Cloudflare request-header transform rule that rewrites the user-agent to a browser string for requests matching the bot, before they reach the origin, fixed it without a support ticket.
In robots.txt
Name the bots you want and give them an explicit Allow: /. A named group ignores the * block entirely, so anything you want them to inherit has to be repeated in their group. Keep the file as the statement of policy, and the firewall as the thing that enforces it; they drift apart when nobody checks, and the firewall always wins.
What to keep blocked
- Everything hitting
/.env,/wp-config.php,/.git, and friends, whatever user-agent it claims. Our origin firewall stops around 16,000 of those a week. Leave it on. - The per-IP rate limit. Real crawlers pace themselves and never trip a limit of 20 requests in 10 seconds. Audit tools do, which is why they report "blocked by WAF" on sites that are open.
- Unverified clients using a verified bot's name. Cloudflare's Verified Bots list and the vendors' IP files are the check. A GPTBot request from a residential ISP is not GPTBot.
Frequently asked questions
How do I know if AI crawlers are blocked on my site?
Request your homepage with an AI crawler's user-agent string, from your own machine and from a cloud server, and check for a 200 with your real HTML. A 403, a challenge page, or a tiny response means a block. Then check three places: robots.txt, your CDN's bot settings, and your hosting provider's firewall. The free scan at willaiseeme.com runs the first test for you.
Does blocking GPTBot stop me appearing in ChatGPT?
No. OpenAI's documentation says GPTBot is the training crawler and that disallowing it does not affect whether you appear in ChatGPT search. OAI-SearchBot controls search results, and ChatGPT-User fetches pages a user asks about. Blocking those two is what removes you from answers.
Does Cloudflare block AI bots by default?
It depends on when the zone was created and what was toggled. Cloudflare's documentation says that from September 15, 2026, new domains block Training and Agent bots on pages that display ads and allow Search. Older zones may still have the legacy "Block AI bots" switch on. Check Security, then Settings, then "Configure AI bot policies", and set Search and Agent to Allow if you want to be cited.
My robots.txt allows AI crawlers but they still get a 403. Why?
robots.txt is a request; a 403 is a wall, and it sits in a CDN bot rule or in your hosting provider's IP firewall. If the 403s only happen on cache misses and your CDN's security log shows no matching event, the origin host is blocking the crawler's IP addresses. Add the vendor's published addresses to the host's allow list.
Want us to find the block for you?
The $99 strategy call is a live teardown of your site: we run the crawler checks, read your edge logs with you, and leave you with the fix list. The fee credits toward Boost if you want it implemented.
Related: Which industries can AI read? · Does llms.txt actually work? · GEO agency services · SEO + GEO services