Most audits of AI crawler access start and end with robots.txt. That file only matters if the request reaches your server first. A WAF rule or Cloudflare setting from 2023, built to stop scrapers, can reject GPTBot, ClaudeBot, and PerplexityBot before robots.txt is ever read. Nothing in robots.txt shows this; only your edge logs do.
Why AI Crawler Access Set in 2023 Still Matters Now
Named AI tokens arrived fast in 2023. OpenAI documents GPTBot as the crawler that "may be used in training our generative AI foundation models," live since August that year. Google added its Google-Extended token on 28 September 2023, at launch controlling whether a site helped improve Bard and Vertex AI.
According to Consent in Crisis, a 14,000-domain audit of AI training corpora, restrictions arrived just as fast, within that same 2023 to 2024 window. 28% or more of the most actively maintained and critical sources in the C4 training corpus became fully restricted from use. The rules that still bite today were often written as categories, not names. A robots.txt disallow rule written for a wildcard `User-agent: *` group catches every crawler with no named rule of its own. That includes agents that did not exist yet when the rule was written.
That is the trap. OAI-SearchBot, one of the crawlers that would actually get a brand cited today, did not exist until 2024, and Claude-SearchBot did not exist until 2025. A wildcard disallow written in 2023 to keep AI training crawlers out catches both anyway, because it never named a crawler at all.
Where the Block Actually Lives
A crawler can be stopped in two places, and only one of them shows up when you check robots.txt. The file itself is advisory: it tells a crawler what it should not fetch, but a compliant bot has to read and honor it. A WAF rule, a CDN firewall setting, or a bot-management toggle works differently. It rejects the request at the edge, before your server or your robots.txt file ever enters the picture.
Cloudflare is the clearest example, because it is where most of this drifted. According to Cloudflare's own announcement, the company shipped a one-click toggle for AI scrapers and crawlers on 3 July 2024. Cloudflare reported a year later that more than one million customers had chosen it. That same report describes a second change. Every new domain signing up with Cloudflare was asked upfront whether to allow AI crawlers, with block as the default. The defaults moved again on 15 September 2026. According to Cloudflare's docs, new domains now block bots classed as Training or Agent on pages that show ads, and leave Search bots allowed. Any setting that blocks training also blocks mixed-purpose crawlers, the ones Cloudflare classes as both Search and Training. Older, unrelated settings compound the problem. Bot Fight Mode has existed since long before any named AI crawler, and it blocks traffic that behaves like an automated script. An AI crawler fetching pages on a schedule looks exactly like the traffic it was built to catch.
The Nine Crawlers to Check
This table is built to forward to whoever owns your WAF or CDN account. Check each token against your robots.txt file and your edge logs separately, since a block can exist in one place and not the other.
| Crawler | Operator | What it does | Trains AI models |
|---|---|---|---|
| GPTBot | OpenAI | Gathers training data | Yes |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results | No |
| PerplexityBot | Perplexity | Surfaces sites in Perplexity answers | No, per Perplexity's own documentation |
| ClaudeBot | Anthropic | Gathers training data | Yes |
| Claude-SearchBot | Anthropic | Improves Claude's search and answer quality | No, per Anthropic's own documentation |
| CCBot | Common Crawl | Builds the open web archive many labs train on | Indirectly |
| Google-Extended | Controls Gemini training and grounding; has no user agent of its own and does not affect Google Search ranking | Yes | |
| Bytespider | ByteDance | Gathers training data for ByteDance's own models | Yes, and ByteDance does not publish crawler documentation comparable to OpenAI's, Anthropic's, or Perplexity's |
| Amazonbot | Amazon | Improves Amazon products and services, including AI models | Yes |
A category rule written to reject "AI bots" as a class does not distinguish training from citation. It catches the search and citation crawlers too, whether or not they existed when the rule was written.
Amazon documents further agents outside this nine-token checklist worth knowing about. Amzn-SearchBot is one Amazon states explicitly "does not crawl content for generative AI model training." It makes a site eligible to appear in search experiences like Alexa, and blocking Amazonbot for training reasons should not mean blocking this one too.
How to Audit Your Own AI Crawler Access
A robots.txt check alone cannot tell you whether a crawler actually reaches your site, and neither can a WAF check alone. Run all four steps below in order. Each one catches what the step before it cannot see.
- *Open robots.txt and search it for all nine tokens above, then read the `User-agent: ` group.** A crawler with no named rule of its own falls under that wildcard group. A broad disallow there blocks agents the file never mentions by name. Either way, this tells you nothing about your WAF.
- Check your WAF or CDN dashboard separately. On Cloudflare, go to Security Settings, then Configure AI bot policies, and read what the Search, Agent and Training presets are set to. Check whether the older Block AI bots toggle is still on, and whether any managed ruleset predates 2024. Then open AI Crawl Control and look for Block in the Action column of the Crawlers tab. Cloudflare documents the policy presets and AI Crawl Control separately.
- Pull 403 responses from your edge logs for each user agent, over the last 30 days. This is the only step that shows a block robots.txt cannot reveal. Google-Extended is the exception. It has no user agent string, so it will never appear in your logs. Robots.txt is the only place it can be set at all.
- Split the fix list by who owns it. A robots.txt edit is usually yours to make directly. A managed WAF ruleset on an enterprise CDN account often needs a ticket to whoever owns that account. That is frequently a central IT or security team at headquarters rather than the person running the brand's own site.
What to Allow and What to Keep Blocking
The nine crawlers split into two jobs, and the split decides what you actually want blocked. GPTBot, ClaudeBot, Google-Extended, Bytespider, and Amazonbot exist to train models. CCBot feeds that same training pipeline indirectly, through the open datasets it builds. OAI-SearchBot, PerplexityBot, and Claude-SearchBot exist to answer a live question with a citation, which is the traffic GEO work depends on.
Whether to allow training crawlers is a policy call for your legal or leadership team, and we do not make it for you. What we will say is that the search and citation crawlers in that second group are worth confirming open. Do not assume a blanket Block AI bots toggle left them alone. Cloudflare's own docs say a block on training also catches crawlers it classes as both Search and Training.
Passing this audit only means a crawler can reach your page. Our guide on how AI crawlers read your site covers what happens next. Our guide on why accordions hide content covers the layer after that. There, the page loads, but the content never renders into the raw HTML a crawler reads. Our Dual-Layer Website Architecture is built for that rendering problem, and it cannot help while an edge rule is still rejecting the request.
FAQs
Does robots.txt alone tell me whether AI crawlers can reach my site?
No. Robots.txt only shows what you have told a compliant crawler not to fetch. A WAF rule or CDN bot-management setting can reject the same crawler before your server or robots.txt file is ever reached. None of that shows up in the file.
Which AI crawlers should a brand doing GEO work keep open?
OAI-SearchBot, PerplexityBot, and Claude-SearchBot are the three built to surface and cite your content in a live answer. Blocking them directly works against AI search visibility, whatever your position is on the training crawlers.
How do I check whether Cloudflare is blocking AI crawlers on my site?
Open Security Settings, then Configure AI bot policies, and read the Search, Agent and Training presets. Check whether the older Block AI bots toggle is still on. Then open AI Crawl Control and look for Block in the Action column of the Crawlers tab. Check the date on any managed ruleset that predates July 2024, when the first one-click AI block shipped.
Is it safe to allow Bytespider?
That is a judgment call. ByteDance does not publish crawler documentation comparable to OpenAI's, Anthropic's, or Perplexity's. According to Fortune's reporting on research from Kasada, Bytespider has also crawled pages disallowed in robots.txt. That puts it in a different trust category from the other eight.
Who should fix a blocked crawler at an enterprise company?
It depends on where the block lives. A robots.txt edit is usually within reach of whoever runs the site. A managed WAF ruleset on a centrally administered CDN account is typically an IT or security team's call. That is the fix worth naming explicitly when you forward this audit internally.
Getting a Second Opinion
A 2023-era block is easy to miss because nobody is looking for it. It was never in a sprint plan, and it does not show up in a content audit. The person who set it may not work there anymore. We can run this check against your own setup and hand you a list split the same way. Book a free AI visibility assessment to get started.