According to Vercel's December 2024 analysis of crawler traffic across its network, none of GPTBot, ClaudeBot or PerplexityBot executed JavaScript. All three read only the raw HTML a server sends, with nothing added by a browser. Each company behind these bots also runs a user-triggered fetcher. Which ones you allow decides what an AI assistant can read when someone asks about your brand.
What GPTBot, ClaudeBot, and PerplexityBot Actually See
All three read a page the way `curl` does, without the scripts a browser runs. Vercel counted 569 million GPTBot requests across its network in one month, and none executed JavaScript. ClaudeBot and PerplexityBot behaved the same. Googlebot, Gemini and AppleBot do render JavaScript, so a page Google indexes cleanly can still be missing content for these three. Why ranking on Google doesn't carry over to other engines covers that gap in detail.
That gap matters because raw HTML can be thinner than what a visitor sees. Content that loads after hydration, behind a JavaScript framework, or inside a client-rendered review widget is invisible to a bot that never runs a script. Why accordions hide content from AI crawlers covers one common pattern. How client-side rendering hides a whole page covers the whole-page version.
Meet the Bots: What Each One Is Actually Built to Do
Each company runs more than one crawler.
| Bot | Company | Job | Follows robots.txt (per vendor docs) |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may train future models | Yes |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search results | Yes |
| ChatGPT-User | OpenAI | Fetches a page live when a user asks | May not apply (user-triggered) |
| ClaudeBot | Anthropic | Collects web content that may inform training | Yes |
| Claude-SearchBot | Anthropic | Navigates the web to improve search results | Yes |
| Claude-User | Anthropic | Fetches a page when a user directs Claude to it | Yes |
| PerplexityBot | Perplexity | Surfaces and links sites in search results | Yes |
| Perplexity-User | Perplexity | Fetches a page live when a user asks a question | Generally ignores it |
Disallowing GPTBot in robots.txt tells OpenAI not to use your content for training. It doesn't touch OAI-SearchBot, a separate crawler for ChatGPT's search feature. Anthropic runs the same split. Perplexity's own documentation says Perplexity-User generally ignores robots.txt, since a live person triggered the fetch. OpenAI says the same rules "may not apply" to ChatGPT-User for the same reason. Anthropic's Claude-User is the exception: it still honors robots.txt even though a user triggered it.
Blocking only the training crawler opts your future content out of that company's training. Blocking the search bot too keeps you out of ChatGPT's search answers, and Anthropic says it may reduce your visibility in Claude's. That's the citation you actually wanted.
Blocking Isn't Always Written to robots.txt
robots.txt is the visible layer, but firewall rules, bot-management defaults and rate limiting can reject a crawler with nothing written there. The AI crawler block you set and forgot about covers that robots.txt, WAF and edge-rule audit in full.
We check every named crawler individually during a technical audit, because an edge rule doesn't show up in robots.txt. What a GEO audit covers includes this exact check.
How to Check What These Bots See on Your Site
- Fetch your own page in raw HTML and compare it against what loads in a browser. Anything missing from the fetch is invisible to GPTBot, ClaudeBot, and PerplexityBot alike.
- Check robots.txt for each bot by name. A wildcard rule can hide a specific block, and a missing rule doesn't exclude a firewall block sitting somewhere else.
- Test a page whose main content depends on JavaScript. That's where the raw-HTML gap actually shows up.
- Confirm the page shows up when you ask an assistant about the topic. The fetch test alone doesn't confirm that. Testing your brand with real prompts covers how to run that check properly.
We cover these checks in our free AI visibility assessment. Book a call on our website and our experts will audit your site. We'll tell you what GPTBot, ClaudeBot and PerplexityBot can and can't read.
FAQs
Do GPTBot, ClaudeBot, and PerplexityBot render JavaScript?
No. According to Vercel's December 2024 crawler analysis, all three read raw server HTML only. Content that appears solely after JavaScript runs is invisible to them.
Does blocking GPTBot also block ChatGPT's search results?
No. GPTBot and OAI-SearchBot are separate crawlers with separate jobs. Disallowing GPTBot opts your content out of training; it does nothing to OAI-SearchBot.
Why would a site block one of these bots without meaning to?
Firewall rules, bot-management defaults and rate limiting can reject a crawler with nothing written to robots.txt. These edge-level blocks are easy to miss during a normal SEO check.
Do the "user" versions of these bots respect robots.txt?
It depends on the company. Perplexity says Perplexity-User generally ignores it, and OpenAI says robots.txt rules may not apply to ChatGPT-User. Anthropic's Claude-User honors it.
How do I know if my content is actually crawlable?
Fetch a page in raw HTML and compare it to what a browser shows. Anything present in the browser and missing from the fetch is content these bots cannot read.
The Bottom Line
As of Vercel's December 2024 data, GPTBot, ClaudeBot and PerplexityBot read raw HTML only. Each company also runs a fetcher for live user requests, which follows robots.txt at Anthropic but may not at OpenAI or Perplexity. A raw-HTML fetch test tells you what the crawlers actually see.