Last updated: October 5, 2026 · 3 min read · By Ali Raza
AI companies run different crawlers for different jobs. Search crawlers such as OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot help your pages appear and be cited in AI answers; blocking them reduces visibility. Training crawlers such as GPTBot and ClaudeBot collect content for model training; blocking them is a separate choice. Google-Extended controls Gemini training and grounding but does not affect Google Search. Most businesses that want visibility allow search crawlers.
Many websites block AI bots without realising it, through security plugins, CDN bot settings or a robots.txt copied from somewhere else. Others want to block training but keep visibility. The key is knowing which bot does what.
Major AI crawlers (October 2026)
| Company | User agent | Purpose | Block it and… |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT search | Not shown in ChatGPT search answers |
| OpenAI | GPTBot | Model training | Content excluded from training |
| OpenAI | ChatGPT-User | Visits pages when a user asks | robots.txt may not apply (user-initiated) |
| Anthropic | Claude-SearchBot | Search result quality for Claude | May reduce visibility in Claude |
| Anthropic | ClaudeBot | Model training | Content excluded from future training |
| Anthropic | Claude-User | Fetches pages for user questions | May reduce visibility in user requests |
| Perplexity | PerplexityBot | Perplexity search results (not training) | Not surfaced in Perplexity search |
| Google-Extended | Gemini training and grounding | No effect on Google Search ranking or inclusion |
Which bots should you allow?
If you want to appear in AI answers, allow the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and of course Googlebot and Bingbot. Whether to allow training crawlers is a business decision about how your content is used; it is separate from search visibility.
An example robots.txt
This example allows AI search crawlers and blocks training crawlers. Adjust it to your own decision:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap_index.xmlWhy might bots still be blocked?
- CDN and firewall settings: some services offer one-click “block AI bots” options that may include search crawlers.
- Security plugins that rate-limit or block unknown user agents.
- Hosting limits that return errors under crawl load.
Check your server logs for these user agents and their response codes. Anthropic publishes the IP ranges of its crawlers so you can verify genuine requests.
What about user-initiated fetchers?
Agents such as ChatGPT-User and Perplexity-User visit pages because a person asked a question. Their providers say robots.txt may not apply in the same way, since the fetch is user-requested.
Does llms.txt help?
llms.txt is a proposed file that summarises a site for AI tools. It is harmless to add, but Google states you do not need new machine-readable files or AI text files to appear in its AI features. Focus first on crawlable pages and clear content.
Crawler access is step one in our guide to getting recommended by AI assistants. See also what GEO and AEO are, or ask our AI search optimization team to check your settings.
Frequently asked questions
Should I block AI crawlers on my website?
If you want visibility in AI answers, allow AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking training crawlers like GPTBot and ClaudeBot is a separate decision.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot is used for ChatGPT search; sites that block it will not be shown in ChatGPT search answers.
Does blocking Google-Extended affect my Google rankings?
No. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
Do I need an llms.txt file?
It is optional. Google says no special AI text files are needed for its AI features; crawlable pages and clear content matter more.