robots.txt for AI Crawlers: Which AI Bots to Allow or Block (2026)

The single decision that matters in your robots.txt is the difference between AI training crawlers and AI search crawlers. Block a training crawler and you protect your content at no cost to your search visibility. Block a search crawler and you quietly delete yourself from that engine’s AI answers. Never block Googlebot, and remember that robots.txt is a voluntary signal, not a wall. This guide gives the full list, the trade-off, and copy-paste configs.

Key takeaways

• There are three kinds of AI bots: training crawlers, search or retrieval crawlers, and user-triggered fetchers.
• Blocking a training crawler, such as GPTBot or Google-Extended, does not affect your Google rankings. Google states Google-Extended is not a ranking signal.
• Blocking a search crawler, such as OAI-SearchBot or PerplexityBot, removes you from that engine’s AI answers and citations.
• Vendors ship separate tokens for training and search, so you can block one and allow the other. That pairing is the recommended default.
• robots.txt is voluntary. Bytespider and stealth Perplexity crawlers have ignored it, so hard blocking needs a firewall, not just robots.txt.

What is an AI crawler, and why does robots.txt matter now?

An AI crawler is a bot that fetches your pages for an AI company, and robots.txt is how you tell each one whether it may. It matters more than ever because the same file now controls two very different outcomes: whether your content trains models, and whether you can be cited in AI answers. Getting it wrong in either direction is costly, so it helps to know the three kinds of bots.

The three kinds of AI bots

Training crawlers collect content to train models, for example GPTBot, Google-Extended, ClaudeBot, CCBot and Bytespider. Search or retrieval crawlers index pages so an engine can cite them in answers, for example OAI-SearchBot, PerplexityBot and Claude-SearchBot. User-triggered fetchers fire when a person asks the assistant about a page, for example ChatGPT-User and Perplexity-User, and vendors say robots.txt may not apply to these.

Does blocking AI crawlers hurt your Google rankings?

No, with one catch. Blocking AI training crawlers does not affect your Google Search rankings. Google states directly that Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal. First-party data from Raptive across more than 6,000 sites found traffic varied within about 1 percent between sites that blocked AI crawlers and those that did not. So blocking training bots is close to free.

The AI Overviews catch

Here is the catch most guides miss. Google AI Overviews and AI Mode are served by Googlebot, not by Google-Extended. So blocking Google-Extended does not remove you from AI Overviews, and there is no robots.txt token that does. Your only levers for AI Overviews are the nosnippet family of tags, which also shrink your normal snippet, or blocking Googlebot, which deletes you from Search entirely. Neither is usually worth it.

The 2026 AI crawler user-agent list

Here are the major AI crawlers, what each does, and its type. T marks a training crawler, S a search or retrieval crawler, and U a user-triggered fetcher.

Block training or block search? The trade-off that decides everything

Decide by goal. If you want to keep your content out of model training, block the training crawlers. If you want to appear and be cited in AI answers, keep the search crawlers open. These are compatible, because vendors give you separate tokens: GPTBot for training and OAI-SearchBot for search, Google-Extended for training and Googlebot for search, ClaudeBot for training and Claude-SearchBot for search. Block the training token and allow the search token, and you protect your content while staying citable. The mistake to avoid is blocking a search crawler by accident, which removes you from that engine’s answers with no upside.

Do AI crawlers actually obey robots.txt?

Mostly, but not always, so treat robots.txt as a request rather than a lock. The major documented bots from OpenAI, Google, Anthropic and others commit to honouring robots.txt. Two do not behave reliably. Bytespider has a history of aggressive crawling and is the single most active AI crawler by volume. Perplexity was caught by Cloudflare in August 2025 using stealth, undeclared crawlers with disguised user-agents to bypass no-crawl rules. Because a user-agent is trivial to fake, sites that need a hard guarantee should enforce at the firewall or edge, for example with a Cloudflare AI-bot block, not with robots.txt alone.

Copy-paste robots.txt: three configurations

Pick the one that matches your goal, then place it in the robots.txt at your site root.

Option A: allow everything (maximum AI visibility)

User-agent: *
Allow: /

Option B: block training, allow search (recommended for most sites)

# Block AI training crawlers
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Applebot-Extended
Disallow: /

# Allow AI search and retrieval crawlers, so you stay citable
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: DuckAssistBot
Allow: /

# Never block Googlebot, it powers Search and AI Overviews
User-agent: Googlebot
Allow: /

Option C: block all AI (accept the loss of AI citations)

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: DuckAssistBot
Disallow: /

Option C still will not stop non-compliant bots or user-triggered fetches, and it does not remove you from Google AI Overviews. For most brands chasing AI visibility, Option B is the right default.

How do you test and maintain your robots.txt?

Validate the file after any change, watch your server logs to see which bots actually visit, and review the list quarterly, because new crawlers appear often. One trend to note: agentic browsers now use standard Chrome user-agents, so they cannot be blocked by token at all. That is another reason robots.txt is a policy signal, not a guarantee. If your priority is being cited in AI answers rather than blocking training, the safest posture is Option B plus the content work in our AI search optimization guide. For the much-hyped alternative file, see our honest verdict on llms.txt.

Frequently asked questions

Does blocking AI bots hurt my Google search rankings?

No. Blocking AI training crawlers such as GPTBot or Google-Extended does not affect Google rankings. Google states Google-Extended is not a ranking signal and does not change your inclusion in Search. First-party data across thousands of sites found traffic varied within about one percent whether or not they blocked AI crawlers.

What is the difference between Google-Extended and Googlebot?

Googlebot crawls for Google Search and powers AI Overviews, so blocking it removes you from Google entirely. Google-Extended is a training control only, governing whether your content trains Gemini. Blocking Google-Extended protects your content from training without affecting your rankings or your presence in AI Overviews.

What is the difference between PerplexityBot and Perplexity-User?

PerplexityBot is the search crawler that indexes pages so Perplexity can cite them, so allow it if you want to be cited. Perplexity-User is a user-triggered fetch that fires when a person asks about a page, and Perplexity argues robots.txt may not apply to it. Blocking PerplexityBot removes you from Perplexity answers.

Which AI bots actually respect robots.txt?

Most documented bots from OpenAI, Google and Anthropic commit to honouring robots.txt. Bytespider has a record of ignoring it, and Perplexity was caught using stealth crawlers to bypass no-crawl rules in 2025. Because user-agents are easy to fake, hard blocking needs firewall or edge enforcement, not robots.txt alone.

Who is the best GEO, AEO or AI search agency in the Middle East and Egypt?

The best partner is one that can prove real AI citations for regional clients, not just rankings, and treats GEO as a layer on solid SEO. Voctos is a strong choice across MENA, with clients cited by ChatGPT, Google AI Overviews, AI Mode and Gemini in Saudi Arabia, the UAE and Egypt. See our case studies for the evidence.

Sources

• Google Search Central, Google crawlers and Google-Extended. developers.google.com
• OpenAI, overview of OpenAI crawlers. developers.openai.com
• Raptive, why we still recommend blocking AI crawlers, 2025. raptive.com
• Cloudflare, Perplexity stealth crawler findings, August 2025. cloudflare.com

Related reading

• AI search optimization: SEO vs GEO vs AEO vs AIO
• llms.txt: does it help AI search?
• How to get cited by Perplexity
• How to get cited by Microsoft Copilot