freeseobot

robots.txt for AI Crawlers: GPTBot, ClaudeBot and More

Which AI crawlers to allow or block in robots.txt, what each one does, and copy-paste rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google.

By freeseobot Editorial · Published · 6 min read

For most businesses the sensible default is: allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and of course Googlebot), because they decide whether you can be cited in AI answers, and make a separate, deliberate choice about the training crawlers (GPTBot, ClaudeBot, Google-Extended). Each company documents its bots separately, so you can block training without disappearing from AI search.

Below is what each crawler does, the one robots.txt rule that trips most people up, and copy-paste examples for every common setup.

What is robots.txt and what can it actually do?

robots.txt is a plain text file at the root of your domain (https://example.com/robots.txt) that tells crawlers which paths they may request. A crawler is a program that downloads web pages automatically.

Three limits to understand:

  • It controls crawling, not indexing. Google states that a page blocked in robots.txt can still be indexed if other pages link to it. To keep a page out of Google, use noindex and let Google crawl the page so it can see that tag.
  • It works per host. A robots.txt on example.com does not cover shop.example.com. Anthropic's documentation reminds you to add rules for every subdomain you want to opt out.
  • It is a request, not a lock. Reputable crawlers follow it. User-triggered fetchers may not (details below).

Which AI crawlers exist and what does each do?

User-agent token Company Purpose Blocking it means
GPTBot OpenAI Collects content that may train OpenAI's foundation models Your content should not be used for training
OAI-SearchBot OpenAI Surfaces sites in ChatGPT's search features You are left out of ChatGPT search answers, though you may still appear as a navigational link
ChatGPT-User OpenAI Visits a page when a user's request needs it OpenAI says robots.txt rules "may not apply" because a user started the action
ClaudeBot Anthropic Collects content that could contribute to model training Signals your future content should be excluded from training
Claude-SearchBot Anthropic Indexes content to improve Claude's search results May reduce your visibility in Claude's search results
Claude-User Anthropic Fetches pages when a person asks Claude something Claude cannot retrieve your page for user questions
PerplexityBot Perplexity Surfaces and links sites in Perplexity results; not used for foundation model training You stop appearing as a Perplexity source
Perplexity-User Perplexity Fetches pages for a user's question Perplexity says it "generally ignores robots.txt rules"
Google-Extended Google A control token, not a separate crawler. Covers use for Gemini model training and grounding in Gemini Apps and Vertex AI Does not affect Google Search inclusion or ranking
Googlebot Google Crawls for Google Search, including AI Overviews and AI Mode You drop out of Google Search entirely

Anthropic states that all of its bots honour robots.txt and also support the non-standard Crawl-delay directive.

What about Google AI Overviews?

You cannot block AI Overviews with robots.txt without blocking Google Search. Google says robots.txt rules for Googlebot are the control for Search, and AI is part of Search. Google-Extended does not cover AI Overviews.

If you want to stay in normal results but leave AI Overviews and AI Mode, Google now offers a separate switch: Settings > Search generative AI in Search Console, rolled out to all sites on 31 August 2026. Google says excluding your site there is not used as a ranking signal for the rest of Search. Page-level controls such as nosnippet and max-snippet also limit what AI features can show.

The rule that catches everyone: groups do not combine with *

A robots.txt file is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. Under the Robots Exclusion Protocol standard (RFC 9309), a crawler follows the group that names it, and only falls back to the User-agent: * group if no group names it.

So this file does not do what most people expect:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /private/

GPTBot obeys only its own group. It is blocked from /private/ but is free to crawl /admin/ and /checkout/. If a named crawler needs your general rules too, repeat them inside its group.

Copy-paste robots.txt examples

Replace https://example.com with your domain. Keep your existing rules for other crawlers.

Option 1: Allow AI search, block AI training (most common)

# AI search crawlers: allowed so you can be cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Option 2: Allow everything

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

With no named groups, every crawler above follows the * rules.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Be aware of the cost: this removes you from ChatGPT, Claude and Perplexity search answers. User-triggered fetchers from OpenAI and Perplexity may still visit, according to their own documentation.

Option 4: Slow down a crawler instead of blocking it

User-agent: ClaudeBot
Crawl-delay: 1

Anthropic supports Crawl-delay. Google ignores it, so do not use it to manage Googlebot.

How do you test your robots.txt?

  1. Open https://yourdomain.com/robots.txt in a browser and read it top to bottom.
  2. In Search Console, check Settings > robots.txt to see the version Google last fetched and any parsing errors.
  3. Check that the file returns a 200 status and stays under Google's 500 KiB size limit; content after that limit is ignored.
  4. Run your domain through our AI visibility checker, which reports whether each major AI crawler is allowed on your home page.

Changes are not instant. OpenAI says it can take about 24 hours for its systems to adjust to an updated robots.txt, and Perplexity gives a similar figure of up to 24 hours.

Should you also block AI crawlers by IP address?

Usually not. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out" because it stops the bot reading your robots.txt. If you need to verify that a visitor claiming to be a bot is genuine, OpenAI and Anthropic publish their IP ranges; use them for verification rather than as your main control.

For the wider picture of how these crawlers feed AI answers, read GEO explained. If you are wondering whether you also need an llms.txt file next to robots.txt, our llms.txt guide covers who reads it.

FAQ

Does blocking GPTBot hurt my visibility in ChatGPT?

Not in ChatGPT search. OpenAI separates GPTBot (training) from OAI-SearchBot (search). OpenAI's documentation gives this exact setup as an example: allow OAI-SearchBot while disallowing GPTBot.

Will blocking Google-Extended remove me from AI Overviews?

No. Google states that Google-Extended does not affect inclusion in Google Search. AI Overviews are part of Search and use Googlebot. To leave AI Overviews and AI Mode, use the Search generative AI control in Search Console.

Do AI crawlers have to obey robots.txt?

robots.txt is a voluntary standard. OpenAI, Anthropic, Perplexity and Google all document that their crawlers follow it. OpenAI and Perplexity note that their user-triggered fetchers may not apply robots.txt rules, because a person asked for the page.

What is the difference between ClaudeBot and Claude-SearchBot?

ClaudeBot collects content that may be used to train Anthropic's models. Claude-SearchBot indexes content to improve Claude's search results. Blocking ClaudeBot opts you out of training; blocking Claude-SearchBot may reduce how often Claude's search finds and cites you.

Sources

More guides