← Back to blog
September 29, 2026Jobayer

GPTBot vs OAI-SearchBot vs ClaudeBot: Which AI Crawlers Should You Allow?

Compare major AI crawler user-agents, what they are used for, and how to build a selective robots.txt policy for search, retrieval and training crawlers.

Abstract connected network representing AI web crawlers, search and data collection

GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, and CCBot often get grouped together as “AI crawlers,” but they do not all do the same job. Some are designed to help AI search products find and cite web pages. Others support user-directed retrieval. Others are used for model improvement or large public datasets.

That distinction matters because a single blanket Disallow: / policy can trade away visibility you actually wanted. This guide compares the major crawler tokens and gives you a practical framework for deciding what to allow or block. You can then generate the exact rules with the free WPToolix AI Crawler Robots.txt Generator.

Quick comparison

Crawler token Operator Primary documented purpose Typical policy question
OAI-SearchBot OpenAI ChatGPT search discovery Do I want my pages eligible for ChatGPT search results?
GPTBot OpenAI Model improvement / training-related crawling Do I want to allow this use while keeping search separate?
Claude-SearchBot Anthropic Search result quality Do I want visibility in Claude search experiences?
Claude-User Anthropic User-directed retrieval Should Claude users be able to retrieve my public pages?
ClaudeBot Anthropic Model improvement / training-related collection Do I want to opt out while allowing search?
PerplexityBot Perplexity AI search indexing Do I want Perplexity to discover and cite my content?
Perplexity-User Perplexity User-triggered retrieval Should live user requests be able to fetch my page?
Google-Extended Google Gemini training / grounding control Do I want to restrict those uses without blocking Google Search?
CCBot Common Crawl Open web dataset collection Do I want my content in the Common Crawl dataset?

The table is intentionally about documented purpose, not a recommendation that every site should choose the same policy.

GPTBot vs OAI-SearchBot: the most important distinction for OpenAI

OpenAI publishes separate crawler controls for different products. OAI-SearchBot is used for search. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they may still appear as navigational links in some situations.

GPTBot is different. OpenAI describes it as a crawler used to make its generative AI foundation models more useful and safe, and says that disallowing GPTBot indicates that a site’s content should not be used for training those models.

For a publisher, that creates a clear selective option:

User-agent: GPTBot
Disallow: /

while leaving OAI-SearchBot unblocked.

This is why a generic rule that blocks every OpenAI user-agent is much less precise than separate rules.

ClaudeBot vs Claude-SearchBot vs Claude-User

Anthropic documents three different robots:

  • ClaudeBot collects public web content that could potentially contribute to future model training.
  • Claude-SearchBot navigates the web to improve the relevance and accuracy of search responses.
  • Claude-User supports user-requested access when a Claude user asks the system to retrieve web content.

Anthropic says its bots honor robots.txt directives. It also notes that blocking Claude-SearchBot can reduce visibility and accuracy in user search results.

A publisher who wants search visibility but not training-oriented collection might therefore block ClaudeBot while leaving Claude-SearchBot accessible:

User-agent: ClaudeBot
Disallow: /

Again, the value comes from separating purposes instead of treating the whole company as one crawler.

PerplexityBot and Perplexity-User

Perplexity publishes crawler documentation for PerplexityBot and Perplexity-User. PerplexityBot is associated with its search index, while Perplexity-User is used for user-triggered retrieval.

If Perplexity citations or discovery matter to your site, think carefully before blocking the index crawler. A content site, product documentation site, or developer blog may see more upside from being discoverable than a private or low-value section of a site would.

You can also combine path-specific restrictions with an otherwise open policy. For example, you might keep public articles accessible while disallowing a directory you do not want crawled.

Google-Extended is not Googlebot

This is one of the easiest crawler controls to misunderstand. Google-Extended is a standalone robots.txt product token. Google says publishers can use it to manage whether content Google crawls may be used for training future Gemini models and for certain grounding uses.

Google also says Google-Extended does not affect a site’s inclusion in Google Search and is not a Google Search ranking signal.

That means a site can use:

User-agent: Google-Extended
Disallow: /

without using a blanket block against Googlebot.

If your objective is “keep normal Google Search, restrict separate Gemini-related uses,” Google-Extended is the targeted control to evaluate.

CCBot and the Common Crawl dataset

CCBot is operated by Common Crawl, a nonprofit that builds and maintains a large open repository of web crawl data. Common Crawl says CCBot checks robots.txt before fetching pages and publishes CCBot as the exclusion user-agent token.

Because Common Crawl data can be used by many downstream researchers and organizations, blocking CCBot is a broader dataset decision rather than a decision about one AI product.

The standard full-site opt-out is:

User-agent: CCBot
Disallow: /

Three policy models you can use

1. Discovery-first

This model leaves most major AI crawlers unblocked. It may fit publishers who want maximum visibility and do not object to training or dataset uses.

Advantages:

  • Maximum potential discoverability.
  • Fewer crawler-specific rules to maintain.
  • Suitable for content intended for broad redistribution.

Trade-off: you give providers and datasets broader access to public content.

2. Search-friendly, training-restricted

This is the policy used by the Balanced preset in the WPToolix generator. It leaves search and user-retrieval crawlers unblocked while disallowing selected training-oriented or dataset crawlers such as GPTBot, ClaudeBot, Google-Extended, and CCBot.

Advantages:

  • Keeps a path open for AI search discovery.
  • Expresses a more targeted opt-out preference.
  • Avoids treating all AI traffic as equivalent.

Trade-off: crawler purposes and product behavior can evolve, so the policy needs periodic review.

3. Strict AI crawler blocking

This model disallows all AI-related crawlers you have identified.

Advantages:

  • Simple policy.
  • Minimizes access by compliant listed AI crawlers.

Trade-off: you may reduce visibility in AI search and user-directed retrieval experiences. It also does not stop noncompliant scrapers, so it should not be confused with enforcement.

How to decide what your site should allow

Ask five questions:

  1. Do AI citations and referral traffic matter to me? If yes, keep search crawlers in mind before blocking.
  2. Am I comfortable with model-improvement or training-related crawling? If not, evaluate the provider’s training-oriented crawler token.
  3. Is the content truly public? If it should be private, robots.txt is the wrong tool.
  4. Do I already have firewall or bot-management rules? Those can override what robots.txt appears to allow.
  5. Can I maintain the policy? Crawler names and documentation can change over time.

Why “Allow” and “Block” are not always the whole story

Robots.txt controls crawling, but it does not necessarily control every way a URL can be discovered or referenced. A blocked URL can still be known from external links or other signals. Provider-specific behavior can also differ.

Similarly, allowing a crawler in robots.txt does not guarantee it can reach your page. A CDN, WAF, rate limiter, authentication layer, geoblock, or server configuration may stop the request first.

For that reason, crawler management has two separate layers:

  • Policy layer: robots.txt tells compliant crawlers what you want.
  • Enforcement layer: server and security controls determine what requests can actually reach content.

Verify bots before making server-level blocks

User-agent strings are easy to spoof. If you move beyond robots.txt and start allowlisting or blocking IP traffic at a firewall, use the verification guidance published by the provider rather than trusting the user-agent alone.

OpenAI publishes IP ranges for its crawlers. Common Crawl also publishes crawler IP information and recommends verification because other clients can falsely identify themselves as CCBot. Anthropic currently notes that it does not publish dedicated IP ranges for its bots, so a different verification strategy may be needed there.

A WordPress-friendly workflow

  1. Open your current /robots.txt and save a copy.
  2. Decide on discovery-first, balanced, or strict policy.
  3. Use the AI Crawler Robots.txt Generator to build targeted rules.
  4. Merge the output with important existing WordPress directives.
  5. Publish through your SEO plugin, host, file manager, or deployment process.
  6. Reload the live /robots.txt and verify the result.
  7. Check your CDN/WAF separately if a crawler still gets 403 responses.

For a more detailed implementation walkthrough, read How to Block AI Crawlers in WordPress Without Blocking AI Search Traffic.

Frequently asked questions

Is GPTBot the same as ChatGPT Search?

No. OpenAI documents GPTBot and OAI-SearchBot separately. OAI-SearchBot is the crawler associated with ChatGPT search discovery.

Can I block ClaudeBot but allow Claude-SearchBot?

Yes. Anthropic publishes separate robots.txt tokens for them, so your file can give each crawler a different rule.

Will Google-Extended block Google Search?

Google says Google-Extended does not affect normal Google Search inclusion and is not a Search ranking signal.

Does blocking CCBot block one AI company?

No. CCBot belongs to Common Crawl, which maintains an open web dataset used by many downstream organizations and researchers.

Should I copy someone else’s robots.txt file?

No. The right policy depends on your site, existing directives, business goals, and technical setup. Generate or write rules intentionally and merge them with your own configuration.

Bottom line

The best AI crawler policy is usually not “allow everything” or “block everything.” Modern providers increasingly expose separate user-agent tokens for search, user retrieval, model improvement, and other uses. That lets you make a more precise decision.

Start by identifying what you want from AI discovery, then use the WPToolix AI Crawler Robots.txt Generator to turn that decision into a clean robots.txt policy. Recheck the provider documentation periodically, because crawler behavior is an evolving part of the web.

If you also manage WordPress server configuration, the .htaccess Redirect Generator and WordPress Memory Limit Generator may be useful next.

Official crawler documentation

Featured image: Growtika via Unsplash.