AI crawlers are no longer one simple category. A site can be visited by a crawler used for AI search, another crawler used for user-requested page retrieval, and a different crawler used to collect public web content that may contribute to model improvement. If you block every AI-related user-agent with one blanket rule, you may also remove your content from AI search experiences that could send visitors back to your site.
This guide shows how to control AI crawlers in WordPress with robots.txt without accidentally blocking the search and retrieval bots you still want. If you want to generate the rules instead of writing them by hand, use the free AI Crawler Robots.txt Generator on WPToolix.
First: understand what robots.txt can and cannot do
A robots.txt file is a public text file served from the root of a website, such as https://example.com/robots.txt. Compliant crawlers read it before crawling and use its rules to decide which paths they should access. Google documents the Robots Exclusion Protocol as the standard mechanism its crawlers use for access rules, and major AI companies publish crawler tokens that site owners can target in the same file.
But robots.txt is not authentication. It does not protect private information, stop a determined scraper, or replace a firewall. If content must actually be private, use passwords, application permissions, server configuration, rate limiting, or a WAF. Think of robots.txt as a publishing policy for cooperative crawlers, not a security boundary.
Why blocking “all AI” can be too aggressive
The biggest change for site owners is that the same AI company may operate different crawlers for different purposes.
OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used to improve generative AI models. Those controls are independent. That means a publisher can allow OpenAI’s search crawler while disallowing GPTBot.
Anthropic follows a similar model. Its documentation separates Claude-SearchBot, Claude-User, and ClaudeBot. Claude-SearchBot is used for search result quality, Claude-User supports user-directed retrieval, and ClaudeBot is associated with collecting content that may contribute to model training.
Google provides a separate Google-Extended robots token for managing certain Gemini training and grounding uses. Google explicitly says that Google-Extended does not affect inclusion in Google Search and is not a Google Search ranking signal.
This separation makes a selective policy more useful than a simple “block every AI bot” approach.
A practical balanced policy
For many publishers, a reasonable starting policy is:
- Allow normal search engine crawling.
- Leave AI search and user-retrieval crawlers unblocked.
- Disallow selected training-oriented or dataset crawlers.
- Keep your existing WordPress and sitemap rules.
A simple example looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap_index.xml
Notice what is missing: there is no block for OAI-SearchBot or Claude-SearchBot in this example. Those crawlers are therefore not specifically disallowed by the AI policy.
The exact policy should reflect your goals. A documentation site that values AI citations may choose a more open configuration. A publisher that wants minimal automated reuse may use a stricter one.
Step 1: check your current WordPress robots.txt
Before changing anything, open:
https://yourdomain.com/robots.txtCopy the current contents somewhere safe. Do not replace existing directives blindly. Your current file may already contain rules added by WordPress, Rank Math, Yoast, another SEO plugin, a security plugin, or your hosting platform.
Also check whether you have a physical robots.txt file in the site root. WordPress can serve a virtual robots.txt when no physical file exists, but a real file can override that behavior depending on the server configuration.
Step 2: decide which AI crawler categories matter to you
It helps to group crawlers by purpose rather than brand.
AI search crawlers
These crawlers help build or refresh indexes used for AI search experiences. Examples include OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Blocking them may reduce your chance of appearing in the related search or answer experience.
User-requested retrieval
Some agents fetch a page because a user asked an AI system to access or summarize it. Examples include Claude-User and Perplexity-User. Depending on the provider, robots.txt behavior for user-triggered agents can differ from ordinary scheduled crawling, so always check the provider’s current documentation.
Training or model-improvement crawlers
Examples include GPTBot and ClaudeBot. Publishers who want AI search visibility but prefer to opt out of selected training uses often block these while leaving search crawlers accessible.
Dataset crawlers
CCBot, operated by Common Crawl, builds an open web dataset used by many researchers and companies. Common Crawl documents CCBot as its robots.txt user-agent token and provides a standard full-site blocking rule.
Step 3: generate or write your rules
The easiest way to avoid syntax mistakes is to use the WPToolix AI Crawler Robots.txt Generator. Choose the Balanced preset, review each crawler, add your site URL for the sitemap line, and copy the generated result.
If you write the file manually, each crawler group follows a simple pattern:
User-agent: GPTBot
Disallow: /
The slash means the entire site is disallowed for that user-agent. To block only a section, use the relevant path instead:
User-agent: ExampleBot
Disallow: /members/
Disallow: /private-reports/
Step 4: update robots.txt in WordPress
There are several valid ways to manage robots.txt on WordPress:
- Your SEO plugin: Some SEO plugins provide a robots.txt editor or filtering feature.
- A physical file: You can place
robots.txtin the public web root using your hosting file manager, SFTP, or deployment workflow. - WordPress code: Developers can modify WordPress’s virtual robots response through hooks or a custom plugin.
- Hosting/CDN tooling: Some platforms add bot-management rules outside WordPress.
Use the method already established on your site. Avoid having multiple systems fight over the same file.
Step 5: verify the live response
After saving changes, reload /robots.txt in a private browser window and confirm the exact rules you expect are visible. Check that the response is plain text and that your sitemap URL is correct.
Then verify that you did not accidentally add a catch-all rule such as:
User-agent: *
Disallow: /
That rule can block ordinary search crawlers too and is one of the most damaging mistakes on a production site.
Do not confuse robots.txt with firewall blocking
A crawler can be allowed by robots.txt and still fail to access your site because Cloudflare, Hostinger CDN, Wordfence, another WAF, or server-level bot protection returns a 403. The reverse is also true: robots.txt can request that a crawler stay away even though the web server would technically serve the page.
Use robots.txt when your goal is to communicate crawler preferences. Use a firewall or authentication when you need enforcement.
What about Google-Extended?
Google-Extended deserves special attention because it is not the same thing as Googlebot. Google describes it as a standalone robots.txt product token that publishers can use to control whether content crawled by Google may be used for certain Gemini model training and grounding purposes. Google states that changing Google-Extended does not change normal Google Search inclusion.
So if your goal is to keep Google Search crawling normally while restricting those separate AI uses, blocking Google-Extended is more targeted than blocking Googlebot.
What about OpenAI search visibility?
OpenAI’s crawler documentation says OAI-SearchBot is used to surface websites in ChatGPT search features. It separately describes GPTBot as a crawler for content that may be used to improve generative AI models. If you want search visibility but prefer not to allow GPTBot, write separate rules instead of treating “OpenAI” as one crawler.
You can compare the major tokens in detail in GPTBot vs OAI-SearchBot vs ClaudeBot: Which AI Crawlers Should You Allow?.
Common mistakes to avoid
- Overwriting existing rules: Preserve directives your site already depends on.
- Blocking all crawlers accidentally: Review every
User-agent: *group carefully. - Assuming robots.txt is security: Private data needs real access control.
- Blocking search crawlers unintentionally: Separate search, user-retrieval, and training bots where providers offer distinct tokens.
- Forgetting subdomains: Robots rules apply to the specific host where the file is served. A separate subdomain can require its own policy.
- Ignoring CDN/WAF behavior: A correct robots.txt file does not fix a firewall block.
Frequently asked questions
Can I block GPTBot but keep ChatGPT Search visibility?
OpenAI provides separate controls for GPTBot and OAI-SearchBot, so you can set different robots.txt policies for them. Keep OAI-SearchBot unblocked if ChatGPT search discovery matters to you.
Does robots.txt remove content that was already collected?
robots.txt controls future crawling behavior by compliant bots. It is not a deletion request for data that may already exist elsewhere. Check each provider’s privacy or content-removal processes for separate removal options.
Should every WordPress site block AI training crawlers?
No. This is a publishing decision. Some site owners want maximum distribution, while others prefer tighter control. The important part is understanding what each crawler does before choosing.
Do I need a plugin?
No. A standard robots.txt file is enough to express crawler rules. A plugin can make management easier, but it is not required.
Final checklist
- Back up your current robots.txt rules.
- Decide whether your priority is discovery, training opt-out, or strict blocking.
- Generate targeted rules with the AI Crawler Robots.txt Generator.
- Keep important WordPress and sitemap directives.
- Publish the file at the site root.
- Verify the live response.
- Revisit your policy periodically because crawler names and provider behavior can change.
For other WordPress configuration tasks, you may also find the .htaccess Redirect Generator and WordPress Memory Limit Generator useful.
Official crawler documentation
- OpenAI crawler documentation
- Anthropic crawler guidance
- Google crawler documentation
- Common Crawl FAQ
- Perplexity crawler guide
Featured image: Markus Spiske via Unsplash.