AI Crawler Robots.txt Generator
Generate robots.txt rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers. Runs locally in your browser.
Runs locally in your browserGenerate AI crawler rules for robots.txt
Used only to add a Sitemap line such as https://example.com/sitemap_index.xml.
robots.txt is a cooperative crawler directive, not a security control. Review the generated file before replacing an existing robots.txt file.
What this AI crawler robots.txt generator does
AI services use different crawlers for search, user-requested retrieval, and model improvement. This generator helps create a clear robots.txt policy for major AI-related crawler tokens while running locally in your browser.
Recommended starting point: Balanced
The Balanced preset is designed for publishers who want AI search and user-directed discovery without automatically allowing training-oriented crawlers. You can switch to Custom and change any individual crawler before generating the file.
How to use the generator
- Choose a preset.
- Enter your site URL if you want a sitemap line.
- Keep or remove the optional WordPress admin rules.
- Review each crawler and adjust the Block switch.
- Add extra crawler user-agent tokens if needed.
- Generate, copy, or download the resulting robots.txt file.
Where robots.txt belongs on WordPress
A robots.txt file applies from the root of a host, for example https://example.com/robots.txt. Check your current robots.txt and preserve rules you still need before replacing anything.
robots.txt is not an access-control system
The Robots Exclusion Protocol is a publishing signal for compliant crawlers. It does not authenticate visitors or make private information private.
Frequently asked questions
Can robots.txt stop every scraper?
No. robots.txt is honored voluntarily by compliant crawlers. Use authentication, server rules, rate limiting, or a WAF when you need actual enforcement.
Can I add my own crawler?
Yes. Enter one user-agent token per line in the Extra user-agents field.
Related WPToolix tools
Try the .htaccess Redirect Generator, WordPress Memory Limit Generator, or AI JSON Repair & Formatter.