AI crawler robots.txt generator · Free tools
Choose which AI crawlers can read your site.
Some crawlers find answers for customers. Others collect training data. You can treat them differently.
Everything you type stays in your browser. Nothing is sent to us.
Your robots.txt
User-agent: *
Allow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Allow: /
Upload it to the root of your site, replacing the old file only after you've checked what's in it.
Good to know
If I block training crawlers, can AI tools still recommend me?
Blocking GPTBot doesn't block OAI-SearchBot, so your site can still appear in ChatGPT search. Training data can still influence what AI models know about you over time, so it's a real choice with no single right answer.
Does every crawler obey robots.txt?
No. It's a request, not a lock. Well-known crawlers say they follow it, but it won't protect private content. Fetches that a person asks for directly, like ChatGPT-User, may be treated differently too. Use a password for anything private.
I already have a robots.txt. Can I just replace it?
Check it first. Your current file may have rules your website platform needs. Add those paths to the private paths box, so they end up in the new file.