Free tool
Configure which AI crawlers can access your site. Block training crawlers while keeping retrieval bots for citation opportunities.
Honest disclaimer: Per Rutgers/Wharton (Dec 2025): blocking AI crawlers reduces site traffic ~23.1% without reliably reducing AI citation rates. The "block training, allow retrieval" preset preserves citation eligibility while denying training-data ingestion.
robots.txt output
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Bot inventory
| Agent | Kind |
|---|---|
| GPTBot | training |
| ChatGPT-User | retrieval |
| OAI-SearchBot | retrieval |
| ClaudeBot | training |
| Claude-User | retrieval |
| anthropic-ai | training |
| PerplexityBot | retrieval |
| Perplexity-User | retrieval |
| Google-Extended | training |
| GoogleOther | ambiguous |
| CCBot | training |
| Applebot-Extended | training |
| Bytespider | training |
| Amazonbot | ambiguous |
| Meta-ExternalAgent | training |
| Meta-ExternalFetcher | retrieval |