AI Search
AI Crawler Access Auditor
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — see exactly which AI crawlers your robots.txt allows or blocks.
How to use
1
Enter your website
We fetch your live robots.txt.
2
See each AI crawler
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — allowed, blocked, or no specific rule.
3
Decide on purpose
Blocking is a legitimate choice; the point is knowing which stance you actually have, not guessing.
What we check
- Fetches your site’s live robots.txt and checks it against 11 named AI crawlers: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Gemini/AI features training — separate from Googlebot search indexing), CCBot (Common Crawl, feeds many open models), Bytespider (ByteDance), Amazonbot, Applebot-Extended (Apple Intelligence) and meta-externalagent (Meta).
- Reports each bot as Allowed, Blocked, or No specific rule (falls back to whatever the wildcard
User-agent: *group allows). - Deliberately does not tell you which stance to take — blocking AI crawlers is a legitimate choice for some sites. The point is knowing which stance you actually have, rather than assuming.
Related tools
- Robots.txt & Meta Robots Tester — Test the same robots.txt against ordinary search crawlers, not just AI ones
- llms.txt Generator + Validator — robots.txt controls access; llms.txt is the complementary AI-discovery layer
- Website Check — A broader AI-scored review of the rest of the site while you are auditing crawler access
Last updated: 19 August 2026 · Built by the CWA Europe PPC team.
Frequently asked questions
Should I block AI crawlers?
That is a legitimate business decision either way — blocking prevents your content training future models but may reduce your visibility in AI-generated answers. This tool tells you your current stance; it doesn't tell you what to choose.
What if I have no robots.txt at all?
With no robots.txt, crawlers generally assume everything is allowed by default.