CitenzoWhich of these can read your site?
Enter your domain in our free Robots.txt Checker to see, bot by bot, which of these crawlers are allowed or blocked — and test any URL against your rules.
Check your site free →Training vs. search vs. on-demand crawlers
Not all AI crawlers do the same job, and you can allow or block each independently:
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) gather content in bulk to train models. AI-search crawlers (OAI-SearchBot, PerplexityBot, Bingbot) build an index so your pages can be cited in AI answers — these are the ones that matter most for being recommended. On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) only visit a page when a user's prompt calls for it.
How to block all AI crawlers
Add a Disallow rule per user-agent in your robots.txt. For example:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: /
Note: most crawlers honor these rules, but a few — notably Bytespider and Perplexity's on-demand fetcher — have been reported to ignore robots.txt. Blocking those reliably needs a firewall or WAF rule, not just robots.txt.
How to allow AI crawlers (to get cited)
If you want ChatGPT, Perplexity and Gemini to recommend you, the crawlers must be able to read you — an engine can't cite a page it was never allowed to fetch. Make sure nothing disallows them, and signal intent explicitly:
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: /
Then go further with an llms.txt file that maps your key pages, and verify everything with the Robots.txt Checker.
Frequently asked questions
What is an AI crawler?
An AI crawler is an automated bot that fetches web pages on behalf of an AI product — to train a model on them, index them so they can be cited in AI answers, or read a page live when a user asks a question. GPTBot, ClaudeBot and PerplexityBot are common examples.
What are the main AI crawler user-agents?
The most common are GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot and Claude-User (Anthropic); Google-Extended and Googlebot (Google); PerplexityBot and Perplexity-User (Perplexity); Bingbot (Microsoft Copilot); plus Applebot-Extended, Amazonbot, Meta-ExternalAgent, Bytespider and CCBot.
How do I block all AI crawlers?
Add a Disallow rule for each AI user-agent in your robots.txt (for example User-agent: GPTBot / Disallow: /). Note that most crawlers honor robots.txt but a few, like Bytespider, have been reported to ignore it — those need a firewall or WAF rule.
How do I allow AI crawlers so my brand gets cited?
Make sure your robots.txt doesn't disallow them, and ideally add explicit Allow rules for the crawlers you care about. An engine can't cite a page it was never allowed to read, so allowing GPTBot, PerplexityBot, ClaudeBot and Google-Extended is the baseline for AI visibility.
Do AI crawlers respect robots.txt?
Most major operators (OpenAI, Anthropic, Google, Perplexity's index bot, Apple, Amazon, Common Crawl) state their crawlers obey robots.txt. Some — notably Bytespider, and Perplexity's on-demand fetcher — have been reported to ignore it, so robots.txt alone isn't a guarantee against every bot.
What's the difference between a training crawler and an on-demand fetcher?
A training or search crawler visits your site in bulk to build a dataset or index. An on-demand fetcher (like ChatGPT-User or Perplexity-User) only fetches a specific page when a user's prompt calls for it. You can allow or block each independently in robots.txt.
User-agents cross-checked against knownagents.com (formerly Dark Visitors) and operators' own documentation, July 2026. Operators add and rename crawlers over time — always confirm against the operator's current docs before relying on a single rule.