CitenzoHow to use this robots.txt checker
Enter your domain and the tool does three things a good robots.txt checker should. First, it fetches and shows your live robots.txt so you can read the actual rules. Second, it tests each major crawler against them — Googlebot and Bingbot for search, plus GPTBot, PerplexityBot, ClaudeBot and the other AI bots — and marks each Allowed or Blocked. Third, you can enter any URL path and pick a user-agent to test whether that exact page is crawlable. On top of robots.txt, it also checks the other things that stop a page being indexed — HTTP status, JavaScript dependency, noindex, canonical and sitemap.
What is crawlability?
Crawlability is whether bots can fetch, read and index your pages. It has always mattered for SEO — if Googlebot can't crawl a page, it can't rank — but AI has raised the stakes: assistants like ChatGPT, Perplexity, Claude and Google's AI answers can only recommend content their crawlers were able to read. This tool runs a live crawlability test on any URL and shows you exactly what's stopping AI and search engines from reading it — the same crawlability issues that hurt SEO also make you invisible to AI.
What this crawlability test checks
Enter a URL and the tool fetches it the way a crawler does — without executing JavaScript — then reports:
noindex meta tag or X-Robots-Tag header is telling engines to leave the page out.This is essentially viewing your page as a bot sees it: it renders no JavaScript, so what the tool reports is close to what GPTBot or a non-rendering crawler actually receives.
Which AI crawlers this tool checks
There isn't one "AI bot" — each major AI product runs its own crawler with its own user-agent, and each can be allowed or blocked independently. This checker reads your robots.txt and reports the status of the 12 that matter most today:
Want every user-agent, not just these? See the full AI crawlers list — 25+ bots with what each does and whether it respects robots.txt.
How to check if your site blocks AI
Almost all AI-crawler blocking happens in one file: robots.txt, at yourdomain.com/robots.txt. You can open it yourself and look for Disallow rules under the user-agents above — but the format is fiddly and a single wildcard rule can block everything. The checker above does it for you: enter your domain and it reads the live robots.txt, resolves the rules per bot, and marks each crawler Allowed or Blocked. The major AI crawlers publicly state they respect robots.txt, so what the file says is what they do — which is exactly why it's worth getting right.
Should you block or allow AI crawlers?
This is a real business decision, not a default. If you're a publisher whose product is your content, blocking AI crawlers protects it from being used without compensation — a legitimate choice. But if you want AI to recommend your brand, product or service, blocking the crawlers is self-defeating: an engine can't cite a page it was never allowed to read. For most businesses trying to be found, allowing the retrieval and live-fetch crawlers is the right call. You can also allow some and block others — for example, permit search and live-fetch bots while blocking pure training crawlers.
How to allow or block a specific crawler
To allow the AI crawlers you care about, make sure your robots.txt doesn't disallow them. Explicit allow rules are the clearest way to signal intent:
User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: /
To block a specific crawler instead — say you want to keep GPTBot out — disallow it by name:
User-agent: GPTBot Disallow: /
Once crawlers can read you, the next step is helping them understand you. An llms.txt file gives AI engines a curated map of your most important pages — read more on getting readable by AI.
Frequently asked questions
What is a robots.txt checker?
A robots.txt checker fetches your site's robots.txt file and shows whether specific crawlers — search engines like Googlebot and AI bots like GPTBot — are allowed or blocked. This one also lets you test any URL against the rules and flags crawlability issues beyond robots.txt.
How do I check my robots.txt file?
Enter your domain above. The tool reads your live robots.txt, shows the raw file, and reports which search and AI crawlers can access it. You can also type any URL path and pick a user-agent to test whether that exact page is allowed or blocked.
What is an AI crawler?
An AI crawler is an automated bot that fetches your web pages on behalf of an AI product — to train its model, index your pages for retrieval, or read a page live when a user asks a question. GPTBot, PerplexityBot and ClaudeBot are examples.
Does ChatGPT respect robots.txt?
Yes. OpenAI's crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) publicly state they follow robots.txt rules, as do Perplexity, Anthropic and Google's AI crawlers. So a Disallow rule in your robots.txt does keep them out — which is why an accidental block quietly costs you AI visibility.
How do I know if my site is blocking AI?
Enter your domain in the checker above. It reads your live robots.txt and shows, bot by bot, whether each major AI crawler is allowed or blocked, plus whether you have an llms.txt.
Should I block AI crawlers?
Only if protecting your content from AI outweighs being recommended by it. If you want ChatGPT, Perplexity or Gemini to mention your brand, you need to allow their crawlers — they can't cite what they can't read.
What's the difference between robots.txt and llms.txt?
robots.txt tells crawlers what they're allowed to access. llms.txt tells AI engines what your site is and which pages matter most, in a clean machine-readable format. robots.txt controls access; llms.txt improves understanding. You want both set correctly.
How do I allow GPTBot or PerplexityBot?
Make sure your robots.txt has no Disallow rule for them, and ideally add an explicit User-agent: GPTBot / Allow: / block. See the robots.txt snippets above.
Can AI crawlers read JavaScript?
Mostly no. Unlike Googlebot, the major AI crawlers (GPTBot, PerplexityBot, ClaudeBot and others) generally fetch your raw HTML without executing JavaScript. If your main content is rendered client-side, they see an almost-empty page — which is why this tool flags JavaScript dependency as a crawlability issue.
What makes a page not indexable?
The most common causes are a noindex directive (in a meta robots tag or the X-Robots-Tag header), a non-200 HTTP status, a robots.txt block, or content that only appears after JavaScript runs. This tool checks all of these in one pass.
How do I see my page the way a crawler does?
Enter the URL above. The tool fetches the page without running JavaScript — close to what a non-rendering bot like GPTBot receives — and shows the status, how much text is in the raw HTML, and any indexability blockers.