Is your site accidentally blocking AI? (and how to fix it)

Before you optimize anything for AI, check the basics: can AI even read your site? A striking number of businesses block the exact crawlers that feed ChatGPT, Perplexity and Gemini — usually by accident — and then wonder why they're never cited.

How sites block AI without meaning to

It's almost always robots.txt. A rule copied from a template, a blanket block added during a traffic scare, or a security plugin's default can disallow AI crawlers site-wide. Some sites do it deliberately — publishers protecting content — but if you want to be recommended by AI, blocking the bots is shooting yourself in the foot.

The crawlers that matter aren't hidden. They announce themselves with clear user-agent names:

• GPTBot, OAI-SearchBot, ChatGPT-User — OpenAI / ChatGPT
• ClaudeBot, anthropic-ai — Anthropic / Claude
• PerplexityBot, Perplexity-User — Perplexity
• Google-Extended — Google Gemini & Vertex AI
• Applebot-Extended, CCBot, Amazonbot — Apple, Common Crawl, Amazon

How to check in 30 seconds

You can read your own robots.txt at yourdomain.com/robots.txt and look for Disallow rules under those user-agents. Or use our free AI Crawlability Checker — it fetches your robots.txt and tells you, bot by bot, who's allowed and who's blocked, plus whether you have an llms.txt.

How to fix it

If you want AI to read and cite you, allow the crawlers explicitly. Add rules like these to your robots.txt:

User-agent: GPTBot → Allow: / · User-agent: PerplexityBot → Allow: / · User-agent: Google-Extended → Allow: /

Then go one step further: publish an llms.txt. It's a short file at yourdomain.com/llms.txt that tells engines what your site is and which pages matter — the machine-readable equivalent of a good elevator pitch. Our free llms.txt generator builds one in a minute.

Being readable isn't the whole game

Unblocking crawlers won't single-handedly make AI recommend you — that still depends on being present on the sources engines trust. But it's the foundation. Skip it, and every other bit of GEO effort is undermined by a door you left locked.

Check yours now

Run your domain through the free AI Crawlability Checker to see if you're blocking AI — then run a full visibility check to see whether it's actually recommending you.