CitenzoCitenzo
GEO Glossary

robots.txt

A file at your site root that tells crawlers — including AI crawlers — which parts of your site they may access.

robots.txt is a plain-text file at yourdomain.com/robots.txt that tells crawlers what they're allowed to access, per user-agent. It's where most AI-crawler blocking happens — often by accident, via a rule that disallows a bot like GPTBot or PerplexityBot.

The major AI crawlers state they honor robots.txt, so what the file says is what they do. If you want to be cited by AI, make sure it isn't quietly blocking the crawlers.

Scanning the signal…