CitenzoAI Crawler
An automated bot that fetches web pages on behalf of an AI product — to train a model, index pages, or read a page live for a user.
An AI crawler is a bot that requests your web pages for an AI product. They fall into three broad jobs: training crawlers (like GPTBot or ClaudeBot) that gather content to train models, search/index crawlers (like OAI-SearchBot or PerplexityBot) that index pages so they can be cited in answers, and on-demand fetchers (like ChatGPT-User) that grab a page live when a user's prompt needs it.
You can allow or block each crawler independently in robots.txt. If you want AI to recommend you, the crawlers have to be able to read you first — an engine can't cite a page it was never allowed to fetch.