CitenzoGPTBot
OpenAI's web crawler that gathers content to train and improve its models. Controlled via robots.txt.
GPTBot is OpenAI's crawler for collecting web content to train and improve its models. It's one of several OpenAI user-agents — alongside OAI-SearchBot (which indexes pages for ChatGPT Search) and ChatGPT-User (which fetches a page live during a session).
GPTBot honors robots.txt, so you can allow or block it by name. Blocking it keeps your content out of OpenAI's training, but if you want ChatGPT to know and recommend you, you'll generally want it allowed.