AI crawler
A bot run by an AI company that fetches web pages to train models or answer questions with live citations.
An AI crawler is an automated fetcher operated by an AI provider — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and others. Some crawl for model training; others fetch pages in real time to ground and cite answers. Each identifies itself with a named user-agent you can allow or block in robots.txt.
If AI answers never mention your site, the first thing to check is whether these crawlers are even allowed to reach it.