robots.txt

A file at your site root that tells crawlers which paths they may fetch — including the named AI bots like GPTBot, ClaudeBot and PerplexityBot.

robots.txt is the oldest crawler-control standard: per user-agent Allow and Disallow rules that tell each bot where it may go. It is advisory — well-behaved crawlers honour it — and it is where you decide whether the AI answer engines can reach your site.

Allowing the AI bots

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Block a bot with Disallow: / under its user-agent instead. Be deliberate — blocking a crawler means that assistant can no longer cite you.