Glossary

Plain-English definitions for the terms behind these tools.

AI crawler

A bot run by an AI company that fetches web pages to train models or answer questions with live citations.

Anthropic's web crawler. Allow it in robots.txt so Claude can read and reference your content.

Optimizing your site to be read, understood and cited by AI answer engines — the AI-era counterpart to SEO.

OpenAI's web crawler. Allowing it in robots.txt lets ChatGPT and OpenAI models read and cite your site.

A companion to llms.txt that inlines the full text of your key pages, so a model can read your content in one fetch instead of crawling.

A plain-text file at your site root that tells AI models what your site is and where the important content lives — the AI-era robots.txt + sitemap.

Perplexity's answer-engine crawler. Allow it in robots.txt so Perplexity can index and cite your pages.

A file at your site root that tells crawlers which paths they may fetch — including the named AI bots like GPTBot, ClaudeBot and PerplexityBot.

An XML file listing your site's URLs so crawlers discover every page, even ones nothing links to.