Most AI companies run two kinds of crawler: one that fetches pages to answer searches now, and one that collects pages to train future models. They have different names. You can block training and still appear in AI search, but only if your robots.txt names the right bots.
Search crawlers: allow these to appear in answers
- Googlebot: Google Search, including AI Overviews and AI Mode.
- Bingbot: Bing and Microsoft Copilot. ChatGPT search also draws on search partners such as Bing.
- OAI-SearchBot: ChatGPT search.
- PerplexityBot: Perplexity answers.
- Claude-SearchBot: search inside Claude.
Training crawlers: your choice
- GPTBot: OpenAI model training.
- ClaudeBot: Anthropic model training.
- Google-Extended: Gemini training and grounding. It does not control Google Search or AI Overviews.
Blocking GPTBot does not remove you from ChatGPT search, and blocking Google-Extended does not remove you from Google. The search bots above decide that.
A safe default
# Search and AI search: allowed User-agent: Googlebot User-agent: Bingbot User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Claude-SearchBot Allow: / # Model training: blocked (remove these lines to allow it) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: /
Two mistakes we see often
- A blanket "User-agent: * / Disallow: /" left over from a staging site. It blocks every crawler, including Googlebot.
- A robots.txt that answers with a server error. Crawlers treat that as "crawl nothing" until it works again.
The SEOSFT AI search check reads your robots.txt the way these crawlers do and shows, bot by bot, who can read your site.