AI Crawlers (GPTBot, ClaudeBot, PerplexityBot)

AI crawlers are automated user agents operated by AI companies to fetch web content. Some collect pages to train models, some build the search index behind an AI assistant, and some fetch a page live because a user asked a question. Each identifies itself with a user-agent name, and most say they respect robots.txt, so the file you wrote years ago for Googlebot now decides whether AI systems can read you at all.

How it works

The major providers now separate their bots by purpose. As of their documentation in 2026:

• OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search results) and ChatGPT-User (fetches made on behalf of a user).
• Anthropic: ClaudeBot (training), Claude-SearchBot (search quality) and Claude-User (user-initiated fetches).
• Perplexity: PerplexityBot (indexing for search results) and Perplexity-User (live fetches for a user).
• Google: Google-Extended is a robots.txt token, not a separate crawler. It controls whether content Googlebot fetches can be used for Gemini models and grounding. Google says it does not affect Search, including AI Overviews.
• Common Crawl: CCBot builds an open web archive that many AI datasets draw on.

Because purposes are split, you can, for example, block training bots while allowing search bots that can cite and link to you. User-initiated fetchers may not follow robots.txt in the same way, since they act for a person.

How it’s used in SEO and marketing

• Auditing robots.txt for blanket blocks written before AI search existed, and allowing the bots you want explicitly.
• Checking server logs to see which AI bots visit, how often and which pages they request.
• Verifying access by fetching your own pages with an AI user-agent string and confirming you get real content, not a block page.
• Keeping pages light and server-rendered so crawlers that do not run JavaScript can read them.
• Deciding a policy on training versus search access with legal and content teams.

Example

Your robots.txt might be the silent killer of your AI traffic describes a robots.txt “written for 2015”: it allows only Googlebot, or was tightened during a scraping scare, and turns AI crawlers away at the door with nothing logged. The fix takes about ten minutes: explicit allow rules for the agent crawlers you want, narrow disallows for private paths only, and a test fetch to confirm.

Limitations and things to watch

• User-agent strings can be spoofed. Verify important bots against the IP ranges providers publish.
• Bot names and purposes change; check each provider’s official page before editing rules.
• Blocking training bots does not remove content already collected.
• High crawl volumes can strain servers, and crawl activity rarely matches the referral traffic you get back.

Learn it on MLforSEO

• Your robots.txt might be the silent killer of your AI traffic
• Blog and Site Architecture for SEO and Conversions in the AI Search Era
• AI Search Optimisation & Agentic SEO
• Agent SEO Implementation Roadmap