robots.txt

robots.txt is a plain text file placed at the root of a domain (for example example.com/robots.txt) that tells automated crawlers which parts of the site they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. In the AI search era it controls access for AI crawlers as well as search engines, which makes it one of the first files to check in any AI visibility audit.

How it works

The file is made of groups. Each group names one or more user agents and lists allow and disallow rules for URL paths; the most specific matching rule wins. A sitemap location can also be declared. Well-behaved crawlers fetch the file before crawling and follow it, but it is a request, not access control: it does not stop a determined scraper, and it does not remove pages from an index.

AI companies publish separate tokens for different purposes. OpenAI, for instance, uses GPTBot for collecting training data and OAI-SearchBot for search, and lets sites allow one while blocking the other. Google-Extended is a token that controls use of content for Gemini model training and grounding in Gemini apps and Vertex AI; Google states that it does not affect inclusion or ranking in Google Search.

How it’s used in SEO and marketing

• Keeping crawlers out of low-value areas such as internal search results, carts or staging paths.
• Choosing, per bot, whether to allow AI training, AI search and user-triggered fetching.
• Diagnosing missing AI visibility: MLforSEO calls a misconfigured robots.txt the “silent killer” of AI traffic, because a blocked agent leaves no error, no analytics event and no alert.
• Making sure images, CSS and JavaScript needed to render pages are not blocked.

Example

An e-commerce site’s robots.txt, written years ago after a scraping problem, disallows everything except Googlebot. Its products never appear in ChatGPT or Perplexity answers. The fix is to add explicit groups for the AI search and user agents it wants, keep disallows narrow (cart, account, internal search), then verify by requesting a page with each bot’s user-agent string and checking that real content comes back.

Limitations and things to watch

• Blocking a URL in robots.txt does not de-index it. Use noindex (on a crawlable page) or removal tools for that.
• Rules are case-sensitive and order-independent for Google; test changes with Search Console’s robots.txt report.
• Bot names and purposes change. Check each provider’s current documentation rather than copying old lists.
• Firewalls and CDNs can block bots even when robots.txt allows them, so check server responses too.

Learn it on MLforSEO

• Your robots.txt might be the silent killer of your AI traffic
• Blog and Site Architecture for SEO and Conversions in the AI Search Era
• Image SEO in 2026: The Complete Guide to Image Optimisation, Advanced Tactics and Multimodal Search
• Agent Readiness Stack Audit
• AI Search Optimisation & Agentic SEO (course)