AI Crawler Access Checker: robots.txt, llms.txt and Bot Rules

Most AI companies now run separate crawlers for training their models, for search, and for fetching a page when a person asks the assistant to. A robots.txt file written to “block AI” a couple of years ago often blocks all three, which can quietly take a site out of AI answers. This free Google Colab notebook checks what your robots.txt actually allows for 27 AI crawlers and control tokens, for one domain or a whole list, and checks your llms.txt file while it’s there.

Who it’s for

Technical SEOs, site owners and agencies who want to know whether OAI-SearchBot, Claude-SearchBot, PerplexityBot and the rest can reach their pages. It’s also a quick first step before any answer engine optimisation conversation with a client: check access before you talk about content.

How to use it

1. Open the notebook in Colab and run it with USE_SAMPLE = True. It uses saved robots.txt files from six real sites plus a fictional shop packed with common mistakes, so you see every output before touching your own site.
2. Set USE_SAMPLE = False, add your domains to DOMAINS and a few paths that matter to PATHS (the homepage, a blog post, a product page).
3. Run the audit and read the summary: how many training bots, search bots and user-initiated fetchers each site blocks.
4. Work through the flags, high severity first.
5. If you like, set RUN_LIVE_UA_CHECK = True to request one of your own pages with each crawler’s user-agent string.
6. Export the CSVs and the Excel workbook, and open it in Google Sheets.

What you get

• An access matrix: every crawler on every site, allowed or blocked, evaluated with Protego, which follows Google’s robots.txt rules (longest match wins, Allow wins ties, wildcards work)
• The crawler list with what each token controls and a link to each vendor’s own documentation, checked on 24 October 2026
• Flags for the usual mistakes: a blanket Disallow for all bots, search bots blocked along with training bots, Googlebot blocked, Applebot blocked when Applebot-Extended was meant, typos and outdated tokens such as anthropic-ai
• llms.txt and llms-full.txt checks: present, soft 404, and basic llmstxt.org structure
• A Sheets-ready workbook with a tab per table

Limits and costs

It’s free and needs no API keys. A few honest limits. robots.txt is a request, and some vendors say their user-initiated fetchers may not follow it. Google-Extended causes the most confusion: it controls Gemini training and grounding, and it has no effect on Google Search, AI Overviews or AI Mode, which run on Googlebot. The live user-agent check only detects blocking based on the user-agent header; CDNs usually verify real crawlers by IP address, so treat it as a hint. Crawler names change, so re-check the vendor pages before you edit a live file. And llms.txt is a community proposal that the big AI search providers haven’t said they read.

For the strategy behind these rules, read Beatrice Gamba’s post Your robots.txt might be the silent killer of your AI traffic, and go further in the AI Search Optimization & Agentic SEO course.