llms.txt

llms.txt is a proposed standard for a plain Markdown file at the root of a website (yourdomain.com/llms.txt) that gives large language models and AI agents a short, curated map of the site: a title, a one-paragraph summary and lists of links to the most important pages. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is a voluntary proposal, and as of October 2026 there is little evidence that major AI search systems read it.

How it works

The specification at llmstxt.org requires only an H1 with the site or project name. It recommends a blockquote summary, optional detail paragraphs, and H2 sections containing Markdown lists of links, each with an optional short description. The proposal also suggests offering clean Markdown versions of pages (for example by appending .md to a URL). The page was revised to a second version in August 2026.

The idea is efficiency: context windows are limited and every wasted token costs time and money, so a compact index helps an agent decide what to fetch. MLforSEO’s Agentic SEO course teaches it as “a sitemap for agents” and recommends organising entries by task, describing outcomes and noting an estimated token count per page. Token counts are our own recommendation, not part of the specification.

How it’s used in SEO and marketing

• Listing hubs, cornerstone guides, pricing, policies and product catalogues so an agent already on your site can find them quickly.
• Documentation sites use it widely; OpenAI, Anthropic and Google publish llms.txt files for their own developer docs.
• Pairing it with an open robots.txt: a map is useless if AI crawlers are blocked from the pages it points to.

Example

A retailer’s file opens with “# Example Outdoor” and a one-line summary, then an H2 “Shop by task” with links such as “Hiking jackets catalogue: compare waterproof ratings, prices and stock” and an H2 “Policies” linking the returns and shipping pages. In the course scenario, an agent reads this, fetches only the jacket catalogue and completes the task without crawling the whole site.

Limitations and things to watch

• Google’s AI features documentation says you don’t need new machine-readable files or “AI text files” to appear in AI Overviews or AI Mode, and that Google Search doesn’t use them.
• OpenAI’s crawler documentation explains OAI-SearchBot, GPTBot and ChatGPT-User through robots.txt and does not say any of them fetch or honour llms.txt. Publishing the file for your own docs is not the same as reading it on other sites.
• Studies cited in MLforSEO’s blog architecture guide found that most llms.txt files receive little or no bot traffic. Treat it as cheap, low-risk housekeeping for agents, not a ranking or citation lever.
• Chrome’s Lighthouse now includes an agentic browsing check for the file, but treats a missing file as not applicable.

Learn it on MLforSEO

• Blog and Site Architecture for SEO and Conversions in the AI Search Era includes llms.txt with an honest caveat and the evidence.
• Your robots.txt might be the silent killer of your AI traffic covers crawler access and drafting your first file.
• The AI Search Optimisation & Agentic SEO course teaches agent infrastructure, including llms.txt.
• Agent Readiness Stack Audit is a free template for checking your agent infrastructure.