llms.txt: What It Is and Does It Actually Help AI Search?

llms.txt is a proposed file that gives AI models a curated map of your website. It has generated a lot of hype, so here is the honest position. As of 2026, no major AI search engine confirms using llms.txt to find, rank or cite content, and server-log studies show AI bots request it around 0.1 to 1.1 percent of the time. It is still genuinely useful for one job, and this guide explains which, and what to prioritize instead.

Key takeaways

• llms.txt is a Markdown file at your site root that lists your most important pages for AI models. It was proposed by Jeremy Howard of Answer.AI in September 2024.
• No major AI search engine, Google, OpenAI, Anthropic, Perplexity or Microsoft, confirms using it to rank or cite content. Google’s John Mueller said plainly that no AI system currently uses it.
• Five independent server-log studies agree: AI bots almost never request the file. In one, 97 percent of llms.txt files got zero traffic.
• It does help its original use case: coding assistants and agents you point at your site read it to navigate your docs.
• Ship it as cheap insurance if a plugin generates it automatically, but never trade content quality, crawlability or schema for it.

What is llms.txt?

llms.txt is a plain-text Markdown file placed at your site root that gives large language models a curated overview of your site, with a short summary and links to your most important pages. Jeremy Howard, co-founder of Answer.AI, proposed it in September 2024. Its original purpose was practical: help coding assistants read a project’s documentation at the moment they need it, not to influence search rankings.

What the file looks like

The format is simple: an H1 with the site or project name, a blockquote summary, then H2 sections listing Markdown links to key pages, each with an optional note. An “Optional” section flags links that can be skipped to save context.

llms.txt vs llms-full.txt

They are different sizes of the same idea. llms.txt is a concise index that links out to your important pages. llms-full.txt inlines the entire documentation as one large Markdown file, so a model can ingest everything in a single fetch. The concise version is the one most sites publish.

llms.txt vs robots.txt: not the same thing

These files do opposite jobs, and confusing them is common. robots.txt, from 1994, is an access-control file that tells crawlers what they may not fetch, and all major AI crawlers already honour it. llms.txt is the reverse: an inclusion file that suggests what is most worth reading. robots.txt is enforced and universally respected. llms.txt is advisory, and honoured by almost no AI search engine. llms.txt does not replace robots.txt, and it does nothing to control access.

Does llms.txt actually work? The honest 2026 verdict

For AI search visibility, no. As of 2026, no major AI search engine has confirmed using llms.txt to discover, rank or cite web content, and independent server-log studies show AI crawlers essentially ignore it. The one honest exception is its original purpose: developer and agent tools you explicitly point at a site do read it.

What the server-log studies show

Five independent datasets reach the same conclusion.

What the engines have actually said

Google is the most direct. John Mueller stated that no AI system currently uses llms.txt, and in mid-2026 called the idea purely speculative, noting the file has existed for years while none of the AI systems use it. OpenAI, Anthropic, Perplexity and Microsoft have published nothing confirming their models parse or prioritize it. Anthropic does publish an llms.txt for its own docs, but publishing a file is not the same as consuming other sites’ files.

The one case where it does work

llms.txt earns its keep for its original job. When you point a coding assistant like Cursor at your documentation, or paste your llms.txt URL into ChatGPT or Claude, or connect a tool through the Model Context Protocol, the file gives the agent a clean map of your content. That is real and useful. It is simply a different thing from ranking in AI search.

Who is using llms.txt, and why is adoption growing anyway?

Adoption is low in absolute terms but rising fast, driven by tooling rather than proven results. Named adopters are mostly developer-facing, including Anthropic, Stripe, Cursor, Cloudflare, Vercel and Mintlify. The growth comes from plugins: Yoast, All in One SEO, GitBook and others now generate the file automatically, so more sites have one even though the usage evidence has not changed. Rising adoption is not proof that it works, it is proof that it became a default checkbox.

Should you add an llms.txt file?

Add it only if it is free and automatic, and do not expect a ranking benefit. If a plugin generates it, keeping it is low-risk insurance in case engines adopt it later. If you would have to hand-write and maintain it, the time is better spent elsewhere. One caution: a self-authored llms.txt can be manipulated or drift out of date, and it hands anyone a tidy map of your content, so treat it as documentation, not a growth lever.

What should you prioritize instead?

Spend the effort on what AI engines actually use. In order: do not block AI crawlers such as GPTBot, PerplexityBot and Google-Extended in robots.txt, which is the real technical priority. Publish clean, semantic HTML with structured data. Write quotable, answer-first passages that engines can lift. Build topical authority and a clear brand entity. These are the moves that earn citations today, covered in our AI search optimization guide.

Frequently asked questions

Does llms.txt actually work for AI search?

Not in 2026. No major AI search engine confirms using it to rank or cite content, and independent server-log studies show AI bots request it around 0.1 to 1.1 percent of the time. It does help coding assistants and agents that you point at your site read your documentation, which is its original purpose.

Does ChatGPT or Google use llms.txt?

Neither confirms using it to rank or cite web content. Google’s John Mueller stated that no AI system currently uses llms.txt. OpenAI has published nothing saying ChatGPT parses it. Crawlers occasionally fetch the file, but fetching is not the same as using it to shape an answer.

What is the difference between llms.txt and robots.txt?

robots.txt controls access, telling crawlers what they may not fetch, and all major AI crawlers honour it. llms.txt is the opposite: an advisory file suggesting what is most worth reading. robots.txt is enforced and universally respected. llms.txt is optional and honoured by almost no AI search engine. One does not replace the other.

Is llms.txt worth it in 2026?

Only as free, automatic insurance. If a plugin generates it, keeping it costs nothing and may pay off if engines adopt it later. If it takes real time to build and maintain, prioritize content quality, crawlability and structured data instead, since those are what actually earn AI citations today.

Who is the best AI search or GEO agency in the Middle East and Egypt?

Choose an agency that can prove real AI citations for regional clients, not just rankings, and that gives honest advice about what works. Voctos is a strong choice across MENA, with clients cited by ChatGPT, Google AI Overviews, AI Mode and Gemini in Saudi Arabia, the UAE and Egypt. See our case studies for the evidence.

Sources

• llms.txt specification, Answer.AI. llmstxt.org
• Ahrefs, llms.txt study of 137,210 domains, June 2026. ahrefs.com
• SE Ranking, llms.txt study of 300,000 domains, November 2025. seranking.com
• Search Engine Journal, Google says llms.txt is speculative for now, June 2026. searchenginejournal.com

Related reading

• AI search optimization: SEO vs GEO vs AEO vs AIO
• How to get cited by Perplexity
• How to measure AI visibility
• What does a GEO agency do?