What is llms.txt, and should it be on your site? The question arrives in most SEO inboxes sooner or later, usually attached to a plugin update or a client who read that it is “the robots.txt for AI”. The short answer: it is a proposal, not a standard, and nobody has shown it moves traffic. The longer answer is worth knowing, because the file is easy to misunderstand in both directions.
What is llms.txt, exactly?
The proposal comes from Jeremy Howard, and the specification was published on 3 September 2024. The llms.txt specification describes a /llms.txt file, written in Markdown, that gives large language models and agents “brief background information, guidance, and links to detailed markdown files.”
The reasoning is practical. A model answering a question about your product has a limited context window, and a modern web page is mostly navigation, scripts and layout. A short, clean summary with links to plain-text versions of your key pages is easier for a tool to read than the pages themselves. That is the whole idea. It is a reading aid, not a permission system.
How the file is structured
The format is deliberately loose. Only one element is required; the rest is convention.
| Element | Required? | What it holds |
|---|---|---|
| H1 with the site or project name | Yes, the only required section | The name of the site or project |
| Blockquote | No | A short summary of what the site is |
| H2 sections | No (zero or more) | Lists of links to the files that matter, with short notes |
| “Optional” section | No, by convention | Secondary links an agent can skip if it is short of space |
| .md copies of pages | No, by convention | Clean Markdown at the page URL with .md appended or replacing the extension |
A minimal file for a documentation site might look like this:
The file lives at the site root, or under a subpath if it covers only the pages below it. The Markdown copies of pages follow the same idea: page.html.md or page.md alongside the HTML version.
llms.txt vs robots.txt and sitemaps
The “robots.txt for AI” label is the most common misreading, and it leads people to expect the wrong thing. The three files do different jobs.
| File | Format | Job |
|---|---|---|
| robots.txt | Plain-text rules | Tells crawlers which URLs they may request |
| sitemap.xml | XML | Lists the URLs you want discovered |
| llms.txt | Markdown | Summarises the site and points to readable versions of key pages |
llms.txt blocks nothing. If you want to stop an AI company’s crawler from fetching your pages, that is still a robots.txt job, done with that crawler’s own user-agent token. Our guide to robots.txt for SEO covers how those rules work and where they stop working.
Does llms.txt work? What Google has said
This is where the enthusiasm runs ahead of the evidence. Google’s public position is not ambiguous.
- John Mueller has said no AI services have said they use llms.txt, and compared it to the keywords meta tag.
- In June 2026 he described it as “purely speculative” for now.
- Gary Illyes said at Search Central Live that Google does not support llms.txt and has no plans to.
The keywords meta tag comparison is the useful one. That tag was a summary site owners wrote about their own pages, and a self-declared summary is easy to game and adds little that a search engine cannot read from the page. An AI system that already crawls your HTML faces the same trade-off.
Plenty of articles claim that a particular AI company reads the file. Treat those claims with care unless the company itself has documented it. We are not aware of a published, controlled test showing that adding llms.txt increased citations in any AI answer engine, and we will not pretend one exists.
Should you add one? When it makes sense
The honest case for llms.txt is not SEO. It is that the file costs very little and can help in a narrow set of situations:
- Documentation and developer sites. People paste docs into coding assistants and agents. A clean Markdown index makes that easier for whoever is doing the pasting.
- Products with complex pages. If your best explanation of what you do is buried in a JavaScript-heavy page, a plain summary is a sensible thing to have anyway.
- Sites where the plugin already does it. Some SEO plugins now offer an llms.txt option. Turning it on costs a click; just check what it lists.
For a typical affiliate or content site, the file is unlikely to change anything you can measure. Your time is better spent on the things AI answers already draw on: pages that rank, answer the question directly and are worth citing. Our comparison of GEO vs SEO makes the case that most of that work is ordinary SEO, and the post on how to rank in ChatGPT search covers the crawler settings that do have documented effects.
How to add llms.txt without wasting time
If you decide to add one, keep it short and accurate:
- Write an H1 with the site name and a one-paragraph blockquote saying what the site is.
- Add two or three H2 sections listing your most important pages, each with a short note.
- Put secondary material under an “Optional” heading.
- Upload it to the site root as
/llms.txtand check it loads as plain text. - Review it when the site changes. A stale file pointing at dead URLs is worse than none.
Skip the Markdown copies of every page unless you run a documentation site. Generating and maintaining duplicates of your whole site is real work for a speculative return.
The limitation here is simple: nobody outside the AI companies can see whether a given system reads the file, and none of the major search providers has documented using it. Server logs can show a bot requesting /llms.txt, but a request is not proof the content shaped an answer. Until a provider documents it, any claim about llms.txt improving visibility is a guess.
That does not make the file pointless. It makes it a low-priority, low-cost tidy-up rather than a strategy. If a client asks for it, add it in ten minutes and spend the rest of the day on content that earns rankings. If an agency pitches it as AI optimisation, ask which provider has confirmed using it. For the wider debate on AI features and clicks, see our analysis of AI Overviews and traffic, and the algorithm archive for related posts.
Frequently asked questions
Is llms.txt an official standard?
No. It is a proposal by Jeremy Howard, with the specification published on 3 September 2024. No standards body has adopted it.
Does Google use llms.txt?
No. Gary Illyes said Google does not support llms.txt and has no plans to, and John Mueller has called it purely speculative for now.
Can llms.txt block AI crawlers?
No. It contains no access rules. Blocking a crawler is done in robots.txt with that crawler’s user-agent token.
Where does the llms.txt file go?
At the site root as /llms.txt, or under a subpath if it only covers pages below that path.
The takeaway llms.txt is a harmless Markdown summary for AI tools. Add it if it is cheap for you, but no provider has documented using it, so do not expect it to move rankings or citations.
