freeseobot

llms.txt Explained: What It Is and Do You Need One?

What llms.txt is, who proposed it, which systems actually read it, what Google says about it, and a copy-paste template if you decide to publish one.

By freeseobot Editorial · Published · 6 min read

llms.txt is a proposed Markdown file at yourdomain.com/llms.txt that gives AI tools a short summary of your site and a curated list of links to your most useful pages. Google has stated that Google Search does not use it, and that having one will neither help nor harm your visibility. None of the crawler documentation published by OpenAI, Anthropic or Perplexity mentions it either. For most small business websites it is optional: worth ten minutes if you have documentation, an API or a large knowledge base, and safe to skip otherwise.

What is llms.txt?

The proposal was published by Jeremy Howard of Answer.AI on 3 September 2024, and is maintained at llmstxt.org, where a revised second version is now live. The idea: AI models have limited "context windows" (how much text they can read at once), and full web pages are cluttered with navigation, scripts and adverts. A small, clean Markdown file pointing to the important content could help a model find what it needs.

It is a proposal, not a standard from a standards body, and no crawler is required to read it.

What goes in an llms.txt file?

The format is plain Markdown in a fixed order:

Part Required? What it contains
H1 heading (# Name) Yes, the only required part Your site or project name
Blockquote (> ...) Recommended A one or two sentence summary
Free text Optional Any extra context, as paragraphs or lists (no headings)
H2 sections (## Docs) Optional Lists of links, each as [name](url) with an optional : note
## Optional section Optional Secondary links a tool can skip when it is short of space

The specification also suggests offering clean Markdown copies of key pages at the same URL with .md added (for example /pricing.md), and, in its current version, pointing to them with rel="alternate" type="text/markdown" links.

Who actually reads llms.txt?

This is the part most articles skip. Here is what the providers themselves document:

Who What they say
Google Search, including AI Overviews and AI Mode Google Search "itself doesn't use them", and creating one "will neither harm nor help" visibility or rankings
OpenAI (GPTBot, OAI-SearchBot) Crawler documentation covers robots.txt; llms.txt is not mentioned
Anthropic (ClaudeBot, Claude-SearchBot) Crawler documentation covers robots.txt; llms.txt is not mentioned
Perplexity (PerplexityBot) Crawler documentation covers robots.txt; llms.txt is not mentioned
Google Lighthouse An experimental "Agentic Browsing" category includes an llms.txt check aimed at AI agents. A missing file is marked "Not Applicable", not a failure

Two practical uses do exist:

  • Coding assistants and developer tools. The proposal is aimed at use when someone asks an AI tool a question (inference), not at training. Giving an assistant a project's documentation is the typical case, which is why developer documentation sites from Anthropic, OpenAI and Perplexity publish one.
  • AI agents in the browser. Chrome's Lighthouse documentation describes llms.txt as an emerging convention that saves agents time working out a site's structure. The category is experimental and requires Chrome 150 or later.

So the accurate summary: llms.txt helps tools that are pointed at it deliberately. There is no public evidence from the major AI search providers that it affects whether you are crawled, ranked or cited.

What does Google say exactly?

Google's guide to optimizing for generative AI features, published in May 2026 and clarified on 15 June 2026, lists llms.txt under things you can ignore for Google Search. In summary:

  • You do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its AI features.
  • Google may discover and crawl many file types, but that "doesn't mean that the file is treated in a special way".
  • It is "completely fine" to maintain llms.txt for other services that use it.

Should your business have one?

Your site Recommendation
Local service business, 5 to 30 pages Optional. Spend the time on robots.txt, indexing and clear page content first
Online shop Optional. Product feeds and structured data matter more
SaaS product or app with documentation Worth doing. Developers and coding assistants use it
Large knowledge base or publisher Worth testing if you can keep it in sync with your content

If you publish one, keep it accurate. An out-of-date file listing deleted pages is worse than none.

Copy-paste llms.txt template

Save this as plain text at the root of your domain, so it loads at https://yourdomain.com/llms.txt:

# Hartley Heating

> Gas Safe registered boiler installation, servicing and repair for homes in Leeds, West Yorkshire. Fixed prices, same-week appointments.

We cover LS1 to LS29. We do not service commercial boilers or oil-fired systems.

## Services

- [Boiler servicing](https://www.hartleyheating.example/boiler-servicing): Annual service, fixed £79 price, what is included
- [Boiler installation](https://www.hartleyheating.example/boiler-installation): Combi and system boilers, quotes, finance options
- [Emergency repairs](https://www.hartleyheating.example/repairs): Call-out times and charges

## Company

- [About us](https://www.hartleyheating.example/about): Team, qualifications, Gas Safe number
- [Contact](https://www.hartleyheating.example/contact): Phone, email, opening hours, service area

## Optional

- [Blog](https://www.hartleyheating.example/blog): Maintenance guides and seasonal advice

Tips:

  • Use Markdown link syntax ([name](url)), as the specification requires.
  • Serve it as plain text with a 200 status.
  • Link only to pages you want found, and make sure they are not blocked in robots.txt.

llms.txt versus robots.txt

They do different jobs, and confusing them is common:

robots.txt llms.txt
Purpose Tells crawlers what they may request Suggests what is most useful to read
Standard IETF RFC 9309 Community proposal
Followed by Google, OpenAI, Anthropic, Perplexity crawlers Yes, documented by each Not documented by any of them
Can block AI training Yes, via GPTBot, ClaudeBot, Google-Extended No

If you want to control which AI tools can use your content, robots.txt is the file that matters. Our guide to robots.txt for AI crawlers has copy-paste rules, and our AI visibility checker shows which AI crawlers your current file allows. For what actually earns citations, read GEO explained.

FAQ

Will an llms.txt file get me cited by ChatGPT or Perplexity?

There is no documentation from OpenAI or Perplexity saying their search crawlers use it. Citations in their answers depend on their search crawlers (OAI-SearchBot, PerplexityBot) being allowed and your pages being useful answers.

Can an llms.txt file hurt my SEO?

Google says it will neither harm nor help your visibility or rankings in Google Search, as Google Search ignores it. The only real risk is an inaccurate file pointing tools to outdated pages.

What is llms-full.txt?

A convention some documentation sites use: a single file containing the full text of their docs, rather than links to them. It is not part of the core llmstxt.org proposal, and it can get very large.

Can I use llms.txt to block AI crawlers?

No. It has no blocking function. Use robots.txt rules for GPTBot, ClaudeBot, Google-Extended and the other user agents you want to restrict.

Sources

More guides