Ruta is invented for this example. She runs a three-page site from a desk that also holds the kettle. Someone told her a file called llms.txt is how the chatbots find you. She pictures a small bell. She will hang the file on the door, and the models will ring it.
The file at /llms.txt is on the site. The 13 November date is the commit that listed the seven articles. I have not read a request log, and I did not time the edit. The other claims are from pages I opened on 16 November 2026.
In plain words
llms.txt is a markdown note a site puts at a known address, naming the pages the author would rather a model read. On 13 November 2026 the copy on this site gained the seven live articles. Google’s Search guide says Google Search does not use that kind of file, and that keeping one will neither help nor harm Search. Other companies publish one for their own docs, or generate one for docs they host. The OpenAI and Anthropic pages that explain their crawlers talk about robots.txt. Neither page says the crawler goes looking for llms.txt on someone else’s site. I would budget about ten minutes for a list this size. I did not time it, and I have no count of requests.
What the proposal specifies
Jeremy Howard wrote it. The announcement on Answer.AI is dated 3 September 2024. The page at llmstxt.org is titled “The /llms.txt file, v2” and shows that date and 10 August 2026.
The file is named llms.txt, at /llms.txt or under a path such as /docs/llms.txt, and it covers the URLs under that path. The only required section is an H1 with the name of the project or site. After that, in order: a blockquote with a short summary, any further detail that is not a heading, then zero or more H2 sections. Each H2 section holds a file list. Each item is a markdown link, [name](url), optionally followed by a colon and a note. An H2 titled Optional is the convention for links an agent can skip.
The same page keeps the file apart from a sitemap, which lists pages for search engines, and apart from robots.txt, which says what automated access is acceptable. Its line on use is that an agent opens llms.txt on demand, when it already needs something from the site. That is the proposer’s expectation. It is not a crawler signing on. Links are supposed to point at something a model can read cleanly, such as a markdown copy of the page.
I requested the index.md address under Three switches before an agent acts. The response was 404. The articles here are the HTML pages. The list points at those.
The file on this site
/llms.txt has the H1, which is the required part. Under it are a few sentences, then seven lines: a title, a dash, and the URL written out. There is no blockquote. The seven lines are not under an H2, and they are not [title](url) links. The spec’s file list is that tighter shape. I have not rewritten the file into it.
A person can still read the list. This draft is not on it. The file names published articles, and a draft is not one.
Who has said what
| Page | What that page says |
|---|---|
| llmstxt.org | The format above, and the expectation that an agent opens the file when it needs the site |
| Google’s Search guide | Search does not use these files. Keeping one “will neither harm nor help” Search visibility or rankings, “as Google Search ignores them.” It also names “AI text files (like llms.txt)” among tactics to ignore. Last updated 2026-07-10 UTC |
| Lighthouse’s llms.txt audit | Lighthouse requests /llms.txt. A server error is flagged. A 404 is Not Applicable, because the file is optional |
| OpenAI’s crawler overview | How OAI-SearchBot, GPTBot, OAI-AdsBot, and ChatGPT-User relate to robots.txt. The page points at OpenAI’s own docs index |
| Anthropic’s crawler article | ClaudeBot, Claude-User, and Claude-SearchBot, and how robots.txt limits them. llms.txt is not named |
| Cloudflare’s style guide | Cloudflare publishes llms.txt and llms-full.txt for its own documentation |
| Mintlify’s docs | Mintlify hosts one for a docs project, “so AI tools like ChatGPT and Claude can index and understand your documentation” |
I opened three of those own-docs indexes. OpenAI’s has an H1, a blockquote, and markdown links into the API docs. Anthropic’s is an overview of Anthropic’s developer documentation. The Gemini API file is the same kind of index. A map on a company’s own docs is a different sentence from “our crawler fetches yours.”
Mintlify’s line about ChatGPT and Claude is Mintlify’s reason for generating the file. I did not find it on the OpenAI or Anthropic crawler pages. Lighthouse requesting the file, and treating a missing one as optional, is also different from Google Search using it.
What a log would have to show
A fetch of the text file does not render an article, so a counter inside an HTML page never sees it. The record, if there is one, is the host’s request log.
This site is on Cloudflare. In the HTTP requests dataset, ClientRequestPath is the path without the query string, and ClientRequestUserAgent is the user agent the client sent. Logpush can export that dataset. The page lists it as available on the Free plan as well as the paid plans, and says the export cannot backfill. Requests from before a job existed are gone.
- A client requests /llms.txt
- The path and the user agent are fields in the request log
- That export is not running here, so there is no count
Google’s guide says Search will not treat the file as a ranking input. For anyone else, I have a method and an empty hand.
Why it is up
The seven titles are the site. One file is an easy place to keep that list true, drafts excluded. If I hand an agent a single address, I would rather hand it this than the navigation. The blockquote and the bracketed links can wait. The list is already readable.
Ruta’s bell was never on the door. The note was. About ten minutes is the budget I would give a heading and seven links. It is not a time I recorded. A later log that shows a crawler asking for the path belongs in another post. This one stops at the list.
Sources
- Jeremy Howard, The /llms.txt file, v2, llmstxt.org, accessed 16 November 2026. The page shows 3 September 2024 and 10 August 2026.
- Answer.AI, the 3 September 2024 announcement, accessed 16 November 2026.
- Google’s generative AI Search guide, last updated 2026-07-10 UTC, accessed 16 November 2026.
- Chrome, Lighthouse llms.txt audit, accessed 16 November 2026.
- OpenAI, Overview of OpenAI Crawlers, accessed 16 November 2026.
- OpenAI API docs, llms.txt, accessed 16 November 2026.
- Anthropic’s crawler article, accessed 16 November 2026.
- Anthropic docs, llms.txt, accessed 16 November 2026.
- Gemini API docs, llms.txt, accessed 16 November 2026.
- Cloudflare, AI consumability, accessed 16 November 2026.
- Cloudflare, HTTP request log fields, accessed 16 November 2026.
- Cloudflare Logpush, accessed 16 November 2026.
- Mintlify, llms.txt, accessed 16 November 2026.
- runapa.com/llms.txt, accessed 16 November 2026.