In the 30 days before we published /llms.txt, 27 requests asked for it. Every one came from an SEO or domain-intelligence tool. Not one came from ChatGPT, Claude, Perplexity or Gemini.
We published it anyway. What we measured, what the larger studies say, and why a file nobody important reads was still worth thirty minutes.
What llms.txt is
A proposal from September 2024: a Markdown file at the site root that lists your important pages, one line of description each, written for language models rather than browsers. Where sitemap.xml lists everything for a crawler, llms.txt says which pages an agent should read first. Documentation platforms generate it by default now, which is why Anthropic, Stripe and Cloudflare have one. SEO tools flag it as "missing", which is how it ends up on the to-do list of a hosting company with a few dozen pages.
Who asked for it on our site
Our access logs go to Loki, so this is one query. The 30 days before 2 September 2026, on coffeesprout.nl: 27 requests for /llms.txt, 26 answered with a 404 because the file didn't exist yet and one with a 403. The ones that identified themselves: Semrush's SiteAuditBot, SEOJuice, BuiltWith, DomainEvidenceBot, PipericBot, OgpnBot. The rest were anonymous scanners with browser user agents, several from the same proxy network. One tried /.well-known/llms.txt for good measure.
GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot: zero. Not because they don't visit. Same window, same site, counting only requests that got a page back (status 200):
| User agent | Pages served in 30 days | What it fetched |
|---|---|---|
| ClaudeBot | 840 | Crawl |
| OAI-SearchBot | 294 | Crawl |
| Claude-User | 269 | Four technical articles, plus robots.txt |
| ChatGPT-User | 218 | Homepage and product pages (with a footnote, below) |
| GPTBot | 137 | Crawl |
| PerplexityBot | 60 | Crawl |
Look at the Claude-User row. That's the fetch Claude makes when someone asks it a question, and four out of five of those pages were one of four articles: Docker 29 breaking Testcontainers (130), Ryuk and zombie containers (61), the JDK 25 AOT cache crash (18) and Januscape (9). Someone hits the error, asks Claude, Claude fetches our write-up. It found the page through search, the same way a person would, and never asked for an index.
The ChatGPT-User row needs its footnote, because the raw line count is misleading: 8,836 lines carry that user agent, and only 218 of them were served a page. The rest are 6,358 redirects, 1,531 blocked requests and 618 not-founds, a large part of them a scanner that uses the ChatGPT-User string to probe /actuator/env, /config/master.key and /.s3cfg with the referrer set to our raw IP address. By raw lines, ChatGPT is our biggest reader by far. By pages served, it fetches the homepage or a product page about once every three hours. Someone asked ChatGPT about us and it went to read.
What the bigger studies found
- Google, twice. John Mueller in June 2025: "no AI system currently uses llms.txt". Gary Illyes in July 2025: Google won't crawl it and won't use it for AI Overviews.
- Ahrefs, May 2026, 137,000 domains. 28% publish the file. 97% of those files got zero requests that month. Of the requests that did arrive, 96% were bots and 77% were SEO auditors. Retrieval bots such as OAI-SearchBot and PerplexityBot were about 1%.
- SE Ranking, 300,000 domains. No correlation between having the file and being cited by AI assistants. Removing the variable improved their model.
- Evil Martians, May to July 2026, 268,000 agent requests. About 660 fetches of
llms.txt, 37 from named AI assistants. ChatGPT fetched HTML almost exclusively. Claude Code sentAccept: text/markdownand got Markdown back 76% of the time.
The one agent population that behaves differently doesn't read an index file. It asks for Markdown with an HTTP header that has existed since 1997, on the page it already wanted.
Why we added it anyway
- Thirty minutes. Hand-written, about fifty links, English and Dutch sections, the "from" price in every product description.
- The one real use case. Someone pastes coffeesprout.nl into Claude or ChatGPT and asks what we do and what it costs. One fetch now answers that, with numbers. We publish our prices for the same reason: an assistant can only quote what is on the page.
- The audits stop nagging. Every SEO report a prospect's agency runs on us flagged it as missing. Now it doesn't.
Two rules we set for ourselves. Short and hand-maintained: we've watched an agent consult a large vendor's llms.txt, find no entry for the page it needed, and fall back to reading source on GitHub. An index that is incomplete is worse than no index, because the agent trusts it. And noindex: the file carries an X-Robots-Tag header so it never shows up as a search result.
What we'll do instead
Plain HTML that reads as text without JavaScript, which this site already is. Markdown versions of each page served through content negotiation when we rebuild the site, because that is what the coding agents ask for. And the same Loki query again in November. If the only readers are still SEO tools, the file stays as hygiene and this is the last time we think about it.
The file took thirty minutes. This post took longer.
Takeaways
llms.txthelps an agent that a person pointed at your site. The assistants that answer questions don't read it, and none of the big vendors has said they will.- Your own logs settle it in one query: filter on
llms(-full)?\.txtand group by user agent. Count status 200, not lines. Redirects and scanner probes inflate every agent several times over. - AI assistants find your pages the way people do: a specific page for a specific error, through search.
- Coding agents send
Accept: text/markdown. Answer that header before you write an index file. - If you add the file: keep it short, write it by hand, put the prices in, set
noindex.
References
- llmstxt.org: the proposal and file format.
- Search Engine Roundtable: John Mueller, "no AI system currently uses llms.txt".
- Search Engine Land: Gary Illyes on AI Overviews and llms.txt.
- Does llms.txt actually work?: summary of the Ahrefs and SE Ranking studies.
- Evil Martians: which AI actually reads your site: two months of measured agent traffic, including the
Accept: text/markdownfinding. - Our llms.txt: the file itself.
Running a Java platform and want it run by people who check the logs first? Talk to us about managed Java hosting.