llms.txt: We Scanned the Web's Top 100,000 Domains

Across the web's top 100,000 domains, 8,598 publish an llms.txt file: 8.6% of the full list, and 11.4% of the 75,353 domains that answered us at all. We requested the file from every one of them on 13 September 2026. Nearly a third of the files we found were generated automatically by the site's platform.

Key findings from our 13 September 2026 census

• 8,598 of 100,000 top domains publish an llms.txt (8.6% of the population, 11.4% of domains that responded).
• More domains fake it than have it. 10,775 returned an ordinary HTML page at /llms.txt with a 200 status. A checker that only looks for "200 OK" overcounts adoption by more than double.
• At least 29.8% of the files carry a platform signature. Shopify alone accounts for 18.2% of every llms.txt we found; WordPress SEO plugins add another 9.6%.
• The files are well-formed and often empty. 95.1% open with an H1 exactly as the spec requires, but 16.4% contain no links at all.
• .ai domains adopt at 39.3%, against 11.4% overall. The people building AI are the ones publishing files for it.
• 186 domains serve their robots.txt at /llms.txt.

Every page currently ranking for this keyword explains what the format is. I wanted to know something the spec can't tell you: whether anyone is actually using it, and who.

How many of the top 100,000 domains actually have an llms.txt?

8,598 do — 8.6% of the top 100,000 domains, measured on 13 September 2026. Narrow the denominator to the 75,353 domains that returned any usable response, and the rate is 11.4%. The remaining 24,647 timed out, failed DNS, or refused the connection, so we excluded them rather than scoring them as absent.

That's the first denominated, current number I'm aware of for this format. The figure most often repeated is a count of 951 domains published by NerdyData in July 2025 — an absolute count with no stated population, now fourteen months old. Ours is a rate against a named list on a named date, which is the only shape that lets you say whether adoption is big or small.

Adoption is small, but it isn't nothing. 8,598 domains is an order of magnitude above the last public estimate, and the format is barely two years old.

Who is really publishing llms.txt, and why it is mostly not a decision

Here's where the number stops meaning what you'd assume. At least 29.8% of every llms.txt file we found carries an identifiable platform or plugin signature — text in the file itself naming the tool that wrote it.

Shopify by itself wrote nearly one in five of the llms.txt files on the web's biggest domains. Add the three WordPress SEO plugins and you're at 27.8% from four vendors.

I want to be careful about the 70.2% with no signature, because it's the easiest number in this study to misread. It is not a measure of hand-written files. Most generators don't stamp their output at all, so that bucket mixes deliberate authorship with unsigned automation. Treat 29.8% as a floor on platform-generated adoption, not an estimate of it.

Google demonstrated this pattern on itself. As Search Engine Roundtable reported, llms.txt files appeared across Google's developer documentation because the CMS Google uses began supporting the format, not because a team chose to adopt it. The Search team then removed it from its own docs while other teams left theirs up. Asked on Bluesky whether the file at ai.google.dev was an endorsement, John Mueller answered: "I'm tempted to say something snarky since this has come up so often, but to be direct, no."

That file is still there. I checked on 13 September 2026: ai.google.dev/api/llms.txt returns 200 with a text/markdown content type and a properly formed link map.

Which corners of the web actually adopted it?

Splitting adoption by top-level domain gives the clearest picture of who this format belongs to right now. Against an overall rate of 11.4% among domains that responded, the spread is wide.

Two in five .ai domains publish an llms.txt. That's 3.4× the overall rate, with .io close behind at 24.9%. At the other end, .net sits at 5.5% and the large European and Japanese country domains cluster between 6% and 9%.

Read that table as a map of an in-group. The companies building AI products, and the developer-tooling crowd on .io, have adopted a convention that the rest of the web has largely ignored. That's consistent with where the file demonstrably gets served: documentation and API references, written by people who talk to agents for a living.

Are the published llms.txt files any good?

Mostly they're well-formed. 95.1% of the 8,598 files open with an H1 title, which is the one structural rule the llms.txt specification is strict about. Generators are good at that part.

Substance is thinner. 1,407 files — 16.4% — contain no markdown links whatsoever. The format exists to be a curated map pointing an agent at a site's best content. A map with no destinations on it is decoration.

The typical file is reasonable, though:

What about llms-full.txt?

The companion convention is llms-full.txt, which is meant to hold a site's actual content rather than a list of links to it. We scanned all 8,598 domains a second time to check.

2,516 of them — 29.3% — publish one as well. That's 2.5% of the top 100,000 overall, so roughly one in three sites that bother with this format goes the extra step.

Then we compared the two files on each domain, and the result is the sharpest quality signal in the study. A "full" file should dwarf the index that points into it. Instead:

Half the sites publishing both files are serving essentially the same file twice. The median llms-full.txt we measured is 4,521 bytes, against 4,542 bytes for the median llms.txt — two names for one document. Only 22.6% contain the substantially larger body of content the convention implies.

Does anything actually read llms.txt?

No major AI vendor has documented consuming it, and Google has said so directly. Mueller has also argued that LLM systems can't use llms.txt to tell websites apart during discovery, seeing a role only once an agent is already on a site — a position Search Engine Journal covered in detail.

The counter-evidence people reach for is that AI companies publish the file themselves. That's true, with a detail worth knowing. On 13 September 2026 I checked the obvious candidates: docs.anthropic.com, docs.perplexity.ai and developers.cloudflare.com all return a live llms.txt. Their root domains don't — anthropic.com returns 404, and so does platform.openai.com.

So the pattern is documentation sites serving structured markdown to agents that arrive at documentation. That's a real use case. It is not evidence that a search or answer engine reads the file to decide what to cite.

The format does have a foothold in tooling. Chrome's Lighthouse ships an llms.txt audit under its agentic-browsing checks, which means teams now get nudged toward the file by a default developer tool regardless of whether any engine consumes it.

llms.txt vs robots.txt: the confusion that shows up in the data

These two files get discussed together constantly, and 186 domains in our scan took that literally: they serve their robots.txt at /llms.txt. The response opens with User-agent: and Disallow: lines instead of markdown.

186 out of 100,000 is a rounding error. I'm reporting it because it's a clean, countable signal of a confusion that shows up everywhere in the discourse, and because the two files do genuinely opposite jobs.

• robots.txt is an access directive. It tells crawlers what they may request, and crawlers obey it.
• llms.txt is a content map. It suggests what's worth reading, and nothing is obliged to look.

One is enforced. The other is a recommendation nobody has agreed to take. If you only have time to get one of them right, it isn't the new one — you can sanity-check yours with our robots.txt tester.

The bigger measurement trap in this space is the soft 404. 10,775 domains returned an ordinary HTML page, with a 200 status, at /llms.txt — more than the 8,598 that actually publish one. Plenty of sites answer 200 for any path you invent. Any adoption study that counts status codes without inspecting the body will report a number more than twice too high.

What our audit data says about AI readiness beyond llms.txt

llms.txt is a narrow slice of a much wider gap. Our audit engine scores an AI/GEO readiness category across every site it runs, and that category fails on 54.6% of sites audited — covering AI bots blocked by default, weak semantic HTML, and missing structured content.

The direction of travel is genuinely positive, which I think gets lost in the cynicism about this topic. The share of sites failing AI/GEO readiness fell from 58.1% in January 2026 to 53.9% by June, making it the fastest-improving category we measure.

Set that beside our July 2026 agent-readiness scan of 109,440 top domains and the shape of the problem is clear:

Effort predicts adoption almost perfectly. Sites do the cheap declarative things and skip the work. GPTBot and ClaudeBot are the two most-named agents in the robots.txt files we parse, which tells you where attention actually goes: deciding who gets in, not preparing anything for them once they're through the door.

If you want the wider picture on what those crawlers take versus what they send back, we've measured the crawl-to-referral ratio separately, and our AI SEO statistics roundup tracks the trend lines.

Should you add an llms.txt file? An honest decision framework

Our own data doesn't support telling you to rush. It also doesn't support telling you not to bother. The cost is close to zero and the evidence of benefit is close to zero, so the answer depends on what kind of site you run.

Add one if: you publish documentation, an API reference, or a large technical knowledge base. This is the case where agents already arrive with a specific question, where the file is demonstrably being served by vendors who know the space, and where you probably have the markdown already.

Let your platform handle it if: you're on Shopify, Wix, HubSpot, or WordPress with an SEO plugin. There's a good chance you already have one. Check before you write anything by hand.

Skip it if: you'd be hand-maintaining a link list that goes stale in a month. A wrong map is worse than no map, and 16.4% of the files we found already list nothing at all.

Whatever you decide, don't let it substitute for the work. A file no engine has committed to reading is not a substitute for content an engine can parse, quote and attribute.

If you do publish one, here's what the data says good looks like

We measured 8,598 real files, so the shape of a normal one isn't guesswork. The median lists 15 links across 7 sections in 4,542 bytes. That's the target to beat, and it's modest — an afternoon of work, not a project.

Four things separate the useful files from the decorative ones:

1. Put links in it. This sounds obvious until you see that 1,407 published files contain none. The format is a map. A map with no destinations is a title card.
2. Open with an H1, then a blockquote summary. 95.1% of files get the H1 right, and it's the one thing the spec insists on. The summary line underneath is what tells an agent what your site actually is.
3. Point at your best content, not your whole sitemap. The value of a curated list is the curation. If you're going to list every URL you have, you already have a sitemap for that, and our site crawler will generate a more accurate one.
4. Skip llms-full.txt unless you mean it. Half the sites publishing both are serving near-duplicates. If your "full" file isn't substantially larger than your index, it's adding a maintenance burden and nothing else.

Then diary it. A link map is a snapshot of a site that keeps changing, and a stale map is the one failure mode that can actively mislead an agent that does read it.

How to check a site properly

If you're auditing your own site or a competitor's, don't trust a green checkmark. Our soft-404 count is the reason: more top domains return an HTML page at this path than a real file, so any check that stops at the status code is wrong more often than it's right.

Request the URL and read the first line of the body. A real llms.txt starts with # and a site name. If you get <!DOCTYPE html>, the site doesn't have one. If you get User-agent:, somebody wired up the wrong file — which 186 domains in our census have done.

For the signals that AI crawlers demonstrably act on, the GEO audit checks the readiness category our audit engine scores, and a free SEO audit covers the rendering and structure problems sitting underneath it.

How we ran this scan

The population is the 100,000 domains in Cloudflare Radar's top-100k ranking dataset, target window 31 August to 7 September 2026. We issued one HTTPS GET per domain to https://{domain}/llms.txt on 13 September 2026, following up to three redirects with a 25-second timeout and a self-identifying user agent.

Classifying the response mattered more than fetching it. 20,171 domains returned a 200 at this path. Only 8,598 of them had actually published a file — a 2.35× overcount if you stop at the status line. We counted a file as real only when it returned 200 and the body was not HTML, not JSON or JavaScript, not a bot-challenge interstitial, not a robots.txt, and not shorter than 20 bytes.

Limitations worth stating plainly:

• We requested the root path only. A file on a documentation subdomain counts as absent. That's not hypothetical: anthropic.com returns 404 while docs.anthropic.com returns 200. True adoption among large organisations is higher than our root-domain rate.
• This is one snapshot, taken on a single day. It's a level, not a trend.
• 24,647 domains never responded. They're excluded from the 11.4% figure rather than counted as missing.
• Generator attribution is a floor. We can only detect generators that sign their output.
• The population is traffic-ranked, so it includes CDN and API hosts that were never meant to serve content to a reader.

Corroborating figures come from SEOmator's crawling and audit data, January to July 2026, across 100,000+ sites audited and 50M+ pages crawled. Those aggregates reflect our customer mix, which runs roughly 60% B2B SaaS.

Frequently asked questions about llms.txt

How many websites have an llms.txt file? 8,598 of the web's top 100,000 domains, or 8.6%, as of 13 September 2026. Against the 75,353 domains that responded to our request, the rate is 11.4%.

Does llms.txt help SEO? There's no evidence it affects Google rankings or AI Overviews. Google has stated it doesn't use the file, and no major AI vendor has documented consuming it for discovery.

Do ChatGPT, Claude or Gemini read llms.txt? None has documented doing so. They publish llms.txt on their own documentation sites, which is a different thing from consuming yours.

Is llms.txt the same as robots.txt? No. robots.txt is an enforced access directive that crawlers obey. llms.txt is an optional markdown map of your content that nothing is obliged to read. In our scan, 186 domains served their robots.txt at the llms.txt path.

Which platforms create llms.txt automatically? Shopify wrote 18.2% of every file we found. Yoast, All in One SEO and Rank Math together account for 9.6%, with Wix and HubSpot adding about 1.5% between them.

What should an llms.txt file contain? An H1 title, an optional blockquote summary, then H2 sections of markdown links to your best content. The links are the point — 16.4% of published files have none.

Can llms.txt hurt my site? Not directly. The risk is a stale link list that misrepresents your content, and the opportunity cost of treating the file as AI-search work when it isn't.

How do I check whether my site has one? Request https://yourdomain.com/llms.txt and read the body. Don't trust the status code — more top domains return an HTML page at that path than a real file.

Why does Google say llms.txt isn't used? Mueller's argument is that an engine can't use the file to differentiate between sites during discovery, since it only helps once an agent has already arrived.

What is llms-full.txt, and do I need one? It's the companion file meant to hold your actual content rather than links to it. 29.3% of the domains that publish an llms.txt also publish one, but half of those serve a near-duplicate of their llms.txt rather than anything fuller. Only add it if you genuinely have a larger body of content to serve.

Which types of sites adopt llms.txt most? Technical ones. .ai domains adopt at 39.3% and .io at 24.9%, against 11.4% across all responding domains. .net sits at 5.5%, and the major European and Japanese country domains run between 6% and 9%.

Is adoption growing? Our 8.6% is well above the 951-domain count reported in July 2025, but the two aren't directly comparable and much of the growth is platform defaults rather than deliberate adoption. We'll re-run this census to measure the trend properly.

What I'd do next

The interesting finding here isn't the adoption rate. It's that a third of it, at minimum, was switched on by somebody else — and that more sites accidentally return an HTML page at this path than deliberately publish a file.

If llms.txt is on your roadmap, spend ten minutes confirming your platform hasn't already shipped one, and spend the rest of the time on the things AI crawlers demonstrably do read. Our site crawler will show you what a rendering crawler actually sees on your pages, which is the layer this format sits on top of.

We'll re-run this census and publish the delta.