Free LLMs.txt Generator: The File AI Agents Actually Fetch

An llms.txt file is a markdown document at the root of a website — yoursite.com/llms.txt — that summarizes what the site is and links to its most useful pages, each with a short description. Jeremy Howard of Answer.AI proposed the format in September 2024 to solve a specific problem: a language model handed a normal web page gets navigation, cookie banners, ads and script tags along with the content, and has a limited context window to spend on the mess.

The structure is deliberately small:

• an H1 with the site or product name
• a blockquote of one or two sentences saying what it does
• H2 sections grouping links, each link followed by a description of what that page covers

That's the whole specification. An optional llms-full.txt carries the expanded content of those same pages for cases where a model should read the material rather than navigate to it.

It is worth being precise about what the file is not. It does not control access — that is robots.txt, and it is enforced. It does not drive indexing — that is sitemap.xml. It is a summary written for a reader that prefers text, published at a predictable address.

Not for the reason most people ship it. Two years after the proposal, the evidence is one-directional and worth stating plainly, because almost every page selling a generator declines to.

Google does not use it. Google updated its AI-optimization guidance in June 2026 to state that llms.txt has no effect — positive or negative — on Search rankings or AI Mode. John Mueller has described the standard as speculative more than once, and has pointed at WebMCP as the direction Google finds more promising.

Almost nothing fetches it. Ahrefs analyzed 137,000 sites and found roughly 97% of published llms.txt files are never requested by AI crawlers at all. A file nothing fetches cannot influence anything downstream of the fetch.

No provider has confirmed reading it. Plenty of well-known companies publish one — Anthropic, Cursor and Mux among them. Publishing is visible and cheap. What no major AI provider has documented is that its models or crawlers retrieve and use the file when answering a question. Those are different claims, and coverage of llms.txt routinely blurs them.

So the honest summary: llms.txt is a real, published format with real adopters on the publishing side and unconfirmed support on the consuming side. If your goal is ranking or citation, this file is not the lever.

The case for shipping one anyway

There is one use that survives all of the above, and it is the reason our own generator exists.

When a person or an agent is handed your URL — a developer pasting your docs link into a coding assistant, a researcher pointing a model at your site, an agent following a link a user gave it — /llms.txt is a predictable address that answers with your site as text. No rendering, no navigation, no cookie interstitial. That is a small, concrete benefit, available today, that does not depend on anyone adopting a standard.

It shows up most for documentation, API references and developer tools, which is exactly the profile of the companies that publish one. If you run a docs site, this is twenty minutes well spent. If you run a local services site, the same twenty minutes in your page titles will do more.

This is where our own data says something the coverage doesn't. We ran the SEOmator audit engine's AI-readiness checks across 109,440 top domains on 20 July 2026:

Source: SEOmator crawling data, scan of 2026-07-20 (109,440 top domains).

Four out of five sites have written rules telling AI agents what they may and may not do. Fewer than one in fifteen serve those agents anything readable when they comply. The web is writing rules for AI agents it isn't ready to serve.

That gap is the honest case for llms.txt, and it is a narrower case than the marketing around the file suggests. Publishing one moves you into the 6.6% for the cost of an afternoon. It does not, on its own, make you citable.

The same scan's robots.txt parsing is worth reading alongside it. Across 4,257 robots.txt files parsed on the same date, GPTBot appeared in 773 and ClaudeBot in 629 — the two most-named agents on the web. AI opt-out is now the single most common reason a site writes crawler rules at all. Those rules are enforced by every major crawler. llms.txt is not enforced by anyone. Spend your attention accordingly, and check what yours currently says with the robots.txt tester.

The generator at the top of this page is sitemap-driven rather than a blind crawl, which matters for interpreting what comes back.

1. Enter a domain. A bare domain is enough — example.com. You don't need to own the site or verify anything.
2. The sitemap gets discovered. robots.txt is read first for a sitemap declaration, then the common paths are tried, then any index sitemap is expanded into its children.
3. Pages are fetched and summarized. Up to 5,000 URLs per domain are queued. Each page's text is passed to a language model that writes one title and one description for it.
4. Progress streams URL by URL. Failures are named individually rather than dropped, and you can cancel a run mid-flight.
5. Edit, then take the file. The output is editable in place. Download it or copy it to the clipboard.
6. Publish it at the root. Upload it so it resolves at yoursite.com/llms.txt, served as plain text.

A thin or empty result is diagnostic, not a bug: it usually means the domain has no discoverable sitemap. That is worth fixing regardless of what you think of llms.txt, since it affects the crawlers that demonstrably do matter. The sitemap finder will tell you what's discoverable.

What a good file looks like

# Acme Docs

> API and SDK documentation for Acme, a payments platform for marketplaces.

## Getting started
- [Quickstart](https://acme.dev/docs/quickstart): create an account, get a key, make a first charge in about ten minutes.
- [Authentication](https://acme.dev/docs/auth): key types, rotation, and the difference between test and live mode.

## API reference
- [Payments API](https://acme.dev/docs/api/payments): create, capture, refund and dispute endpoints with request and response shapes.
- [Webhooks](https://acme.dev/docs/api/webhooks): event types, retry behaviour, and signature verification.

## Optional
- [Changelog](https://acme.dev/changelog): dated list of API changes and deprecations.

Note what is missing: no tag archives, no paginated listings, no marketing pages, no "Solutions" hub. Roughly a dozen entries, each described in a sentence that says what the reader will find.

Four patterns account for most of the bad files we see.

Shipping the whole sitemap. The generator will queue every URL it finds. That's the raw material, not the file. An index that points at 2,000 pages including your tag archives is a sitemap with different punctuation — it gives a model no signal about what matters. Cut it to the pages you'd genuinely want quoted.

Leaving the generated summaries untouched. The descriptions are written from each page's own text, which makes them accurate and a little flat. For your top twenty pages, write the line yourself. That's where the file stops being mechanical.

Not checking that it's reachable. More than one in three crawler requests on the sites we monitor was rejected outright by June 2026 — a 36.4% 4xx rate across our panel. A WAF or bot-management rule configured to block unfamiliar user agents will block the agents you wrote the file for. Fetch your own /llms.txt with a non-browser user agent before assuming it is live.

Letting it rot. A stale index pointing at 404s is worse than no index: every dead link in it wastes the one request you were hoping for. Re-generate when the structure changes — a new docs section, a renamed product, a batch of retired pages. Quarterly is a reasonable default for a stable site.

If the actual goal is showing up in AI answers, llms.txt is a footnote. Three things matter more, in order.

Access. Decide per bot purpose rather than with a blanket rule. Training crawlers, retrieval bots and user-triggered agents are three different things: blocking the retrieval and user-triggered agents removes you from answers where you would otherwise be cited, which is usually not what a site means to do. Those decisions live in robots.txt, and they are enforced.

Readability. A page an agent cannot render is a page it cannot quote. Content that only exists after JavaScript executes is invisible to a large share of the bots fetching it. This is where the 6.6% figure above bites: sites write the rules and then serve agents markup they can't use. A GEO audit checks what agents can actually parse on your pages.

Measurement. Everything above is a hypothesis until you look. Server logs — or Agent Analytics — tell you which bots reach you and what they fetch, including whether anything ever requests the file you just published. An AI visibility check tells you whether your brand appears in AI answers at all. That last number is the one llms.txt is usually bought to move, and the only one worth watching.

Generate the file. It costs twenty minutes and it puts you in a small minority of sites that serve agents something readable. Then go and do the work that the evidence actually supports.

• The llms.txt proposal — Jeremy Howard, Answer.AI, September 2024
• Answer.AI's write-up of the standard
• Ahrefs, "We analyzed 137K sites: 97% of llms.txt files never get fetched" (linked once above)
• Anthropic's llms.txt and Cursor's llms-full.txt — examples in production
• SEOmator crawling and audit data, Jan–Jul 2026 (100K+ websites audited, 50M+ pages crawled; agent-readiness scan of 109,440 top domains, 2026-07-20)