llms.txt: We Scanned the Web's Top 100,000 Domains
We checked the web's top 100,000 domains for llms.txt. Only 8.6% publish one, and nearly a third of those were generated automatically by the platform.
Team Member
Charles Duncan
Co-founder & CTO
Charles Duncan is co-founder and CTO of SEOmator, where he leads the engineering behind the platform's site crawler, audit engine, and rank-tracking infrastructure. He writes about the technical foundations of SEO — how search engines and AI answer engines crawl, render, and index the modern web — and what that means for the people building sites on it.
Articles by Charles Duncan
Monthly AI Crawler Report: July 2026 — ClaudeBot Cools, Meta Rebounds
ClaudeBot cooled from 22.4% to 19.2% of AI-bot hits across the sites we track in July 2026 as Meta rebounded to 12.7%. Our monthly AI crawler report.
SEO Prompts for ChatGPT: 50+ Proven Prompts to Boost Google Rankings (2026)
50+ ChatGPT prompts for keyword research, content optimization and competitor analysis — grouped by the job each one actually does well, with the outputs you need to verify before acting on them.
30 Link Building Statistics You Should Know for 2026
91.3% of the AI Overview citations we tracked in H1 2026 went to pages already in the organic top 20. See the 30 link building numbers that matter in 2026.
10 Best Ranktracker Alternatives for 2026 (Tested & Compared)
The best Ranktracker alternatives for 2026 are SEOmator ($49/mo), Keyword.com ($46/mo), AccuRanker ($116/mo), SEO PowerSuite ($29/mo), and SE Ranking ($52/mo). Each offers unique strengths in accuracy, pricing, and features.
SEO for Pest Control Companies: 7-Step Local SEO Guide (2026)
SEO for pest control companies focuses on local search optimization: claim Google Business Profile, target "near me" keywords, optimize title tags, build local backlinks, research pest-specific keywords, publish quality content, and implement internal linking. These 7 steps can increase organic traffic by 280%.
9 Best Surfer SEO Alternatives: Compared and Tested (2026)
Surfer SEO is great for content optimization but lacks audits, backlinks, and keyword depth. Here are 9 tested alternatives from $12/month to full SEO suites.
Technical SEO Guide for SaaS: 8-Step Audit Checklist (2026)
Organic search drives 53% of SaaS traffic at a 702% ROI. This 8-step technical SEO audit fixes crawlability, Core Web Vitals, canonicals, and AI-search visibility.
Complete SEO Audit Guide for Charities and Nonprofits (2026)
An SEO audit for charities covers 16 areas from site indexing to local SEO. With organic search driving 44% of nonprofit website traffic and only 0.16% of visitors donating, a thorough audit helps maximize every visitor's potential to support your cause.
6-Step SEO Audit for Law Firms: Complete Guide (2026)
An SEO audit for law firms covers keyword research, on-page optimization, technical health, local SEO, backlinks, and competitor analysis. With 87% of clients using Google to find lawyers, regular audits are essential for maintaining visibility and generating qualified leads.
How to Recover Lost SEO Rankings in 9 Steps (2026)
Lost SEO rankings or traffic? Diagnose the drop in Search Console, rule out core updates and AI Overviews, then work through these 9 recovery steps.
How to Remove URLs from Google (For Different URL Types)
In the virtual universe of Google, every URL carries a significant value. Whether it is a page that reflects the features of your product or a blog post narrating your thoughts, each URL is a gateway for the outside world, including customers, peers, or potential collaborators, to know about you, your business, or your ideas.
HTTP Content-Type: What is It & How to Check
We frequently deal with various types of web content. Haven't we all wondered about the process that goes behind ensuring this content is properly and correctly displayed in web browsers?
Canonical Issues: How to Find & Fix and Avoid Them
Recognizing canonical issues is your first step towards resolving them efficiently. As with any other SEO strategy, you must equip yourself with the right knowledge, take proactive steps to avoid common pitfalls, and keep a close eye on your implementation.
Subdomain vs. Subdirectory: Which is Better for SEO?
We probed 232 ranking domains on 31 July 2026: 87.9% keep the blog in a subfolder, but 66% run docs on a subdomain. See the split, then pick yours.
How to Create a Topical Map in 6 Steps
A topical map is essentially your website's game plan. So, its importance in shaping your website's content strategy and SEO effectiveness can't be overstated.
E-Commerce Site Audit: 10-Step 2026 Conversion Checklist
The median e-commerce site we audit scores 68 (D+); 52% carry Product schema errors. Run this 10-step audit to find your leaks and fix them in order.
How to Fix 'Crawled-Currently Not Indexed' Issue in GSC
Have you ever come across the pesky ‘Crawled-Currently Not Indexed’ error in your Google Search Console?
Website Migration Checklist for SEO: 24 Steps (2026)
Redirect chains fail on 24.3% of the sites we audit. A 24-point website migration checklist for SEO covering pre-launch, launch day, and the 90 days after.
How to Find Niche Keywords in 7 Steps (2026)
AI Overviews now hit ~27% of the keywords we track (July 2026). Here's how to find niche keywords that still earn a click, in 7 steps, plus a 2026 tools table.
Tools reviewed by Charles Duncan
