GEO Data Report, September 2026: Which AI Crawlers & LLM Bots Take the Most and Give the Least?

Perplexity's crawler now fetches 2,373 pages for every referral its platform sends back to a website owner. In January the figure was 112. That more-than-twentyfold climb makes Perplexity, as of September 2026, the most extractive AI operator that still sends any traffic back at all. Mistral is worse in a different way: since August, Cloudflare has counted no referrals from it, so its ratio has no finite value. Anthropic went the other way, from 47,779:1 in January to 549:1. OpenAI sits at 341:1, and DuckDuckGo runs at 3.6:1. Every figure comes from Cloudflare Radar's crawl-to-refer ratio, pulled month by month from January to September 2026. That ratio is the metric at the center of this report: the crawl-to-refer ratio is the number of pages an AI crawler or LLM bot fetches divided by the referrals its parent platform (ChatGPT, Claude, Perplexity, Mistral, Copilot) sends back.

Data sources: Cloudflare Radar — AI Insights (crawl-to-refer ratio) and Bot & Crawler Analytics, calendar months January–September 2026 via the Radar API; robots.txt and agent-readiness scans dated October 5, 2026; and SEOmator's own analytics for the AI-assistant visits seomator.com received in September 2026.

What Is the Crawl-to-Refer Ratio and Why Does It Matter for GEO and SEO?

The crawl-to-refer ratio measures how many pages an AI crawler or LLM bot crawls from your website for every referral visit it sends back. A ratio of 100:1 means the bot crawled 100 of your pages before its parent platform directed a single user to your site.

This metric matters because AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others — consume your server resources (bandwidth, compute, and crawl budget) while indexing your content for their LLM models. The referral side measures whether that investment pays off through actual traffic returned to your site.

For SEO and GEO professionals, this ratio functions as a return-on-investment metric for crawler access. Every page crawled by an LLM bot is a page that could have been crawled by Googlebot instead, directly affecting your site's crawl budget allocation. And the load is no longer a rounding error. In September 2026, AI bots alone made up 5.3% of the requests Cloudflare classified, and mixed-purpose crawlers such as Googlebot, which feed both search and AI products, made up another 3.6%.

How Do the Top AI Crawlers and LLM Bots Compare on Crawl-to-Refer Ratios?

The gap between the best and worst crawl-to-refer ratios still spans three orders of magnitude, and the top of the ranking has changed again since the July edition of this report. Perplexity has climbed from the middle of the table to the top. Mistral has dropped out of the measurable range entirely. Anthropic has kept falling.

All figures here are calendar months. The July edition used a rolling 28-day window ending July 21, so its numbers differ slightly from the July column (it had Anthropic at 2,237:1 and Mistral at 3,389:1).

Mistral and Perplexity sit at the extractive end for different reasons. Mistral's consumer product sends back almost no clickable traffic. Perplexity does cite and link its sources, but its crawling grew far faster than its referrals (more on both below). Google and DuckDuckGo sit at the efficient end because their core function is to send users to websites — every search result click is a referral. I've been tracking these operators in server logs for the better part of a year, and the disparity is still striking when you see it at the individual-site level.

What changed since the July edition

• Perplexity: 296:1 → 2,373:1. The biggest move of any operator, and it accelerated: 822:1 in August, then nearly triple that in September.
• Mistral: 29,944:1 → no measurable referrals. Its crawling kept growing after July; Cloudflare counted no referrals back in either August or September.
• Anthropic: 1,914:1 → 549:1. It has fallen every month since May and is now within about 1.6× of OpenAI's 341:1.
• OpenAI: 256:1 → 341:1. July was the low point. The ratio rose in August (424:1) and stayed above its July level in September.
• Baidu: 12:1 → 40:1. Most of the jump came in a single month.

Understanding this ratio is now a core part of any serious AI search optimization strategy.

Why Did the Extraction Gap Collapse in 2026?

In January 2026, Anthropic's ratio was a five-figure number and OpenAI's was well over a thousand. The reason both fell so far is not that they crawl less — it's that their platforms finally started sending traffic back. As ChatGPT Search and Claude's cited-source surfaces matured, referrals appeared on the denominator of the ratio, and the number dropped fast.

Cloudflare Radar's monthly series shows the descent, and the reversals that came after it:

Calendar months, Cloudflare Radar bots/crawlers/summary/crawl_refer_ratio. "No referrals" means Radar reports the ratio as infinite: it counted crawling but no referrals.

The Anthropic line is the clearest signal in the dataset: an operator that took roughly 48,000 pages per referral in January was down to 549 by September, an 87-fold improvement, as Claude's user-facing surfaces began returning clicks. OpenAI followed the same path until July, then drifted back up as ChatGPT's share of referrals fell (see the referral section below). This is the referral side of the bargain reasserting itself — and it validates a claim I've made to clients all year: extraction ratios are not fixed, and they move the moment a platform ships a product that links out.

Why Are Mistral and Perplexity Now the Most Extractive Operators?

They got there by different routes, and Radar's user-agent data shows both.

Mistral is the mirror image of the Anthropic story. For the first five months of 2026 its ratio sat in the low double digits. From June, its crawling took off: MistralAI-User, Mistral's fetcher, went from 0.004% of AI-bot requests in June to 0.78% in September, a roughly 190-fold rise in share. Its platform sent back almost nothing in return. By July the ratio was 29,944:1, and in August and September Radar counted no Mistral referrals at all.

Perplexity's jump is also crawl-driven. PerplexityBot's share of AI-bot requests rose from 0.18% in June to 2.18% in September, about twelve times higher, and its referrals didn't keep pace. Unlike Mistral, Perplexity does send visitors, because its answers cite sources and link out. The ratio says its crawling now runs about 2,400 pages ahead of each visit it returns.

That combination — a sharp increase in crawling with little or no reciprocal referral — is exactly the profile that made ClaudeBot the poster child for extraction a year ago. If you are auditing AI crawler access this quarter, MistralAI-User and PerplexityBot are the names to watch in your logs. Few sites have caught up: MistralAI-User isn't among the 40 user agents that robots.txt files most often disallow (more on that below). The lesson from Anthropic is that the ratio can improve once a consumer product links out; the question for publishers is whether to subsidize the crawling in the meantime.

How Much of Your Crawl Load Is Actually AI — and What Are the Bots Fetching?

Before deciding what to block, it helps to see the composition of the traffic. On Cloudflare's network, humans and AI bots now split the request stream like this:

The 5.3% "AI bot" figure understates the pressure on content sites specifically, because it is an average across everything Cloudflare serves. The industry breakdown further down shows where the crawling concentrates.

What are all these bots actually pulling down? By declared crawl purpose across AI bots:

Search is the line that moved. It quadrupled, from 2.6% of AI-bot requests in January to 10.6% in September. Training is the largest declared purpose at 44.0%, about where it started the year, after peaking at 50.7% in July. The practical reading: AI bots increasingly crawl your site to answer live user questions, not only to train — which is precisely the traffic you want to be citable for.

And by content type, AI crawlers are overwhelmingly after your prose, not your assets:

More than two-thirds of AI crawler requests target HTML. Your written content — articles, documentation, product descriptions — is the corpus. That is both the risk (it's being taken) and the opportunity (it's what gets cited). Plain text is the line to watch: its share rose from 5.9% in July to 9.2% in September, while HTML slipped from 72.5%.

Which AI Crawlers and LLM Bots Actually Crawl the Most Pages?

Looking at total user-agent share across all crawler traffic (not just AI), Googlebot still leads by a wide margin. The bot in second place is still an AI crawler: Anthropic's ClaudeBot.

Radar's unclassified "other" bucket grew from 13.5% to 18.5% of crawler traffic between July and September, so treat small shifts in these shares with caution.

ClaudeBot's lead has narrowed. Its share of all crawler traffic fell from 19.5% in July to 11.6% in September, and GPTBot's fell from 12.7% to 7.8%. Within the AI-bot population specifically, Meta-ExternalAgent overtook ClaudeBot in August and kept the lead in September, at 13.1% of AI-bot requests against ClaudeBot's 12.0% and GPTBot's 6.2%. OpenAI's search crawler, OAI-SearchBot, more than doubled its share, from 1.6% in July to 3.6%. Keep the Meta number in mind for the blocking section: it is now the largest AI-only crawler, and Meta has no referral product at all. Our earlier research on AI SEO statistics tracked the Anthropic-over-OpenAI crossover, and the monthly AI crawler report follows these shares month by month.

Where Does Referral Traffic Actually Come From?

Despite all that crawling, referral traffic is still overwhelmingly Google. The ChatGPT line has reversed since the July edition, and Claude has joined it among the top referrers.

The July edition reported ChatGPT crossing 1% of referrals (1.05% over the 28 days to July 21) and called that ahead of schedule. It didn't hold. On calendar months, chatgpt.com rose from 0.18% in January to a peak of 0.90% in July, then fell to 0.53% in August and 0.41% in September. Over the same two months claude.ai joined the list at 0.30% and then 0.37%, so on Cloudflare's network Claude now sends almost as large a share of referrals as ChatGPT does. The fall in ChatGPT referrals is the other half of why OpenAI's ratio rose after July.

SEOmator's own site shows how different a single site can look. In September 2026, seomator.com received 63,012 sessions, and 1,664 of them (2.6%) came from AI assistants. Here is how those AI sessions split:

On Cloudflare's network, Claude's referral share is close to ChatGPT's. On seomator.com, ChatGPT sent more than 120 times as many sessions as Claude. One site is not a sample, and an SEO tool's audience isn't typical. Still, it's a useful warning: network-wide data tells you which platforms send traffic in aggregate, and only your own analytics tell you which ones send it to you. Check yours before deciding which assistants are worth being crawled by.

Which Industries Are Crawled the Most by AI Crawlers and LLM Bots?

Retail and infrastructure content absorb a disproportionate share of crawler attention relative to their footprint on the web.

Shopping sites absorb over a quarter of all crawler traffic, more than any other vertical, and these shares have barely moved since July. The mix of what gets crawled is far more stable than who does the crawling. When I run technical SEO audits for SaaS clients, industry-specific crawl patterns are always the first thing I check — a finance SaaS and a consumer-electronics retailer get fundamentally different returns from the same set of bots. I documented related AI bot traffic patterns by country in an earlier analysis, where geography drove comparable variation.

Is the Web Actually Ready for the AI Agents Crawling It?

Here is a gap that reframes the whole "should I block?" question. Cloudflare's agent-readiness scan checked 111,711 top domains on October 5, 2026:

Four in five top sites now write explicit AI-crawler rules, yet only 3.3% serve clean markdown to agents, and agent-native protocols (MCP, Web Bot Auth) are still effectively at zero. The web is writing rules for AI agents it isn't actually equipped to serve well. If you're deciding how to handle AI crawlers, that's the real state of the art: most sites are still fighting the last war (allow/deny in robots.txt) instead of the next one (serving agents structured, efficient content).

How Many Websites Are Blocking AI Crawlers and LLM Bots?

Website operators are responding to unfavorable ratios by naming AI crawlers in their robots.txt files, and the two names at the top of the list belong to the training crawlers of OpenAI and Anthropic. Cloudflare's robots.txt scan parsed 4,257 top-domain robots.txt files on October 5, 2026. Counting Disallow rules only:

The July edition counted every file that named a bot in any rule, Allow or Disallow, which is why its numbers were higher (GPTBot 773, ClaudeBot 629). The table above counts only files that disallow the bot.

The most extractive operators are among the least blocked. MistralAI-User doesn't appear among the 40 user agents robots.txt files most often disallow, despite sending no measurable referrals since August. PerplexityBot is disallowed by 348 files and explicitly allowed by almost as many (344).

Sites are also starting to treat an operator's training crawler differently from its search crawler:

For OpenAI the split is already clear. GPTBot is disallowed far more often than it's allowed, while OAI-SearchBot is allowed more often than it's disallowed: block the training, keep the search visibility. For Anthropic, most files haven't caught up. Claude-SearchBot is named in less than half as many files as ClaudeBot.

Notably, blocking is rising even as some ratios improve. On Cloudflare's network, the share of crawler requests answered with a 4xx response climbed from 33.1% in January to 38.1% in September, and the share actually served (2xx) slipped from 51.5% to 46.6%. More than one in three crawler requests now ends in a 4xx. If you're unsure how your own robots.txt is configured for LLM bots, our robots.txt guide walks through the AI-crawler rules, and our llms.txt adoption data shows how many sites pair them with an llms.txt file.

Should You Block AI Crawlers and LLM Bots Based on This Data?

The decision depends on your industry, traffic goals, and long-term GEO strategy. Based on the September 2026 data, the operators sort into four groups — and the membership has shifted again since July.

Block with confidence: Mistral (no measurable referrals since August) and Meta-ExternalAgent. Mistral's fetcher grew roughly 190-fold as a share of AI-bot requests since June with nothing Cloudflare could count coming back, and Meta-ExternalAgent, now the largest AI-only crawler, has never had a referral product. These provide near-zero return and consume resources exclusively for the operator's benefit.

Evaluate carefully: PerplexityBot (2,373:1), ClaudeBot (549:1) and GPTBot (341:1). All three operators run answer engines that cite and link to sources, so blocking them means forfeiting representation in those answers — a real AI search optimization cost. But they are moving in opposite directions: Anthropic's ratio has improved 87-fold since January, while Perplexity's has worsened more than twentyfold. For OpenAI and Anthropic you don't have to choose all or nothing: block the training crawler and keep the search crawler (GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-SearchBot), as many of the sites in Cloudflare's scan already do.

Allow strategically: Microsoft Copilot (40:1). Moderate ratio, referral traffic through Bing and Copilot surfaces, and prominent source citation.

Keep unblocked: Google (4.7:1), DuckDuckGo (3.6:1), and traditional search crawlers. These deliver measurable referral traffic that justifies their crawl volume.

One caveat before you write any robots.txt rule: referrals are not the only return on being crawled. A bot can shape how an assistant describes your brand without ever sending a click, so blocking is not a free action. And for the operators you do decide to allow, access is not the only lever — you can generate an llms.txt file to tell those bots which of your pages actually matter, rather than leaving them to infer it from a full crawl.

What Does This Mean for the Future of GEO, AI Search, and Publisher Relations?

The crawl-to-refer data from the first nine months of 2026 revises the story the web told itself in 2025. The fear was that LLM platforms would crawl endlessly and never send traffic back. What actually happened is more mixed: the biggest extractors improved dramatically the moment they shipped consumer products that link out, while other operators (Mistral, then Perplexity) took over the extraction lead in their place. Three trends point to where this is heading:

1. Extraction ratios are elastic in both directions. Anthropic went from ~48,000:1 to 549:1 in nine months because referrals appeared. Perplexity went from 112:1 to 2,373:1 because its crawling outran them. Today's worst offender is not necessarily tomorrow's, and neither is today's best.

2. AI referrals are real, but no single platform's share is stable. ChatGPT's share of referral traffic on Cloudflare peaked at 0.90% in July and fell to 0.41% by September, while Claude's rose to 0.37%. On seomator.com, AI assistants still sent 2.6% of September sessions. GEO work for these engines shows up in analytics now; just don't plan around any one platform's share holding steady.

3. The readiness gap is the next battleground. With 80% of sites writing AI rules but only 3.3% serving markdown to agents, the winners of the next phase won't be the sites that block hardest — they'll be the ones that serve AI agents efficient, structured, citable content. Blocking is defense; being citable is offense.

Website owners who monitor these ratios quarterly and adjust their AI crawler access based on measurable returns will outperform those who either block everything or allow everything. That is the foundation of a data-driven GEO strategy.

How to Monitor Your Site's Crawl-to-Refer Ratio

While Cloudflare Radar provides network-wide aggregates, individual site owners can approximate their own crawl-to-refer ratios using server logs and analytics. Here's the process I follow for clients:

1. Server log analysis: Count requests from known AI crawler and LLM bot user agents (GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, MistralAI-User, Meta-ExternalAgent, PerplexityBot) over a 30-day period
2. Referral tracking: In Google Analytics 4 or your analytics platform, filter referral traffic from chatgpt.com, claude.ai, perplexity.ai, gemini.google.com, and other LLM-powered surfaces
3. Calculate the ratio: Divide total AI crawler requests by total AI-referred visits, per operator
4. Compare against benchmarks: Use the September 2026 ratios in this report as baselines, and re-check quarterly — as the Perplexity and Mistral swings show, the numbers move fast

If you don't have server-log access, an AI crawler access report covers the crawl half of the equation — it shows which LLM bots your robots.txt currently admits or blocks, which is the input you can actually change.

SEO tools like SEOmator can help automate the technical side — analyzing your robots.txt configuration, monitoring AI crawler access patterns, and identifying which LLM bots are consuming your crawl budget without delivering proportional value.

Methodology

This report is built on Cloudflare Radar, which monitors traffic across Cloudflare's global network, a significant share of all internet traffic, making its bot and crawler data broadly representative of web-wide patterns. All traffic figures were pulled from the Radar API on October 7, 2026, as calendar months from January 1 to September 30, 2026, so every month-to-month comparison covers a full month. The crawl-to-refer ratios come from bots/crawlers/summary/crawl_refer_ratio (the metric as Cloudflare defines it); client types, crawler user agents, referrers, verticals and response codes from the web crawlers endpoints; crawl purpose, AI user agents and content types from the AI bots endpoints; robots.txt counts from robots_txt/top/user_agents/directive, queried separately for Disallow and Allow; and agent-readiness signals from agent_readiness. The robots.txt and agent-readiness figures are single-day scans dated October 5, 2026.

SEOmator's own figures cover one site we can fully verify: the AI-assistant sessions seomator.com received from September 1 to 30, 2026, measured by our first-party analytics. They describe one SEO-tool website's audience, not the web. The July edition of this report also quoted a multi-site SEOmator panel; those figures are not carried forward, and this edition reports Cloudflare's network and our own site only.

Data dimensions used:

• Crawl-to-refer ratio: Pages crawled per referral sent, by operator, monthly
• Referral source share: Referral traffic by source domain (Cloudflare) and AI-assistant sessions (seomator.com)
• Client type: Traffic classification (Human, Non-AI Bot, AI Bot, Mixed Purpose)
• User agent + crawl purpose + content type: Crawler identification and what AI bots fetch
• Vertical / industry: Traffic segmentation by website category
• Response status: Share of crawler requests answered with 2xx and 4xx codes
• Agent readiness + robots.txt analysis: 111,711 domains scanned; 4,257 robots.txt files parsed

Date range: Cloudflare traffic data: calendar months, January–September 2026. Robots.txt and agent readiness: October 5, 2026. seomator.com: September 1–30, 2026.

Limitations: Network-wide ratios measure aggregate behavior; individual-site ratios vary with content type, domain authority, and traffic patterns. Referral attribution may undercount AI-driven visits that arrive through intermediate pages or omit a referrer header. Radar reports shares and ratios, not absolute request counts, and a ratio it reports as infinite means it counted crawling but no referrals in the period, not a measured value. The unclassified share of crawler user agents grew between July and September, so small shifts in user-agent shares are less reliable than the large ones.

Cloudflare data sourced from Cloudflare Radar AI Insights and Bot & Crawler Analytics (calendar months January–September 2026; robots.txt and agent-readiness scans October 5, 2026). seomator.com figures: SEOmator first-party analytics, September 2026. Last updated: October 7, 2026.