GEO Data Report, October 2026: Which AI Crawlers & LLM Bots Take the Most and Give the Least?

Perplexity's crawler now fetches 4,056 pages for every referral its platform sends back to a website owner. In January the figure was 112. That 36-fold climb makes Perplexity, as of October 2026, the most extractive AI operator that still sends any traffic back at all. Mistral is worse in a different way: for the third month running, Cloudflare counted no referrals from it, so its ratio has no finite value. Anthropic went the other way, from 47,779:1 in January to 482:1, and for the first time this year it returns more referrals per page crawled than OpenAI. OpenAI's ratio more than doubled to 775:1, mostly because ChatGPT's links stopped carrying a referrer, a change we could watch happen in seomator.com's own analytics. DuckDuckGo runs at 4.0:1. Every figure comes from Cloudflare Radar's crawl-to-refer ratio, pulled month by month from January to October 2026. That ratio is the metric at the center of this report: the crawl-to-refer ratio is the number of pages an AI crawler or LLM bot fetches divided by the referrals its parent platform (ChatGPT, Claude, Perplexity, Mistral, Copilot) sends back.

Data sources: Cloudflare Radar — AI Insights (crawl-to-refer ratio) and Bot & Crawler Analytics, calendar months January–September 2026 plus October 1–30 via the Radar API; robots.txt and agent-readiness scans dated October 26, 2026; and SEOmator's own analytics for the AI-assistant visits seomator.com received in September and October 2026.

What Is the Crawl-to-Refer Ratio and Why Does It Matter for GEO and SEO?

The crawl-to-refer ratio measures how many pages an AI crawler or LLM bot crawls from your website for every referral visit it sends back. A ratio of 100:1 means the bot crawled 100 of your pages before its parent platform directed a single user to your site.

This metric matters because AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others — consume your server resources (bandwidth, compute, and crawl budget) while indexing your content for their LLM models. The referral side measures whether that investment pays off through actual traffic returned to your site.

For SEO and GEO professionals, this ratio functions as a return-on-investment metric for crawler access. Every page crawled by an LLM bot is a page that could have been crawled by Googlebot instead, directly affecting your site's crawl budget allocation. And the load is no longer a rounding error. In October 2026, AI bots alone made up 6.3% of the requests Cloudflare classified, up from 5.3% in September, and mixed-purpose crawlers such as Googlebot, which feed both search and AI products, made up another 4.0%.

How Do the Top AI Crawlers and LLM Bots Compare on Crawl-to-Refer Ratios?

The gap between the best and worst crawl-to-refer ratios still spans three orders of magnitude, and the order in the middle of the table changed again in October. Perplexity kept climbing. Anthropic dropped below OpenAI for the first time in 2026. Mistral stayed outside the measurable range.

*October covers October 1–30, 2026: the data was pulled on October 31, and Radar won't return a window that ends in the future. Every other month is a full calendar month. Re-pulling January–September on October 31 returned the same values as the September edition's pull, so no earlier month was revised.

Mistral and Perplexity sit at the extractive end for different reasons. Mistral's consumer product sends back almost no clickable traffic. Perplexity does cite and link its sources, but its crawling grew far faster than its referrals (more on both below). Google and DuckDuckGo sit at the efficient end because their core function is to send users to websites — every search result click is a referral. I've been tracking these operators in server logs for the better part of a year, and the disparity is still striking when you see it at the individual-site level.

What changed since the September edition

• Perplexity: 2,373:1 → 4,056:1. Up 71% in a month. Its ratio has risen every month since March.
• OpenAI: 341:1 → 775:1. More than doubled, but most of the move is about measurement, not crawling: chatgpt.com's share of referrals on Cloudflare fell from 0.41% to 0.17% as ChatGPT's links stopped passing a referrer.
• Anthropic: 549:1 → 482:1. Its fifth straight monthly decline, and the first month this year in which Anthropic's ratio is lower than OpenAI's.
• Mistral: still no measurable referrals. The third month running. Its fetcher's share of AI-bot requests halved, from 0.78% to 0.37%.
• Yandex: 31:1 → 16:1. Nearly halved, the lowest it has been all year.
• Baidu: 40:1 → 56:1. Up again after nearly tripling in September.

Understanding this ratio is now a core part of any serious AI search optimization strategy.

Why Did the Extraction Gap Collapse in 2026?

In January 2026, Anthropic's ratio was a five-figure number and OpenAI's was well over a thousand. The reason both fell so far is not that they crawl less — it's that their platforms finally started sending traffic back. As ChatGPT Search and Claude's cited-source surfaces matured, referrals appeared on the denominator of the ratio, and the number dropped fast.

Cloudflare Radar's monthly series shows the descent, and the reversals that came after it:

Calendar months, Cloudflare Radar bots/crawlers/summary/crawl_refer_ratio; *October 1–30. "No referrals" means Radar reports the ratio as infinite: it counted crawling but no referrals.

The Anthropic line is the clearest signal in the dataset: an operator that took roughly 48,000 pages per referral in January was down to 482 by October, a roughly 99-fold improvement, as Claude's user-facing surfaces began returning clicks. OpenAI followed the same path until July, then drifted back up as ChatGPT's share of referrals fell, and in October its ratio more than doubled for a reason that has little to do with crawling (see the referral section below). This is the referral side of the bargain reasserting itself — and it validates a claim I've made to clients all year: extraction ratios are not fixed, and they move the moment a platform ships a product that links out.

Why Are Mistral and Perplexity Now the Most Extractive Operators?

They got there by different routes, and Radar's user-agent data shows both.

Mistral is the mirror image of the Anthropic story. For the first five months of 2026 its ratio sat in the low double digits. From June, its crawling took off: MistralAI-User, Mistral's fetcher, went from 0.004% of AI-bot requests in June to 0.78% in September, a roughly 190-fold rise in share. Its platform sent back almost nothing in return. By July the ratio was 29,944:1, and from August through October Radar counted no Mistral referrals at all. October brought the first pullback in its crawling: the fetcher's share halved to 0.37%, which is still about 90 times its June level.

Perplexity's jump is also crawl-driven. PerplexityBot's share of AI-bot requests rose from 0.18% in June to 2.57% in October, nearly fifteen times higher, and its referrals didn't keep pace. Unlike Mistral, Perplexity does send visitors, because its answers cite sources and link out. The ratio says its crawling now runs about 4,000 pages ahead of each visit it returns.

That combination — a sharp increase in crawling with little or no reciprocal referral — is exactly the profile that made ClaudeBot the poster child for extraction a year ago. If you are auditing AI crawler access this quarter, MistralAI-User and PerplexityBot are the names to watch in your logs. Few sites have caught up: MistralAI-User isn't among the 40 user agents that robots.txt files most often disallow (more on that below). The lesson from Anthropic is that the ratio can improve once a consumer product links out; the question for publishers is whether to subsidize the crawling in the meantime.

How Much of Your Crawl Load Is Actually AI — and What Are the Bots Fetching?

Before deciding what to block, it helps to see the composition of the traffic. On Cloudflare's network, humans and AI bots now split the request stream like this:

The 6.3% "AI bot" figure understates the pressure on content sites specifically, because it is an average across everything Cloudflare serves. The industry breakdown further down shows where the crawling concentrates.

What are all these bots actually pulling down? By declared crawl purpose across AI bots:

Search is the line that moved. It more than quadrupled, from 2.6% of AI-bot requests in January to 11.6% in October. Training is the largest declared purpose at 43.8%, about where it started the year, after peaking at 50.7% in July. User-triggered fetches, where an assistant opens a page because a person asked it to, grew to 3.9%. The practical reading: AI bots increasingly crawl your site to answer live user questions, not only to train — which is precisely the traffic you want to be citable for.

And by content type, AI crawlers are overwhelmingly after your prose, not your assets:

Nearly three-quarters of AI crawler requests target HTML. Your written content — articles, documentation, product descriptions — is the corpus. That is both the risk (it's being taken) and the opportunity (it's what gets cited). Plain text is the line to watch: its share rose from 5.9% in July to 9.2% in September and held at 8.4% in October, while HTML recovered to 72.9% after dipping to 68.9% in September.

Which AI Crawlers and LLM Bots Actually Crawl the Most Pages?

Looking at total user-agent share across all crawler traffic (not just AI), Googlebot still leads by a wide margin. The bot in second place is still an AI crawler: Anthropic's ClaudeBot.

Radar's unclassified "other" bucket grew from 13.5% of crawler traffic in July to 18.7% in October, so treat small shifts in these shares with caution.

ClaudeBot's lead has narrowed since summer. Its share of all crawler traffic fell from 19.5% in July to 11.8% in October, and GPTBot's fell from 12.7% to 6.7%. Within the AI-bot population specifically, ClaudeBot took the lead back from Meta-ExternalAgent in October, at 12.6% of AI-bot requests against Meta's 11.2% and GPTBot's 5.5%. OpenAI's search crawler, OAI-SearchBot, has nearly tripled its share since July, from 1.6% to 4.3%. One new name entered the top nine: ShapBot, the crawler Parallel Web Systems uses to index pages for its web APIs, at 4.6% of all crawler traffic. Cloudflare counts it among AI bots but doesn't list it in its bot directory. Keep the Meta number in mind for the blocking section: Meta-ExternalAgent is the second-largest AI-only crawler, and Meta has no referral product at all. Our earlier research on AI SEO statistics tracked the Anthropic-over-OpenAI crossover, and the monthly AI crawler report follows these shares month by month.

Where Does Referral Traffic Actually Come From?

Despite all that crawling, referral traffic is still overwhelmingly Google. On Cloudflare's numbers, ChatGPT has fallen to the bottom of the list, and Claude now sends more than twice its share.

The July edition reported ChatGPT crossing 1% of referrals (1.05% over the 28 days to July 21) and called that ahead of schedule. It didn't hold. On calendar months, chatgpt.com rose from 0.18% in January to a peak of 0.90% in July, then fell to 0.53% in August, 0.41% in September and 0.17% in October, slightly below where it started the year. claude.ai went the other way, from 0.30% in August to 0.40% in October.

Did ChatGPT stop sending traffic in October?

Not to us. On the evidence we have, the October drop is mostly a measurement change: ChatGPT's links stopped passing a referrer. Cloudflare counts a referral when a visit arrives with a Referer header naming the platform. On seomator.com we can see both that header and the utm_source=chatgpt.com tag ChatGPT adds to the links it shows, and in October the two came apart:

*Oct 25–30, six days.

ChatGPT visits kept arriving, and grew, while the share carrying a chatgpt.com referrer fell from about half to almost none in two weeks. Radar's weekly referral series breaks at the same moment: chatgpt.com's share of referrals was 0.37% in the week of September 28 and 0.07% in each of the last two weeks of October. Over the same weeks, OpenAI's weekly crawl-to-refer ratio climbed from 362:1 to 1,884:1. Radar also began attributing a new OpenAI bot in the last days of September, "ChatGPT agent", which made up 65% of OpenAI's bot requests in October. OpenAI's ratio barely moved in the week that bot appeared. It jumped in the week the referrers disappeared.

So OpenAI's October ratio of 775:1 overstates how little ChatGPT gives back, and so does any analytics report that identifies ChatGPT traffic by referrer alone. If your GA4 "chatgpt.com / referral" line fell off a cliff in October, check utm_source=chatgpt.com before you conclude that ChatGPT stopped sending visitors. One site can't tell us why the header disappeared or whether every ChatGPT surface is affected. What it does show is that ChatGPT sessions on seomator.com rose 48% from September to October.

SEOmator's own site also shows how different a single site can look. From October 1 to 30, 2026, seomator.com received 77,672 sessions, and 2,281 of them (2.9%) came from AI assistants, up from 2.6% in September. Here is how those AI sessions split:

On Cloudflare's network, Claude's referral share is now more than double ChatGPT's. On seomator.com, ChatGPT sent about 87 times as many sessions as Claude. Part of that gap is the referrer change above: our analytics recognize ChatGPT by its utm tag, while Radar's referral count needs the header. The rest is audience. One site is not a sample, and an SEO tool's audience isn't typical. Still, it's a useful warning: network-wide data tells you which platforms send traffic in aggregate, and only your own analytics tell you which ones send it to you. Check yours before deciding which assistants are worth being crawled by.

Which Industries Are Crawled the Most by AI Crawlers and LLM Bots?

Retail and infrastructure content absorb a disproportionate share of crawler attention relative to their footprint on the web.

Shopping sites absorb nearly a quarter of all crawler traffic, more than any other vertical, and these shares have barely moved since July. The mix of what gets crawled is far more stable than who does the crawling. When I run technical SEO audits for SaaS clients, industry-specific crawl patterns are always the first thing I check — a finance SaaS and a consumer-electronics retailer get fundamentally different returns from the same set of bots. I documented related AI bot traffic patterns by country in an earlier analysis, where geography drove comparable variation.

Is the Web Actually Ready for the AI Agents Crawling It?

Here is a gap that reframes the whole "should I block?" question. Cloudflare's agent-readiness scan checked 114,605 top domains on October 26, 2026:

Four in five top sites now write explicit AI-crawler rules, yet only 4.0% serve clean markdown to agents, and agent-native protocols (MCP, Web Bot Auth) are still effectively at zero. The web is writing rules for AI agents it isn't actually equipped to serve well. If you're deciding how to handle AI crawlers, that's the real state of the art: most sites are still fighting the last war (allow/deny in robots.txt) instead of the next one (serving agents structured, efficient content).

How Many Websites Are Blocking AI Crawlers and LLM Bots?

Website operators are responding to unfavorable ratios by naming AI crawlers in their robots.txt files, and the two names at the top of the list belong to the training crawlers of OpenAI and Anthropic. Cloudflare's robots.txt scan parsed 4,367 top-domain robots.txt files on October 26, 2026. Counting Disallow rules only:

Blocking moved in three weeks, and it moved toward training crawlers. Between Cloudflare's October 5 and October 26 scans, the training crawlers each gained roughly 75 to 100 disallowing files (GPTBot +93, ClaudeBot +94, CCBot +92, Google-Extended +92), while the search and user-triggered fetchers barely moved (OAI-SearchBot +13, Claude-SearchBot +16, ChatGPT-User +22). The October 26 scan also parsed 110 more files than the earlier one, so the counts mix new sites with changed rules. The direction is the useful part: the new blocking is aimed at bots that train on pages, not at the ones that fetch a page to answer a question.

The most extractive operators are among the least blocked. MistralAI-User doesn't appear among the 40 user agents robots.txt files most often disallow, despite three months without measurable referrals. PerplexityBot is disallowed by 375 files and explicitly allowed by almost as many (360).

Sites are also treating an operator's training crawler differently from its search crawler:

For OpenAI the split is already clear. GPTBot is disallowed far more often than it's allowed, while OAI-SearchBot is allowed more often than it's disallowed: block the training, keep the search visibility. For Anthropic, most files haven't caught up. Claude-SearchBot is named in less than half as many files as ClaudeBot.

Notably, blocking is rising even as some ratios improve. On Cloudflare's network, the share of crawler requests answered with a 4xx response climbed from 33.1% in January to 38.9% in October, and the share actually served (2xx) slipped from 51.5% to 45.8%. More than one in three crawler requests now ends in a 4xx. If you're unsure how your own robots.txt is configured for LLM bots, our robots.txt guide walks through the AI-crawler rules, and our llms.txt adoption data shows how many sites pair them with an llms.txt file.

Should You Block AI Crawlers and LLM Bots Based on This Data?

The decision depends on your industry, traffic goals, and long-term GEO strategy. Based on the October 2026 data, the operators sort into four groups.

Block with confidence: Mistral (no measurable referrals since August) and Meta-ExternalAgent. Mistral's fetcher grew roughly 190-fold as a share of AI-bot requests between June and September, and even after halving in October, nothing Cloudflare could count came back. Meta-ExternalAgent, the second-largest AI-only crawler, has never had a referral product. These provide near-zero return and consume resources exclusively for the operator's benefit.

Evaluate carefully: PerplexityBot (4,056:1), GPTBot (775:1) and ClaudeBot (482:1). All three operators run answer engines that cite and link to sources, so blocking them means forfeiting representation in those answers — a real AI search optimization cost. But they are moving in opposite directions: Anthropic's ratio has improved roughly 99-fold since January, while Perplexity's has worsened 36-fold. Read OpenAI's October figure with the referrer caveat above; its real return is better than 775:1 suggests. For OpenAI and Anthropic you don't have to choose all or nothing: block the training crawler and keep the search crawler (GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-SearchBot), as a growing number of the sites in Cloudflare's scan already do.

Allow strategically: Microsoft Copilot (40:1). Moderate ratio, referral traffic through Bing and Copilot surfaces, and prominent source citation.

Keep unblocked: Google (4.8:1), DuckDuckGo (4.0:1), and traditional search crawlers. These deliver measurable referral traffic that justifies their crawl volume.

One caveat before you write any robots.txt rule: referrals are not the only return on being crawled. A bot can shape how an assistant describes your brand without ever sending a click, so blocking is not a free action. And for the operators you do decide to allow, access is not the only lever — you can generate an llms.txt file to tell those bots which of your pages actually matter, rather than leaving them to infer it from a full crawl.

What Does This Mean for the Future of GEO, AI Search, and Publisher Relations?

The crawl-to-refer data from the first ten months of 2026 revises the story the web told itself in 2025. The fear was that LLM platforms would crawl endlessly and never send traffic back. What actually happened is more mixed: the biggest extractors improved dramatically the moment they shipped consumer products that link out, while other operators (Mistral, then Perplexity) took over the extraction lead in their place. Three trends point to where this is heading:

1. Extraction ratios are elastic in both directions. Anthropic went from ~48,000:1 to 482:1 in ten months because referrals appeared. Perplexity went from 112:1 to 4,056:1 because its crawling outran them. Today's worst offender is not necessarily tomorrow's, and neither is today's best.

2. AI referrals are real, but measuring them is fragile. ChatGPT's share of referral traffic on Cloudflare peaked at 0.90% in July and read 0.17% in October, largely because its links stopped carrying a referrer, while Claude's rose to 0.40%. On seomator.com, AI assistants sent 2.9% of October sessions, and ChatGPT's count went up, not down. GEO work for these engines shows up in analytics now; just make sure your analytics can still see it.

3. The readiness gap is the next battleground. With four in five sites writing AI rules but only 4.0% serving markdown to agents, the winners of the next phase won't be the sites that block hardest — they'll be the ones that serve AI agents efficient, structured, citable content. Blocking is defense; being citable is offense.

Website owners who monitor these ratios quarterly and adjust their AI crawler access based on measurable returns will outperform those who either block everything or allow everything. That is the foundation of a data-driven GEO strategy.

How to Monitor Your Site's Crawl-to-Refer Ratio

While Cloudflare Radar provides network-wide aggregates, individual site owners can approximate their own crawl-to-refer ratios using server logs and analytics. Here's the process I follow for clients:

1. Server log analysis: Count requests from known AI crawler and LLM bot user agents (GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, MistralAI-User, Meta-ExternalAgent, PerplexityBot, ShapBot) over a 30-day period
2. Referral tracking: In Google Analytics 4 or your analytics platform, filter referral traffic from chatgpt.com, claude.ai, perplexity.ai, gemini.google.com, and other LLM-powered surfaces. Count sessions tagged utm_source=chatgpt.com as well: since October 2026, most ChatGPT clicks we see arrive without a referrer
3. Calculate the ratio: Divide total AI crawler requests by total AI-referred visits, per operator
4. Compare against benchmarks: Use the October 2026 ratios in this report as baselines, and re-check quarterly — as the Perplexity, Mistral and OpenAI swings show, the numbers move fast

If you don't have server-log access, an AI crawler access report covers the crawl half of the equation — it shows which LLM bots your robots.txt currently admits or blocks, which is the input you can actually change.

SEO tools like SEOmator can help automate the technical side — analyzing your robots.txt configuration, monitoring AI crawler access patterns, and identifying which LLM bots are consuming your crawl budget without delivering proportional value.

Methodology

This report is built on Cloudflare Radar, which monitors traffic across Cloudflare's global network, a significant share of all internet traffic, making its bot and crawler data broadly representative of web-wide patterns. All traffic figures were pulled from the Radar API on October 31, 2026, as calendar months from January 1 to September 30, 2026, plus October 1–30 (Radar won't return a window that ends in the future, so October is one day short). Re-pulling January–September returned the same values as the October 7 pull behind the September edition. The crawl-to-refer ratios come from bots/crawlers/summary/crawl_refer_ratio (the metric as Cloudflare defines it), with weekly values from the matching timeseries_groups endpoint; client types, crawler user agents, referrers (monthly and weekly), verticals and response codes from the web crawlers endpoints; OpenAI's bot mix from the bots endpoints; crawl purpose, AI user agents and content types from the AI bots endpoints; robots.txt counts from robots_txt/top/user_agents/directive, queried separately for Disallow and Allow; and agent-readiness signals from agent_readiness. The robots.txt and agent-readiness figures are single-day scans dated October 26, 2026; the three-week robots.txt comparison uses the October 5, 2026 scan quoted in the September edition.

SEOmator's own figures cover one site we can fully verify: the AI-assistant sessions seomator.com received from October 1 to 30, 2026 (with September 1–30 for comparison), and the weekly referrer and utm_source of its ChatGPT sessions from September 13 to October 30, 2026, measured by our first-party analytics. They describe one SEO-tool website's audience, not the web. The July edition of this report also quoted a multi-site SEOmator panel; those figures are not carried forward, and this edition reports Cloudflare's network and our own site only.

Data dimensions used:

• Crawl-to-refer ratio: Pages crawled per referral sent, by operator, monthly (weekly for OpenAI, Anthropic and Perplexity in September–October)
• Referral source share: Referral traffic by source domain (Cloudflare) and AI-assistant sessions (seomator.com)
• Referrer vs utm tag: ChatGPT sessions on seomator.com by week, with and without a chatgpt.com referrer
• Client type: Traffic classification (Human, Non-AI Bot, AI Bot, Mixed Purpose)
• User agent + crawl purpose + content type: Crawler identification and what AI bots fetch
• Vertical / industry: Traffic segmentation by website category
• Response status: Share of crawler requests answered with 2xx and 4xx codes
• Agent readiness + robots.txt analysis: 114,605 domains scanned; 4,367 robots.txt files parsed

Date range: Cloudflare traffic data: calendar months, January–September 2026, plus October 1–30, 2026. Robots.txt and agent readiness: October 26, 2026. seomator.com: September 1–30 and October 1–30, 2026.

Limitations: Network-wide ratios measure aggregate behavior; individual-site ratios vary with content type, domain authority, and traffic patterns. Referral attribution may undercount AI-driven visits that arrive through intermediate pages or omit a referrer header, and from October 2026 that applies to most ChatGPT clicks we can observe, so ChatGPT's referral share and OpenAI's ratio understate ChatGPT's real referrals from that month on. Radar reports shares and ratios, not absolute request counts, and a ratio it reports as infinite means it counted crawling but no referrals in the period, not a measured value. The unclassified share of crawler user agents grew between July and October, so small shifts in user-agent shares are less reliable than the large ones.

Cloudflare data sourced from Cloudflare Radar AI Insights and Bot & Crawler Analytics (calendar months January–September 2026 plus October 1–30, 2026; robots.txt and agent-readiness scans October 26, 2026). seomator.com figures: SEOmator first-party analytics, September–October 2026. Last updated: October 31, 2026.