AI Bot Traffic by Country: Who Crawls, Who Gets Blocked (October 2026)

In October 2026, 17% of AI-bot requests got a 403, up from 10% in July. GPTBot's block rate tripled while Googlebot's held at 3.7%. 15 markets inside.

Emre Elbeyoglu · Founder & CEO, SEOmator

16+ years in B2B SaaS SEO and growth

Orange crawler bots stand in clusters of very different sizes on a tiled world map

How we know this

Full method ↓
Tested
AI-bot requests on Cloudflare's network by user agent, crawl purpose, HTTP response, content type and vertical, globally and for 20 markets, as calendar months (February, July and October 2026), plus Radar's robots.txt scan of 4,409 top-domain files on November 9, 2026 · November 10, 2026
Method
Cloudflare Radar REST API (ai/bots/summary, ai/bots/timeseries, bots/crawlers/summary, http/summary/bot_class and robots_txt/top/user_agents/directive), pulled on November 10, 2026. Shares are of AI-bot requests, not of all web traffic. Refusal rates per bot and per crawl purpose leave out six markets whose data moved in lockstep from September 21, 2026.
Limits
Radar sees only sites on Cloudflare and assigns each request to the country of the customer whose site received it, so a few large sites can move a small market. Radar revises past months, so figures here can differ from earlier editions of this post.
Run by
SEOmatorWritten up by Emre Elbeyoglu
On this page14 sections
  1. What is AI bot traffic?
  2. What does global AI bot traffic look like in October 2026?
  3. How often do websites refuse AI bots?
  4. How does AI bot traffic differ by country?
  5. Why does the map keep redrawing itself?
  6. What content are AI bots actually fetching?
  7. Which crawlers are websites trying to govern?
  8. Which industries do AI bots target most?
  9. What are AI bots actually doing with the content they crawl?
  10. How has AI bot traffic trended?
  11. How do you measure AI bot traffic on your own site?
  12. What should website owners do about AI bot traffic?
  13. Frequently asked questions
  14. How I analyzed this data

AI bot traffic is the share of web requests made by crawlers and agents that collect pages for AI models, AI search and AI assistants. In October 2026, Cloudflare Radar shows Googlebot leading it at 23.7% and ClaudeBot second at 12.6%, while websites answered one AI-bot request in six with a 403 Forbidden, up from one in ten in July.

📋
Key Findings:
  • Refusals nearly doubled in three months: 17.1% of AI-bot requests got a 403 in October 2026, against 9.6% in July (13.3% against 9.2% outside six markets flagged below). Only 57.7% got a 200 OK, down from 71.6%.
  • Sites now single out AI-only bots. GPTBot's 403 rate tripled (4.9% to 15.4%) and ClaudeBot's nearly doubled (14.1% to 27.1%), while Googlebot's stayed at 3.7%.
  • The bots that fetch pages for a user's question are refused most: user-action bots such as ChatGPT-User got a 200 on just 24.4% of requests in October.
  • Search crawling went from 3.4% of AI-bot requests in February to 11.6% in October. Half of OpenAI's AI-bot requests now come from its search and user bots, up from 23%.
  • Googlebot leads 11 of the 15 markets with a stable signal. ChatGPT-User leads Australia (54.7%) and Switzerland (43.5%), and ByteDance's crawlers make 57.4% of AI-bot requests to sites in Singapore.
  • Since the week of September 21, Radar's figures for six markets (Germany, the Netherlands, Canada, Japan, Brazil and Belgium) have moved in lockstep, so I treat them separately below.

When we first ran this analysis in February 2026, Googlebot made 35.9% of AI-bot requests and the question was simply which crawler led where. By October the bigger story is the response those crawlers get. Websites have started refusing AI bots at scale, and they're picking which ones: Googlebot gets through almost every time, while the training and answer bots of OpenAI, Anthropic and Meta hit a wall far more often. The web has started fighting back, and it is doing so at very different intensities depending on where your audience sits.

I re-ran the full analysis across 20 markets, by user agent, crawl purpose, content type, HTTP response, and robots.txt directives, to see exactly how AI-crawler traffic looks in late 2026 and how it shifted since February. Every number comes straight from the Cloudflare Radar API, compared across three calendar months (February, July and October 2026), and every share below is a share of AI-bot requests unless I say otherwise.

What is AI bot traffic?

AI bot traffic is every request a website receives from a crawler or agent working for an AI product, whether it collects training data, builds a search index or fetches a page because a person asked an assistant a question. Cloudflare Radar sorts these bots by the purpose their operators declare, and in October 2026 the split looked like this:

Crawl purposeWhat the bot doesExample botsShare of AI-bot requests, Oct 2026
TrainingDedicated training crawlers explicitly collecting data to build and improve AI modelsGPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider43.6%
Mixed purposeCrawlers that index for search and collect training data at the same time, without cleanly separating the twoGooglebot, Bingbot39.3%
SearchAI-powered search bots that crawl pages to generate search answersOAI-SearchBot, Claude-SearchBot, PerplexityBot, ShapBot11.6%
User actionBots that fetch pages in real time when users ask questionsChatGPT-User, Claude-User, Perplexity-User3.9%
UndeclaredNo purpose published by the operatorSmaller and unknown agents1.5%

That is a slice of a much larger bot pie. Across all HTTP requests Cloudflare served in October 2026, Radar classed 39.3% as bots and 60.7% as humans (http/summary/bot_class). The often-quoted "51% of internet traffic is bots" comes from Imperva's 2025 Bad Bot Report, which counts all automated traffic in 2024, including scrapers and attack bots. The two figures measure different things, so they don't contradict each other, and neither one is the AI-only share.

What does global AI bot traffic look like in October 2026?

Googlebot still leads all AI-related crawler traffic globally, but its share has fallen to 23.7%, down 12.2 percentage points from the 35.9% Radar shows for February. ClaudeBot is second at 12.6%, with Meta-ExternalAgent close behind at 11.0%. This is according to Cloudflare Radar (ai/bots/summary/user_agent), covering websites protected by Cloudflare's network of 330+ cities in 125+ countries.

AI botOperatorPrimary purposeFeb 2026Jul 2026Oct 2026
GooglebotGoogleSearch indexing + AI (mixed)35.9%24.5%23.7%
ClaudeBotAnthropicModel training10.5%16.5%12.6%
Meta-ExternalAgentMetaAI training14.3%12.5%11.0%
ApplebotAppleSearch + AI features2.6%6.5%8.3%
BingbotMicrosoftSearch indexing + AI (mixed)9.3%8.9%7.4%
GPTBotOpenAIModel training12.7%9.7%5.5%
AmazonbotAmazonAI training5.2%6.0%5.0%
ShapBotParallelSearch index for web APIs0.0%0.3%4.3%
OAI-SearchBotOpenAIChatGPT search2.5%1.6%4.3%
BytespiderByteDanceAI training3.3%5.3%3.7%
PerplexityBotPerplexityAI search0.2%0.3%2.6%
Claude-SearchBotAnthropicClaude web search0.6%3.5%1.8%
ChatGPT-UserOpenAIUser-triggered fetches1.3%1.3%1.5%

The top four crawlers now account for 55.5% of identified AI bot requests, a less concentrated market than February's 73.4%, because the runners-up have grown while Google shrank. Three shifts stand out:

  • ClaudeBot peaked in June at 19.8% and has settled at 12.6%, still above February's 10.5%.
  • GPTBot more than halved, from 12.7% to 5.5%, while OpenAI's search bot grew from 2.5% to 4.3%.
  • New search crawlers arrived fast. ShapBot, which Parallel uses to index pages for its web APIs, went from nothing in February to 4.3% in October, level with OAI-SearchBot.
📊
By the Numbers: In February 2026, 23% of OpenAI's AI-bot requests came from its search and user bots (OAI-SearchBot and ChatGPT-User). In October it was 51%: 5.8 of OpenAI's 11.3 points. OpenAI's crawling has shifted from collecting training data to answering questions. (Cloudflare Radar, ai/bots user_agent, calendar months, pulled November 10, 2026.)

How often do websites refuse AI bots?

Much more often than they did in the summer, and here is the finding that reframes the entire "should I block AI bots" debate. Cloudflare Radar's response-status breakdown (ai/bots/summary/response_status) shows the HTTP status codes crawlers actually receive, and websites are already saying no, at scale. In October 2026 only 57.7% of AI-bot requests got a 200 OK, and 17.1% got a 403 Forbidden, up from 9.6% in July:

HTTP responseMeaningFeb 2026Jul 2026Oct 2026
200 OKServed68.6%71.6%57.7%
403 ForbiddenRefused (blocked)7.5%9.6%17.1%
404 Not FoundPage doesn't exist3.7%4.2%8.1%
301 Moved PermanentlyRedirected6.2%5.3%6.4%
429 Too Many RequestsRate-limited (throttled)0.6%1.5%2.8%
302 FoundTemporary redirect7.7%2.5%2.6%

All 4xx responses together went from 16.0% of AI-bot requests in July to 29.1% in October. About half of that jump comes from the six markets I set aside further down. Leave them out and the 403 share still rose from 9.2% to 13.3%, and 4xx from 15.4% to 22.2%.

In other words, roughly one in five AI-bot requests (17.1% 403 plus 2.8% 429) is already being turned away by the sites it targets, through WAF rules, bot-management products, or origin rate limits. The "AI is scraping everything unchecked" narrative is out of date; a large and growing share of the web is refusing these bots by default. That also means AI answer engines are increasingly working from partial crawls: if your competitors block and you don't, your pages are the ones available to cite.

⚠️
Correction: The July edition of this post said only 46% of AI-crawler requests got a 200. That number came from Radar's bots/crawlers dataset, which also counts human visitors (35% of it in October) and non-AI bots. Filtered to AI bots, July's success rate was 71.6%. The refusal trend is real; the July level was overstated.

Which bots get refused most?

Sites aren't refusing AI bots evenly. They let Googlebot and Bingbot through and push back on the AI-only crawlers. The table leaves out the six lockstep markets so that one traffic source can't inflate any single bot:

AI bot403 rate, Jul 2026403 rate, Oct 2026200 OK, Oct 2026
Googlebot3.7%3.7%72.2%
Bingbot3.5%5.4%73.7%
Applebot6.8%11.2%66.4%
Amazonbot6.9%12.4%71.5%
GPTBot4.9%15.4%69.9%
OAI-SearchBot10.1%15.9%60.7%
Bytespider15.4%16.2%41.3%
PerplexityBot41.3%17.2%57.1%
Meta-ExternalAgent15.0%19.0%59.8%
Claude-SearchBot13.7%26.5%58.3%
ClaudeBot14.1%27.1%58.1%
ChatGPT-User22.3%28.2%33.6%
CCBot39.7%51.1%9.7%

GPTBot's refusal rate tripled in three months while Googlebot's didn't move. Anthropic's search bot is refused almost as often as its training bot, which suggests many sites block Anthropic at the firewall rather than token by token in robots.txt. PerplexityBot went the other way, from 41.3% refused in July to 17.2% in October, as its share of AI-bot requests grew more than sevenfold. And ShapBot, the newest big crawler, is almost never refused: 85% of its requests got a 200 in October, most likely because few firewall rules name it yet.

Are the bots that could cite you refused the most?

Yes. Grouping the same requests by crawl purpose shows the bots that answer users' questions, and could send you a visitor, getting refused more often than the training crawlers:

Crawl purpose403 rate, Jul 2026403 rate, Oct 2026200 OK, Jul 2026200 OK, Oct 2026
Mixed purpose (Googlebot, Bingbot)4.1%5.5%75.3%71.3%
Training11.9%18.1%71.7%62.7%
Search14.0%19.8%66.5%56.0%
User action25.4%34.9%47.2%24.4%
🚩
Red Flag: User-action bots, the ones that fetch your page while someone waits for an answer, got a 200 on only 24.4% of requests in October 2026. Another 18.9% hit a page that doesn't exist. If you block them, or an assistant cites a URL you've moved without a redirect, the answer gets written without your page.

Radar's AI-bot dataset identifies bots by their user agent, so some refused requests may come from impostors borrowing a real bot's name, which firewalls are right to stop. Impostors don't explain the split, though: in the same three months, sites kept letting Googlebot through at the same rate while turning GPTBot away three times as often.

💡
Quick Insight: In some markets a joke status code is doing real work. 418 I'm a teapot made 12.9% of AI-bot responses to sites in Mexico, 9.9% in Spain and 8.1% in Italy in October 2026. It's sometimes returned to unwanted bots in place of a 403, so count it as a refusal when you read your own logs.

How does AI bot traffic differ by country?

In February this section described a clean binary, a "Googlebot Belt" versus a "ChatGPT Corridor." That framing no longer survives contact with the data. The table covers the 15 markets in my set whose data behaved normally in October 2026. Radar assigns each request to the country of the Cloudflare customer whose site received it, not to the country the bot crawls from, so read each row as "AI bots visiting sites in this market".

Market#1 bot#1 share#2 bot#2 shareLargest purpose200 OK403
TurkeyGooglebot91.7%Bingbot1.3%Mixed (94.3%)15.2%58.3%
MexicoGooglebot85.5%Bingbot4.1%Mixed (89.7%)26.1%26.0%
PolandGooglebot70.4%ChatGPT-User12.7%Mixed (73.2%)33.6%32.3%
AustriaGooglebot68.6%imgproxy8.6%Mixed (73.6%)62.6%17.2%
IndonesiaGooglebot59.7%ChatGPT-User14.3%Mixed (67.3%)52.3%15.9%
ItalyGooglebot54.5%ChatGPT-User18.0%Mixed (57.8%)60.9%14.5%
SpainGooglebot49.9%ChatGPT-User25.4%Mixed (59.2%)52.6%16.3%
FranceGooglebot44.9%Meta-ExternalAgent11.4%Mixed (60.2%)60.2%15.7%
IndiaGooglebot44.1%ChatGPT-User16.3%Mixed (57.9%)64.0%13.6%
United KingdomGooglebot40.4%Meta-ExternalAgent14.2%Mixed (55.9%)64.6%12.3%
United StatesGooglebot27.1%Meta-ExternalAgent12.8%Mixed (45.0%)67.8%12.0%
South KoreaBingbot29.0%Googlebot28.0%Mixed (58.8%)72.5%13.7%
AustraliaChatGPT-User54.7%Googlebot16.1%User action (55.2%)73.9%15.2%
SwitzerlandChatGPT-User43.5%Googlebot24.3%User action (64.0%)69.4%10.8%
SingaporeBytespider48.9%TikTokSpider8.5%Training (65.3%)30.2%26.8%

Pattern 1: Googlebot majority markets

Googlebot holds the #1 position in 11 of the 15 markets I analyzed, and in Turkey (91.7%), Mexico (85.5%), Poland (70.4%) and Austria (68.6%) it makes most of all AI-related crawling. The United States is the most mixed market in the set: Googlebot leads with 27.1%, and Meta-ExternalAgent (12.8%) and ClaudeBot (12.7%) are tied behind it. Training and mixed-purpose crawling make 88.1% of AI-bot requests to US sites, so American websites are still mostly raw material for model building.

Pattern 2: ChatGPT-User markets

Australia and Switzerland are the two markets where ChatGPT-User, the bot that visits pages when real users ask ChatGPT questions, is the dominant crawler. It makes 54.7% of AI-bot requests to Australian sites, and user-action crawling is 55.2% of the total there. Switzerland goes further: ChatGPT-User (43.5%) and Claude-User (20.4%) push user-action crawling to 64.0%, the highest of any market I checked. ChatGPT-User is also the #2 bot in Spain (25.4%), Italy (18.0%), India (16.3%), Indonesia (14.3%) and Poland (12.7%). South Korea, where ChatGPT-User led in February, is now a three-way race: Bingbot 29.0%, Googlebot 28.0%, ChatGPT-User 24.6%.

Pattern 3: The Bytespider markets

ByteDance's Bytespider (48.9%) and TikTokSpider (8.5%) together make 57.4% of AI-bot requests to sites in Singapore, and training is 65.3% of crawling there. This is a training-first pattern, and the most training-heavy market I looked at. Where ChatGPT-User traffic signals end-user demand for AI answers, a Bytespider surge signals bulk data collection for model training: a different pressure on your infrastructure, and one that returns no referral traffic at all. India's July headline, Bytespider in first place with 29.4%, didn't last: by October Bytespider had dropped to 8.3% there, behind Googlebot (44.1%), ChatGPT-User (16.3%) and Bingbot (11.8%).

Where do sites refuse AI bots most?

Sites in Turkey answered 58.3% of AI-bot requests with a 403 in October 2026, by far the highest rate in the set, followed by Poland (32.3%), Singapore (26.8%) and Mexico (26.0%). Switzerland (10.8%), the United States (12.0%) and the United Kingdom (12.3%) refused the least. Indonesia stands out for throttling: 11.2% of its AI-bot responses were 429 Too Many Requests, four times the global rate.

Turkey's number needs a second look. Googlebot makes 91.7% of AI-bot requests there, so most of those 403s are landing on requests that say they're Googlebot. The real Googlebot is refused 3.7% of the time worldwide. Either many requests are impostors that firewalls are correctly blocking, or a few large Turkish sites are turning Googlebot away. If you run a site there, check that your firewall isn't doing the second.

Why does the map keep redrawing itself?

In February this post described two camps, a Googlebot belt and a ChatGPT corridor; in July, three. Each model broke because the AI-crawler market is still forming. Four forces are visibly at work in the country-level data:

  1. New entrants scale faster than incumbents. ShapBot, PerplexityBot, OAI-SearchBot and Applebot all grew their share while Googlebot's proportional footprint shrank, not because Google crawls less, but because everyone else crawls more. A ranking built on relative share reshuffles every month during a land grab.
  2. User-driven crawling follows consumer habit, not headquarters. Markets like Australia and Switzerland show high "User Action" percentages because people there ask assistants questions that need a live page. When someone in Sydney or Zurich asks ChatGPT a question, the bot fetches a page in real time, so consumer behavior, not the location of AI labs, drives that traffic.
  3. Training crawlers chase language and volume. English-heavy markets (the US) and large content pools draw the training-first crawlers, GPTBot, ClaudeBot, Meta-ExternalAgent and Bytespider, which is why those countries skew "Training" and "Mixed Purpose" rather than "User Action".
  4. A few large sites can move a whole market. Because Radar reports by the customer's country, one big platform can set a smaller market's mix. That's how Brazil went from ChatGPT-User at 55.2% in February to Googlebot first in July, and how India's Bytespider lead appeared and vanished within a quarter.

Six markets that moved in lockstep

From the week of September 21, 2026, Radar's figures for Germany, the Netherlands, Canada, Japan, Brazil and Belgium became almost identical week after week: ClaudeBot at 14% to 15% of AI-bot requests, PerplexityBot at 5% to 10%, and 71% to 76% of October's responses in the 4xx range. Six markets on four continents don't move together like that. The likeliest reading is one large traffic source that now outweighs everything else in those markets, so until it fades their rows describe that source, not the country.

MarketJul 2026 #1 botJul 2026 4xxOct 2026 #1 botOct 2026 4xx
GermanyGooglebot (39.2%)30.8%ClaudeBot (14.2%)70.8%
NetherlandsGooglebot (28.3%)57.9%ClaudeBot (14.1%)73.5%
CanadaGooglebot (38.9%)35.9%ClaudeBot (15.0%)74.9%
JapanGooglebot (23.1%)46.7%ClaudeBot (14.8%)75.9%
BrazilGooglebot (42.6%)42.5%ClaudeBot (14.5%)75.9%
BelgiumGooglebot (11.9%)70.3%ClaudeBot (14.9%)74.2%
⚠️
Common Mistake: Quoting a single market's Radar figure as "how that country treats AI bots". Before you cite a country, check its weekly series: a share that jumps in one week and then stays flat to the decimal is one source talking, not a national trend.

What content are AI bots actually fetching?

A breakdown Cloudflare Radar exposes but few analyses use is content type (ai/bots/summary/content_type), the MIME category of what AI crawlers request. Globally, AI bots overwhelmingly fetch rendered pages, but the non-HTML slice is where the interesting behavior hides.

Content typeFeb 2026Jul 2026Oct 2026
HTML74.8%72.5%73.0%
Plain text4.4%5.9%8.4%
JSON7.1%6.9%5.4%
JavaScript4.5%5.4%4.9%
Images4.9%5.3%4.5%
XML1.8%1.6%1.4%
CSS1.5%1.3%1.3%

Nearly three-quarters of AI-crawler requests are for HTML, the readable page. But 8.4% is plain text and 5.4% is JSON, and together that 13.8% tells you AI crawlers are increasingly pulling structured and raw data, not just prose: API responses, sitemaps, and the plain-text files (llms.txt, robots.txt, feeds) that agents parse directly.

Plain text nearly doubled this year, from 4.4% in February, and holds at 8.2% even without the six lockstep markets. The mix also varies by market: JSON makes 16.2% of AI-bot requests to UK sites and 15.0% in India, while images reach 14.5% in Austria and Singapore. If you want AI systems to consume your facts cleanly rather than guessing from rendered HTML, a machine-readable surface matters. Our scan of llms.txt adoption across the top 100,000 domains shows how few sites publish one so far.

Which crawlers are websites trying to govern?

If response codes show how sites react in real time, robots.txt shows their stated intent. Cloudflare Radar parses robots.txt files across its domain population and ranks the user agents those files name most often (robots_txt/top/user_agents/directive). The list is a near-perfect ranking of AI crawlers, a direct signal of which bots site owners are consciously writing rules for. Its November 9, 2026 scan parsed 4,409 files:

User agentOperatorFiles disallowing itOf which fullyFiles allowing it
GPTBotOpenAI754549360
ClaudeBotAnthropic687506323
CCBotCommon Crawl651526194
Google-ExtendedGoogle (AI training opt-out)622453317
BytespiderByteDance603526118
meta-externalagentMeta523430150
AmazonbotAmazon519420140
Applebot-ExtendedApple (AI training opt-out)493393187
PerplexityBotPerplexity385194357
ChatGPT-UserOpenAI362177323
GooglebotGoogle32715295

These are counts of domains that name each agent in a directive (a single-day snapshot), not traffic shares. A file can allow a bot into one folder and restrict it from the rest, so it can count in both columns.

The striking part is the last row. 549 of these files fully block GPTBot; 15 fully block Googlebot, the classic search crawler that has been around for two decades. Googlebot's 327 disallow rules are almost all partial, keeping it out of specific folders. GPTBot is also the bot most often explicitly allowed (360 files), so site owners are making a deliberate call on it either way.

Site owners are writing rules specifically for the AI-training crawlers, and the presence of Google-Extended (622 disallow rules) and Applebot-Extended (493), the opt-out tokens Google and Apple created specifically so publishers can refuse AI training while keeping search indexing, confirms this is a deliberate governance effort, not incidental config.

One caveat that makes this actionable: a robots.txt directive only works if it is syntactically valid and targets the right token, and it's a request, not enforcement. ShapBot isn't among the 40 agents robots.txt files name most, and it got a 200 on 85% of its requests. ClaudeBot is named in 687 disallow rules and got a 403 on 27.1% of its requests, which is the firewall layer at work.

Whichever you decide to allow or block, validate your robots.txt file afterwards: a malformed rule fails open, and you will believe you are blocking a crawler that is walking right through. Our robots.txt guide covers the syntax, and for one operator's bots in detail, see what ClaudeBot is and which Anthropic crawler you can safely block.

Which industries do AI bots target most?

According to Cloudflare Radar (ai/bots/summary/vertical), shopping and e-commerce remain the single most-crawled vertical, and its share keeps rising.

Industry verticalFeb 2026Jul 2026Oct 2026
Shopping and General Merchandise30.5%31.7%32.8%
Internet and Telecom16.9%16.4%16.2%
Computer and Electronics15.0%14.4%13.6%
News, Media and Publications9.0%9.8%8.2%
Business and Industry5.1%4.9%4.9%
Professional Services3.6%3.2%4.6%
Travel and Tourism4.0%4.3%3.7%
Gambling3.3%2.2%3.4%
Finance3.1%3.1%2.9%

Retail's 32.8% (up from 30.5% in February) still leads, which tracks with how heavily AI models are used for product research and comparison shopping: when a user asks an assistant "what's the best running shoe under $150," the bot needs current product pages and reviews to answer. Internet/Telecom (16.2%) and Computer/Electronics (13.6%) follow, because technical documentation and product specs are prime training and retrieval material. Together the top three verticals take 62.6% of AI-bot requests.

What are AI bots actually doing with the content they crawl?

Cloudflare Radar classifies AI bot activity into purpose categories (ai/bots/summary/crawl_purpose). The global split has tilted from training toward search since the summer:

Month (2026)TrainingMixed purposeSearchUser action
February46.2%47.9%3.4%2.1%
April46.8%47.5%3.1%2.2%
June52.6%39.0%5.0%2.6%
July50.7%39.9%5.6%2.6%
August48.2%37.1%8.4%4.7%
September44.0%40.5%10.6%3.1%
October43.6%39.3%11.6%3.9%

Training peaked at 52.6% in June and has fallen back below February's level, while search more than tripled, from 3.4% to 11.6%. Radar revises past months, and these are the figures it reported on November 10, 2026; our monthly AI crawler report tracks the same series month by month.

The country-level split remains sharp. In the United States, 88.1% of AI bot activity is Training or Mixed Purpose: US websites are primarily raw material for model building. In Switzerland, 64.0% of activity is User Action, and in Australia 55.2%: the majority of the crawl is triggered by real people asking AI for answers.

🔑
Key Takeaway: AI search crawling grew more than threefold this year, yet search bots are now refused more often than training bots (19.8% vs 18.1% in October, outside the six lockstep markets). A blanket "block AI" rule that made sense against training crawlers now also shuts out the bots that decide whether AI search can find you.

How has AI bot traffic trended?

AI-crawler request volume has climbed through 2026. The chart below tracks Cloudflare Radar's AI-bot traffic index over the trailing 24 weeks:

Source: Cloudflare Radar ai/bots/timeseries (normalized index, peak = 1.0). The weekly index sat around 0.65 in February, 0.78 in July and reached its 2026 peak of 1.00 in the week of September 28, a week after the six lockstep markets changed, so part of that late-September step is likely the same traffic source.

Zooming into the last month, the composition kept shifting even as the total grew. Comparing September and October 2026, calendar months:

AI botSep 2026Oct 2026Change
Googlebot24.5%23.7%-0.8 pp
ClaudeBot12.0%12.6%+0.6 pp
Meta-ExternalAgent13.1%11.0%-2.1 pp
Applebot8.5%8.3%-0.2 pp
GPTBot6.2%5.5%-0.7 pp
Amazonbot6.2%5.0%-1.3 pp
ShapBot1.5%4.3%+2.8 pp
OAI-SearchBot3.6%4.3%+0.7 pp
PerplexityBot2.2%2.6%+0.4 pp
Claude-SearchBot2.7%1.8%-0.8 pp

ShapBot (+2.8 pp) and OAI-SearchBot (+0.7 pp) were the month's biggest gainers, and Meta-ExternalAgent (−2.1 pp) the biggest loser. Bytespider rose 0.4 pp globally in the last month even after collapsing in India, a reminder that a crawler can be fading in a single market while holding steady in aggregate.

How do you measure AI bot traffic on your own site?

Radar shows the network-wide picture; your own logs show what reaches you. Three checks cover most of it:

Count requests by user agent in your server or CDN logs

Filter on the tokens in the tables above (GPTBot, ClaudeBot, Claude-SearchBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Meta-ExternalAgent, Bytespider, ShapBot, Googlebot, Bingbot). JavaScript analytics won't do this job, because most crawlers never run your scripts.

Verify the big ones

OpenAI, Anthropic, Google and Microsoft publish how to verify their crawlers by IP range or reverse DNS. Bytespider and PerplexityBot aren't in Cloudflare Radar's bots directory (checked November 10, 2026), so you can't confirm their requests the same way.

Read the status codes you return

Group each bot's requests by response: 200, 403, 404, 418, 429. A search or user bot you meant to allow that mostly gets 403s means a firewall rule is overriding your robots.txt.

For the visits AI assistants send back, rather than the crawls, our AI traffic analytics page shows which assistants send people to a site and what those visitors do after they land.

What should website owners do about AI bot traffic?

The blocking decision ultimately turns on a single number: how many pages a bot takes for every visitor it sends back. Our crawl-to-refer ratio analysis ranks every major operator on exactly that, a spread of four orders of magnitude, from DuckDuckGo's near-parity to Anthropic's lopsided extreme. With that ratio in hand, tailor your response to your market:

If your audience is in the US, the UK or France:

  • Expect Googlebot and, increasingly, Meta-ExternalAgent and ClaudeBot to be your primary AI crawlers. Because Google bundles search indexing with AI training under one bot, a robots.txt block on the AI portion is not cleanly possible: Google-Extended opts you out of Gemini training, but your control over AI features inside Google Search is limited.
  • Watch Meta-ExternalAgent and Bytespider. Meta's crawler is the #2 AI bot in all three markets; if your logs show Bytespider climbing, remember it isn't in Radar's bots directory, so you may need a WAF rule rather than a polite directive.
  • GPTBot and ClaudeBot honor robots.txt. Blocking both removes about 18% of all AI-bot requests (12.6% plus 5.5%) without touching OAI-SearchBot or Claude-SearchBot, which have their own tokens. Whichever way you go, validate the file: a malformed directive fails open.

If your audience is in Australia, Switzerland, South Korea or Spain:

  • ChatGPT-User is your primary concern and opportunity. In Australia it drives 54.7% of AI-crawler activity through real user questions. Blocking it means your content will not appear when someone in your market asks ChatGPT, a growing referral source, not a training cost.
  • Training crawlers are a smaller share here, so the "AI is stealing my content" problem is less about model training and more about whether you want real-time AI-search visibility.
  • Optimize for AI citation. Since so much AI-bot traffic in your market is user-triggered, structuring content with clear answers, statistics, and authoritative sources increases the odds of being quoted. A GEO audit for AI search engines shows how well your pages currently hold up on those signals, and the AI brand visibility checker shows whether assistants name you today.

If your audience is in Singapore or other training-heavy markets:

  • Bytespider may be your single largest AI crawler. It is training-first and returns no referral traffic, so the cost-benefit skews toward controlling it, but plan for enforcement (rate limits, bot management) rather than robots.txt alone, since you can't verify its requests the way you can GPTBot's. Our guide to preventing bots from crawling your site covers the options at the server and the edge.

If your audience is in Turkey, Mexico or Poland:

  • Sites there refuse AI bots at two to five times the US rate. Check that your firewall's bot rules match what you meant, especially for Googlebot and the user-action bots.

For everyone, expect to be refused, and design for it:

  • With only 57.7% of AI-bot requests globally receiving a 200 and 19.9% getting a 403 or 429, the crawl your competitors are surviving may be one you are silently blocking, or vice versa. Audit your own logs for AI user agents and the status codes you return them; an unintended block can quietly erase you from AI answers.
  • Serve a clean machine-readable surface (structured data, sitemaps, llms.txt) so the crawls you do allow extract your facts accurately instead of guessing from rendered HTML.
  • Keep your URLs stable and redirect what you move. User-action bots hit a 404 on 18.9% of their requests in October, and every one of those is an answer written without you.
📌
Pro Tip: Treat new crawlers as a standing review item. ShapBot went from 0.3% to 4.3% of AI-bot requests between July and October 2026 with almost no rules naming it. Check your logs monthly for user agents you haven't decided on, and test each new token in the robots.txt tester before you write the rule.

Frequently asked questions

How much web traffic comes from AI bots?

Cloudflare Radar classed 39.3% of all HTTP requests as bots in October 2026, but it doesn't publish AI bots as a share of all requests in the same series. Its AI-bot figures, like every share in this post, are shares within AI-bot traffic. Imperva's widely quoted 51% counts all automated traffic in 2024, including malicious bots.

Which AI bot crawls the most?

Googlebot, at 23.7% of AI-bot requests in October 2026, followed by ClaudeBot (12.6%) and Meta-ExternalAgent (11.0%). Counting only AI-specific crawlers, ClaudeBot is the largest.

Which AI bots get blocked most often?

Among the major crawlers, CCBot (51.1% of requests refused with a 403 in October 2026), ChatGPT-User (28.2%), ClaudeBot (27.1%) and Claude-SearchBot (26.5%). Googlebot was refused 3.7% of the time.

Does blocking GPTBot or ClaudeBot hurt Google rankings?

No. Google Search uses Googlebot, which is a separate token. Google also says Google-Extended, its AI-training opt-out, doesn't affect inclusion or ranking in Google Search (Google's crawler documentation).

Which country has the most AI bot traffic?

Radar doesn't publish a comparable cross-country volume figure for AI bots, only each market's internal mix. By mix, Turkey and Mexico are the most Googlebot-dominated markets, Switzerland and Australia the most user-driven, and Singapore the most training-heavy.

Why do AI bots get so many 404 errors?

8.1% of all AI-bot requests and 18.9% of user-action requests hit a page that doesn't exist in October 2026. Bots request old URLs, guessed URLs and links copied into chats, so redirects for moved pages matter more than they used to.

How I analyzed this data

This analysis uses Cloudflare Radar AI-bot data pulled directly from the Radar REST API on November 10, 2026. Every series is a calendar month: February 2026, July 2026 and October 2026, plus every month from February to October for the global user-agent and crawl-purpose trends. The robots.txt figures come from Radar's single-day scan of November 9, 2026; the month-over-month comparison uses September and October 2026, and the trend chart uses the trailing 24 weeks, with the weekly index from January to early November 2026 behind the figures in its caption. February and July figures cited in earlier editions came from 28-day windows and an earlier state of Radar's data, so they can differ from the calendar months here.

Endpoints queried: ai/bots/summary/{user_agent, crawl_purpose, content_type, response_status, response_status_category, vertical} for global and per-country breakdowns, with userAgent and crawlPurpose filters for the refusal tables; ai/bots/timeseries_groups weekly per market to check stability; bots/crawlers/summary/response_status by clientType to reconcile the July edition; robots_txt/top/user_agents/directive with directive=DISALLOW and ALLOW for the robots.txt governance snapshot; http/summary/bot_class for the all-traffic bot share; and ai/bots/timeseries for the trend index. Country breakdowns cover 20 markets: United States, United Kingdom, Germany, France, Netherlands, Canada, Japan, Australia, South Korea, India, Brazil, Spain, Italy, Mexico, Austria, Switzerland, Singapore, Indonesia, Turkey and Poland, with Belgium added to check the lockstep pattern.

All percentages represent share of identified AI bot requests, not share of total web traffic, except where noted: the robots.txt figures are domain counts (a snapshot), and the trend line is a normalized index (peak = 1.0), not an absolute request count. The location filter corresponds to the billing country of the Cloudflare customer whose site received the traffic, aggregated across Cloudflare's global network of 330+ cities in 125+ countries. Germany, the Netherlands, Canada, Japan, Brazil and Belgium are excluded (location=-DE,-NL,-CA,-JP,-BR,-BE) from the per-bot and per-purpose refusal tables and from every "excluding" figure, in both July and October so the comparison stays like for like. Radar identifies AI bots by user agent in this dataset, so a share of requests may come from impostors. Values shift daily and Radar revises history; re-run the queries before quoting them.

Data source: Cloudflare Radar: radar/ai/bots/*, radar/bots/crawlers/*, radar/http/*, and radar/robots_txt/* endpoints (radar.cloudflare.com), calendar months February, July and October 2026, pulled November 10, 2026. Last updated: November 10, 2026.

About this article

Who wrote it, what changed and how to tell us about a mistake.

Written by
Emre Elbeyoglu

Emre Elbeyoglu

Founder & CEO, SEOmator

16+ years in B2B SaaS SEO and growth

Emre Elbeyoglu is founder and CEO of SEOmator. Alongside running the company, Emre is its Head of Growth and leads all of SEOmator's growth work. Emre is also the founder and director of Next Marketing Technology Ltd, the London company that has owned and operated SEOmator since 2023. Emre has worked in B2B SaaS SEO and growth marketing for more than 16 years and builds every acquisition channel on search: that approach has generated $10M+ in total revenue from organic search across Emre's products, driven 300K+ monthly organic visitors and earned 10K+ referring backlinks.

In 2013 Emre co-founded Flatart, a digital marketing agency that ran SEO and growth programs for brands including Investing.com, Rakuten and EuroDNS. That consulting work turned into products: Popupsmart, an AI-powered conversion rate optimization platform used by 600+ clients, and LiveChatAI, an AI customer support tool covered by WIRED. Emre also publishes growth marketing experiments at GrowthMarketing.ai.

History
  1. October 2026 refresh. Every figure re-pulled from the Cloudflare Radar API as calendar months (February, July and October 2026) for 20 markets, with refusal rates per bot and per crawl purpose added. Corrects the July edition's "46% of AI-crawler requests get a 200 OK": that figure came from a Radar dataset that also counts human visitors and non-AI bots. Filtered to AI bots, July was 71.6%. The industry table moved to the AI-bot dataset for the same reason.Written by Emre Elbeyoglu

  2. July 2026 update (28 days ending July 23, 2026).Written by Nick Sawinyh

  3. Published.Written by Nick Sawinyh

Errors

Spot a mistake or a number that has gone stale? Tell us. We correct it and log the change in the History above. Report an error →

Explore more stories