ClaudeBot is Anthropic's AI training crawler, one of three. In our check of 10,000 top sites, 64% of those that block it still let Claude-SearchBot in.
Emre Elbeyoglu · Founder & CEO, SEOmator
16+ years in B2B SaaS SEO and growth
How we know this
Full method ↓
Written up by Emre ElbeyogluClaudeBot is Anthropic's web crawler for AI training data. It's one of three Anthropic bots: Claude-SearchBot indexes pages for Claude's search answers, and Claude-User fetches a page when someone asks Claude about it. Blocking ClaudeBot keeps new content out of training without removing you from Claude: of the top-10,000 sites that fully block it, 64% still let Claude-SearchBot in.
ClaudeBot token, reads robots.txt, and is separate from Claude-SearchBot (search indexing) and Claude-User (user-requested fetches), so each can be allowed or blocked on its own.Searching for Clawdbot instead? That's a different thing: an open-source personal AI assistant that was renamed Moltbot and then OpenClaw in January 2026, after Anthropic raised trademark concerns, according to Wikipedia's OpenClaw entry. It doesn't crawl the web for Anthropic. Everything below is about Anthropic's crawlers.
Anthropic runs three bots, each with its own robots.txt token and its own job. Anthropic's crawler help page, last updated 7 April 2026, lists them and spells out what disabling each one does. The October share column is Cloudflare Radar's measure of AI-bot requests across the sites it serves.
| Bot | robots.txt token | Job | What blocking it does (Anthropic's wording, summarised) | Share of AI-bot requests, Oct 2026 |
|---|---|---|---|---|
| ClaudeBot | ClaudeBot | Collects pages that may be used to train Claude models | Your future pages are excluded from training datasets | 12.6% |
| Claude-SearchBot | Claude-SearchBot | Indexes pages to improve Claude's search results | May reduce your visibility and accuracy in Claude's search results | 1.8% |
| Claude-User | Claude-User | Fetches a page when a Claude user's question needs it | Claude can't retrieve your page for that user; may reduce visibility | 0.04% |
The split matters because the three bots map to three different trade-offs. ClaudeBot is about training: whether your text helps shape the next model. Claude-SearchBot and Claude-User are about retrieval: whether Claude can find, read and cite your page when a user asks a question right now. A site can say no to the first and yes to the other two.
Two older tokens still turn up in robots.txt files: anthropic-ai and Claude-Web. Neither appears on Anthropic's current list of three bots. Under the robots.txt standard a crawler matches its own token, so a rule that names only the old tokens doesn't apply to ClaudeBot, which falls back to your * rules instead. More on how often that happens in the study section.
Cloudflare's bot directory files the three under different categories: ClaudeBot as an AI crawler, Claude-SearchBot as AI search, Claude-User as an AI assistant. OpenAI runs the same three-way split (GPTBot, OAI-SearchBot, ChatGPT-User), which is why the decision below carries over to ChatGPT almost one for one.
ClaudeBot made 12.6% of the AI-bot requests that Cloudflare Radar recorded in October 2026 (1 to 30 October), second only to Googlebot at 23.6% and more than twice GPTBot's 5.5%. Add Claude-SearchBot and Claude-User and Anthropic's three bots made 14.4% of AI-bot requests.
The monthly series shows how bursty ClaudeBot is. Its share doubled from May to June, then settled back near 12%:
| Month (2026) | ClaudeBot | Claude-SearchBot | GPTBot | OAI-SearchBot |
|---|---|---|---|---|
| April | 11.6% | 0.8% | 9.8% | 1.9% |
| May | 9.9% | 2.1% | 11.2% | 1.7% |
| June | 19.8% | 3.3% | 9.4% | 1.5% |
| July | 16.5% | 3.5% | 9.7% | 1.6% |
| August | 11.9% | 3.7% | 8.3% | 3.1% |
| September | 12.0% | 2.7% | 6.2% | 3.6% |
| October (to the 30th) | 12.6% | 1.8% | 5.5% | 4.3% |
Two caveats before you quote these. Radar reports shares, not request counts, so a falling share can still mean more requests if total AI-bot traffic grew. And Radar revises recent months as its bot classification improves, so we pulled every month on the same day rather than reusing older pulls. The monthly AI crawler report tracks the full operator league table, and our AI bot traffic by country study shows how the mix flips between markets; this post sticks to Anthropic.
Yes, and much more than it did in the spring. Radar's crawl-to-refer ratio divides an operator's crawl requests by the visits its products refer back to sites. For Anthropic it fell from 11,198 crawls per referral in April 2026 to 482 in October, a 23-fold improvement in six months.
| Month (2026) | Anthropic | OpenAI | Perplexity |
|---|---|---|---|
| April | 11,198 : 1 | 1,027 : 1 | 123 : 1 |
| May | 12,874 : 1 | 1,066 : 1 | 139 : 1 |
| June | 3,401 : 1 | 652 : 1 | 209 : 1 |
| July | 1,914 : 1 | 256 : 1 | 296 : 1 |
| August | 910 : 1 | 424 : 1 | 822 : 1 |
| September | 549 : 1 | 341 : 1 | 2,373 : 1 |
| October (to the 30th) | 482 : 1 | 775 : 1 | 4,056 : 1 |
In October Anthropic's ratio fell below OpenAI's for the first time this year, but read that with care: most of OpenAI's jump is a measurement change. ChatGPT's links stopped passing a referrer in early October, so Cloudflare counts fewer of its clicks; the breakdown linked below shows the evidence from our own analytics. Perplexity moved the other way, from 123:1 to 4,056:1. A ratio has two moving parts (crawl volume and referral volume), so a falling number can mean Claude users clicked through more, Anthropic's bots crawled less, or both.
ClaudeBot is the one you can block without losing your place in Claude's answers. Anthropic says a ClaudeBot block signals that your future pages should be excluded from its training datasets, and its help page attaches the "may reduce your visibility" warning only to Claude-SearchBot and Claude-User. Blocking ClaudeBot also has no effect on Google or Bing rankings.
What a ClaudeBot block can't do is undo the past. Anthropic describes it as covering your future materials, so pages it already collected aren't removed, and Claude can still describe your brand from earlier training or from other sites.
Here is how the three tokens line up against four common goals:
| Your goal | ClaudeBot | Claude-SearchBot | Claude-User |
|---|---|---|---|
| Most visibility in Claude | Allow | Allow | Allow |
| In Claude's answers, out of training | Disallow | Allow | Allow |
| Less crawl load, still in | Crawl-delay | Allow | Allow |
| Out of Anthropic entirely | Disallow | Disallow | Disallow |
My default for a B2B software site is the first row, because open retrieval bots are the starting point for any answer engine optimization work. Training is how a model learns that your category exists and that your product sits in it; a model that never read your pages can only describe you from what others wrote. The second row is the right call when your text is the product (paid research, a publisher's archive, course material) and you'd rather license it than donate it.
Most sites that block ClaudeBot already get the split right. The trouble sits in catch-all rules and in tokens Anthropic no longer lists. We fetched the robots.txt of every domain in the Tranco top 10,000 on 31 October 2026, and 5,386 of them served one. Every percentage below is out of those 5,386 files, and the method is at the end.
| Token | Named in | Blocked by name | Blocked incl. * |
|---|---|---|---|
| ClaudeBot | 1,045 (19.4%) | 685 (12.7%) | 843 (15.7%) |
| Claude-SearchBot | 387 (7.2%) | 139 (2.6%) | 302 (5.6%) |
| Claude-User | 395 (7.3%) | 183 (3.4%) | 345 (6.4%) |
| anthropic-ai (old) | 755 (14.0%) | 544 (10.1%) | n/a |
| Claude-Web (old) | 628 (11.7%) | 469 (8.7%) | n/a |
| GPTBot, for comparison | 1,179 (21.9%) | 740 (13.7%) | 887 (16.5%) |
| OAI-SearchBot, for comparison | 609 (11.3%) | 172 (3.2%) | 323 (6.0%) |
"Blocked" means fully blocked. "Blocked incl. *" adds the sites that never name the bot but disallow everything in their User-agent: * group. The old tokens get no figure there, because none of Anthropic's current bots reads them.
Claude-SearchBot is barely on anyone's list. Only 387 files (7.2%) name it, against 1,045 that name ClaudeBot. Of those 1,045 files, 364 (34.8%) mention Claude-SearchBot at all; the rest leave it to whatever their * group says.
Deliberate ClaudeBot blocks mostly keep AI search open. Of the 685 sites that block ClaudeBot by name, 538 (78.5%) leave Claude-SearchBot able to crawl. The 147 that block it too nearly all do it by name (139 of them), which reads as a decision, not an accident.
The catch-all * is where search gets blocked without anyone naming it. Across all 843 sites where ClaudeBot is fully blocked, 302 also shut out Claude-SearchBot, and 163 of those 302 do it only through a User-agent: * group that disallows everything.
The old tokens are the other surprise. 806 files (15.0%) still name anthropic-ai or Claude-Web, more than twice the 387 that name Claude-SearchBot. Most of those files name ClaudeBot as well, so the old lines are dead weight rather than a broken block. But 47 sites fully block an old token without ever naming ClaudeBot. Going by the standard's matching rules, ClaudeBot isn't fully blocked on any of the 47: it falls through to the * group, which on 42 of them closes only a few paths.
anthropic-ai or Claude-Web but never says ClaudeBot, you're relying on tokens Anthropic no longer documents, and the standard says ClaudeBot won't match them. We found 47 top-10,000 sites in exactly that state on 31 October 2026.Cloudflare's own robots.txt scan ranks the tokens the same way on a different set of sites. In its 26 October 2026 scan of 4,367 robots.txt files from top domains, ClaudeBot had a Disallow rule in 567 files, the old anthropic-ai and Claude-Web tokens in 356 and 307, and Claude-SearchBot in 200.
Write one group per token in the robots.txt at the root of each host. Under RFC 9309, a crawler follows the group (or groups) that name its token and ignores the rest; if no group names it, it follows your User-agent: * group. That fallback is where most accidental blocks come from.
To keep new pages out of training but stay in Claude's answers:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
The two Allow groups look redundant, and on a file with a permissive * group they are. They earn their place when your * group disallows anything: without them, Claude-SearchBot and Claude-User inherit those rules.
User-agent: * / Disallow: / group written to stop scrapers also stops every bot you didn't name, including Claude-SearchBot and Claude-User. In our check, 185 top sites (3.4%) run a blanket * block, and only 22 of them name Claude-SearchBot. If you run a blanket block, add an explicit group for each AI search and user bot you want to keep.To slow ClaudeBot down instead of blocking it, Anthropic supports the non-standard Crawl-delay directive, which Googlebot ignores:
User-agent: ClaudeBot
Crawl-delay: 1
To keep all of Anthropic out, disallow all three tokens. Add anthropic-ai and Claude-Web too if you want to cover older tooling, but don't rely on them alone:
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /
Before you ship a change, check it the way the bot will read it:
Paste your file into our robots.txt tester and test ClaudeBot, Claude-SearchBot and Claude-User one at a time against a key URL. Each should return the verdict you meant.
Anthropic reads robots.txt per host, so docs.example.com and blog.example.com need their own rules. A root-domain file doesn't cover them.
Bot-management rules at the CDN or WAF can block a crawler your robots.txt allows. If you want Claude-SearchBot in, make sure the edge lets it through too.
Our robots.txt guide covers the rest of the syntax, including the rules Google ignores.
Match the token in the user-agent string, then check the IP against Anthropic's published list. User-agent strings are trivial to fake, and the full strings vary: Cloudflare's bot directory lists two different Claude-User strings, for example. Matching on ClaudeBot, Claude-SearchBot or Claude-User catches the real ones; the IP check filters the impostors.
Anthropic publishes its crawler IP ranges at claude.com/crawling/bots.json and links that file from its help page. The version we fetched was dated 20 October 2026 and held 14 IPv4 prefixes. Some pages ranking for "claudebot" still say Anthropic publishes no IP ranges, but its help page links this list.
You also won't see any of this in Google Analytics or any other script-based analytics. Crawlers don't run JavaScript, so ClaudeBot, Claude-SearchBot and Claude-User show up only in server logs or your CDN's bot reports. What script-based analytics can show is the other side: the visitors Claude sends you. Our AI traffic analytics splits those visits by assistant, so you can see whether Claude referrals move after a robots.txt change.
The ClaudeBot problems we found in robots.txt files come from rules written for a different bot, or for a version of Anthropic's lineup that it no longer documents.
anthropic-ai and Claude-Web aren't on Anthropic's current list. We found 47 top sites that block an old token without naming ClaudeBot; by the standard's matching rules, ClaudeBot isn't fully blocked on any of them. Add a ClaudeBot group.* block that takes AI search with it. Rules meant for scrapers end up covering Claude-SearchBot and Claude-User. Name the bots you want to keep.ClaudeBot identifies itself with the token ClaudeBot in its user-agent string; Cloudflare's bot directory lists it as ClaudeBot/1.0 (+https://anthropic.com/claudebot). In robots.txt you only need the token: User-agent: ClaudeBot. Anthropic's other two bots use Claude-SearchBot and Claude-User.
No. ClaudeBot is Anthropic's training crawler. Clawdbot was an open-source personal AI assistant, unrelated to Anthropic, that renamed itself Moltbot and then OpenClaw in January 2026. If you landed here looking for that project, you want OpenClaw.
Anthropic documents robots.txt as the way to opt out of its bots and supports the non-standard Crawl-delay directive. We can't audit compliance from robots.txt files alone; if you see ClaudeBot on paths you disallowed, check that the IP is on Anthropic's list (user-agent strings are easy to fake) and then email claudebot@anthropic.com from an address on your domain, as Anthropic's help page asks.
No. ClaudeBot has no role in Google or Bing search. Google's own crawlers (Googlebot, and the Google-Extended token for Gemini training) are separate, and a ClaudeBot rule doesn't touch them.
No. Claude's live web answers rely on Claude-SearchBot and Claude-User, which a ClaudeBot rule doesn't cover. To leave Claude's answers you'd have to disallow those two as well, and Anthropic warns that doing so may reduce your visibility in Claude's search results.
Both are training crawlers: GPTBot collects pages for OpenAI's models, ClaudeBot for Anthropic's. On Cloudflare's network ClaudeBot currently makes more than twice as many AI-bot requests as GPTBot. Each lab also runs a search bot (OAI-SearchBot, Claude-SearchBot) and a user-fetch bot (ChatGPT-User, Claude-User), and you control all six separately.
Claude-SearchBot is the Anthropic crawler that indexes pages to improve the results of Claude's web search. It's the bot that lets Claude find your page when a user's question needs fresh information. Anthropic lists training as ClaudeBot's job, not Claude-SearchBot's, so a Claude-SearchBot block is about whether Claude's search knows your page exists.
No. ClaudeBot doesn't execute JavaScript, so script-based analytics never record it. Look in your server access logs or your CDN's bot analytics for the ClaudeBot token, and verify the IP against Anthropic's bots.json list.
We fetched /robots.txt from every domain in the Tranco top-sites list (list ID 746VX, downloaded 31 October 2026), taking the top 10,000. Each domain got one HTTPS request with a self-identifying user agent, retried on www. when the bare domain returned no file, with a 10-second timeout. A response counted as a robots.txt file only if it returned HTTP 200 and its body parsed as robots.txt rather than an HTML page.
5,386 of the 10,000 domains served one. The other 4,614 returned an error or an HTML page, timed out, or had no website at all: 1,508 names didn't resolve, because many Tranco entries are CDN and API hostnames rather than sites.
For each file we grouped the rules the way RFC 9309 describes: consecutive User-agent lines share one group, tokens match case-insensitively, a bot follows every group that names it, and a bot named nowhere follows the * group. We then classified each token two ways: what the file says when it names the token, and what the bot actually gets once the * fallback applies. "Fully blocked" means Disallow: / (or /*) with no Allow rule; "partly blocked" means path-level disallows only.
The traffic and referral figures come from Cloudflare Radar's API, pulled on 31 October 2026: ai/bots/summary/user_agent for shares of AI-bot requests and bots/crawlers/summary/crawl_refer_ratio for crawl-to-refer ratios. We pulled each calendar month separately so every month covers the same span; October runs from 1 to 30 October. Radar sees only traffic that passes through Cloudflare, reports shares rather than counts, and revises recent months.
The limits are worth stating plainly. We read the root robots.txt only, so a subdomain with its own rules isn't covered. This is one day's snapshot. robots.txt states intent and doesn't enforce anything, so a site can block a bot at its CDN while the file allows it, and we didn't test whether any bot obeys the files it reads.
Open your robots.txt and look for four tokens: ClaudeBot, Claude-SearchBot, Claude-User and *. Decide which of the four goals in the table above is yours, write one group per bot, and test each token in the robots.txt tester before you deploy. If your goal is to stay in Claude's answers, keep the two retrieval bots open, then check your brand's AI visibility to see how assistants describe you today. robots.txt is the only file Anthropic documents for crawler control; an llms.txt is optional, and our llms.txt census found few top sites publish one.
Who wrote it, what changed and how to tell us about a mistake.
Emre Elbeyoglu
Founder & CEO, SEOmator
16+ years in B2B SaaS SEO and growth
Emre Elbeyoglu is founder and CEO of SEOmator. Alongside running the company, Emre is its Head of Growth and leads all of SEOmator's growth work. Emre is also the founder and director of Next Marketing Technology Ltd, the London company that has owned and operated SEOmator since 2023. Emre has worked in B2B SaaS SEO and growth marketing for more than 16 years and builds every acquisition channel on search: that approach has generated $10M+ in total revenue from organic search across Emre's products, driven 300K+ monthly organic visitors and earned 10K+ referring backlinks.
In 2013 Emre co-founded Flatart, a digital marketing agency that ran SEO and growth programs for brands including Investing.com, Rakuten and EuroDNS. That consulting work turned into products: Popupsmart, an AI-powered conversion rate optimization platform used by 600+ clients, and LiveChatAI, an AI customer support tool covered by WIRED. Emre also publishes growth marketing experiments at GrowthMarketing.ai.
Published
Spot a mistake or a number that has gone stale? Tell us. We correct it and log the change in the History above. Report an error →
· 21 min read
Perplexity now crawls 4,056 pages for every referral it sends back, up from 112 in January. Anthropic fell to 482:1, below OpenAI for the first time, as ChatGPT's links stopped passing a referrer. Cloudflare Radar's crawl-to-refer ratios, month by month to October 2026.
· 35 min read
I completed six AI SEO courses in the past year. Here's the order I'd take them in, who each one suits, and what they cost, from free to $319 a month.
· 17 min read
Only 46% of AI-crawler requests now get a 200 OK, more than a quarter are blocked or throttled, and which bot dominates flips completely by country.
