Only 45.9% of crawler requests return a 200. The 2026 crawl waste report breaks the waste down by client type, industry, and operator — and finds AI bots are the cleanest traffic on the web, not the dirtiest.
Only 45.9% of requests in Cloudflare Radar's Web Crawlers dataset return a usable 200. The rest is blocks, errors, and redirect hops. But split that number by who is asking and it inverts: verified AI bots get a 200 back 73.0% of the time, while non-AI bots — SEO tools, scrapers, monitors — get one only 33.3% of the time. The wall the web has built is not aimed where most audits assume.
Key findings from the July 2026 data:
- Only 45.9% of requests return a 200; efficient 304 re-validations add just 1.2%.
- Verified AI bots see a 73.0% success rate — the cleanest of any client type. Non-AI bots see 33.3% and absorb a 29.2% block rate.
- 403 Forbidden is still the largest single non-200 bucket at 20.6%, but it fell 1.20 points month over month.
- Crawl-to-refer ratios for AI operators swing by multiples month to month (Anthropic halved, Mistral rose 20×); search engines stay flat.
- Retail absorbs 24.1% of all crawler traffic — more than any other industry.
- SEOmator's crawl of 10 major sites hit 99.2% 200s — the waste never touched the internal link graph.
Crawl budget waste is the share of a search engine's crawl activity spent on URLs that return no usable, indexable content: blocked pages, dead links, redirect chains, and near-duplicate parameter URLs. Every one of those requests spends part of your crawl budget without adding a single page to the index.
Crawl budget itself is the number of URLs a bot will fetch from your site in a given window. Google splits it into two parts: crawl capacity (how hard it can hit your server without degrading it) and crawl demand (how much it actually wants your pages). Most people talk about the budget. Almost nobody quantifies the waste, which is odd, because the waste is the part you can cut.
The waste shows up as common HTTP status codes that aren't 200: a 403 the bot was refused, a 404 it guessed wrong on, a 301 that sends it on another round trip. On a small site this rounds to zero. On anything past a few thousand URLs, and especially on sites with deep crawl depth, it decides how fast your real pages get indexed.
Across the whole web, fewer than half of requests come back with content. Here's the full response-status distribution from Cloudflare Radar for the 28 days ending 19 July 2026, with the change against the preceding 28-day window.
| Status | Share | Change vs prior 28d | Verdict |
|---|---|---|---|
| 200 OK | 45.9% | +0.38 | Usable content — the goal |
| 403 Forbidden | 20.6% | −1.20 | Blocked. Largest non-200 bucket |
| 301 Moved Permanently | 8.0% | +0.30 | Redirect hop |
| 404 Not Found | 7.8% | +0.75 | Wasted: guessed or dead URL |
| 429 Too Many Requests | 6.3% | −0.13 | Rate-limited |
| 302 Found | 4.6% | +0.12 | Redirect hop |
| 204 No Content | 1.6% | +0.06 | Empty response |
| 503 Service Unavailable | 1.3% | −0.20 | Server error |
| 304 Not Modified | 1.2% | −0.03 | Efficient re-validation — good |
| Other | 2.8% | −0.03 | Mixed |
Rolled up by status class, the same window reads 2xx 47.8%, 4xx 35.7%, 3xx 14.3%, 5xx 2.2%. So for every 100 requests, roughly 47 come back with something to read, 36 come back with a wall, and 14 send the client somewhere else first.
The headline number barely moves. Against the prior 28 days the 200 rate rose by about a third of a point. What changed underneath it was the shape of the waste: 404s climbed three-quarters of a point while 403 blocking eased by 1.20 points — the single largest movement in the table, and the opposite direction from the one most 2026 commentary predicts.
This is the breakdown that changes how you should read every other number on this page. Radar's crawler dataset can be filtered by client type, and the four segments behave nothing like the blended average.
| Client type | 200 | 403 | 404 | 429 | Share of requests |
|---|---|---|---|---|---|
| Mixed-purpose bots | 73.4% | 5.2% | 5.2% | 1.1% | 3.3% |
| Verified AI bots | 73.0% | 13.5% | 4.1% | 1.0% | 5.7% |
| Human traffic | 54.5% | 12.9% | 7.4% | 4.5% | 42.5% |
| Non-AI bots | 33.3% | 29.2% | 8.8% | 8.8% | 48.5% |
| All requests (blend) | 45.9% | 20.6% | 7.8% | 6.3% | 100% |
Verified AI bots are the best-treated automated clients on the web. They get content back nearly three-quarters of the time and get refused at 13.5% — below the 20.6% blended rate and less than half the 29.2% that non-AI bots absorb.
Non-AI bots are the segment dragging the global figure down. At 48.5% of requests they are the largest single population in the dataset, and a third of what they ask for comes back as a 403 or 429. That bucket is SEO crawlers, scrapers, uptime monitors, webhooks, and preview fetchers — and it is where the web's blocking budget is actually being spent.
One honest caveat, and it matters: Radar classifies verified AI bots — crawlers that identify themselves and pass Cloudflare's verification. An unverified scraper harvesting content while presenting a browser user-agent does not appear in that 5.7% row. It lands in the human or non-AI bucket. So the clean 73.0% describes bots that declare themselves and follow the rules, not the full population of AI-driven collection. Read it as "well-behaved AI crawlers are being let in," not "no one is scraping you."
Point a crawler at a site's internal links and it almost never finds the waste. In SEOmator's July 2026 crawl, a crawler that followed each site's own navigation and in-body links landed on a valid page 99.2% of the time. Radar's 45.9% and this 99.2% describe the same web from two different doors.
The gap comes from where each number is measured. A site crawl walks the link graph a site chose to expose. Those links point at real, canonical pages by design, so a well-run site returns almost all 200s. The request firehose Radar measures includes everything else: URLs guessed from old sitemaps, inbound links from other sites to pages you deleted, bots hammering paths you blocked, and years of redirects left behind by past migrations.
That's why this waste is so easy to miss. It doesn't live in the link graph a crawler discovers on its own. It lives in your server logs and in Search Console's crawl stats, where every real request is recorded, including the tens of thousands your navigation never points to. Two of the twelve sites we tried never reached the internal-link stage at all: Etsy returned a 403 and IMDb returned a 202 at the root — blocked at the door, exactly the pattern Radar measures globally.
One more caveat. Ten large, well-maintained sites are not a census of the web. A clean internal link graph on a well-run site understates real crawl waste, which piles up on small, legacy, and faceted e-commerce sites. Read the 99.2% only as a contrast, never as a web-wide success rate.
The bots crawling your site have changed. Radar's bot-category data for the same window shows AI-related crawlers now generating more requests than search engines do.
| Bot category | Share of bot traffic |
|---|---|
| Search-engine crawlers | 26.2% |
| AI crawlers (training) | 17.7% |
| SEO tools | 11.8% |
| AI assistants | 11.3% |
| Advertising and marketing | 7.8% |
| AI search | 6.6% |
| Page preview | 6.0% |
| Webhooks | 5.2% |
| Monitoring and analytics | 3.3% |
Add the three AI rows and you get 35.6% of bot traffic, comfortably ahead of the 26.2% that classic search crawlers like Googlebot generate. Within the AI segment specifically, ClaudeBot (18.7%) now trails only Googlebot (24.6%), ahead of Meta-ExternalAgent (11.2%) and GPTBot (9.7%).
Sorted by declared intent, 46.7% of AI crawler requests are for model training, 39.2% are mixed-purpose, and only 11.0% serve a live search index. That ratio is the real argument in the AI-blocking debate: most AI crawling is not fetching your page to cite it to a user right now. The economics of that trade are covered in our crawl-to-refer ratio report.
Crawler traffic is not spread evenly across the web. Radar's industry and vertical splits show it concentrating hard in commerce.
| Industry | Share of crawler traffic |
|---|---|
| Retail | 24.1% |
| Computer software | 16.8% |
| Gambling & casinos | 6.2% |
| IT and services | 6.1% |
| Marketing and advertising | 5.3% |
| Media | 5.0% |
| Internet | 4.7% |
| Adult entertainment | 4.6% |
Radar also publishes a broader site-vertical split, which tells the same story with different labels: shopping and general merchandise 26.7%, internet and telecom 20.5%, computer and electronics 18.6%, news and media 9.3%, gambling 6.7%.
Retail and shopping take roughly a quarter of all crawler attention — and retail is also the category most likely to run faceted navigation, the single largest generator of phantom URLs. The industry absorbing the most crawling is the one architecturally best equipped to waste it. If you run an e-commerce site, the parameter section below is the one to read twice.
Radar's non-200 buckets aren't random noise. Five of them account for almost all the waste, and each has a distinct root cause on the front end.
Every redirect is a wasted round trip: the bot asks for a URL, gets told to go elsewhere, then asks again. Chains multiply that cost. A page that 301s to a URL that 302s to a third address burns three requests to deliver one page. Site migrations, CMS replatforms, and HTTP-to-HTTPS moves each leave a layer of these behind, and they stack.
Flatten every chain to a single hop, then repoint your internal links and sitemap at the final destination. Start by learning to detect a 3xx redirect chain, then clean up the underlying 301 redirect and canonicalization rules so new ones don't accumulate.
A 404 usually means a page was removed but is still linked, whether from an old sitemap, an external site, or your own footer. Malformed parameters and stale internal links generate the rest. This is the fastest-growing waste bucket in the current data, up three-quarters of a point in a month while blocking fell.
Fix the internal links that point at dead pages, return 410 Gone for content you deliberately retired, and keep your XML sitemap free of URLs that no longer resolve.
This is where front-end architecture quietly generates waste. A filter UI that appends ?color=, ?size=, and ?sort= to the URL, or a client-side router that mints ?ref= and session permutations, can spawn thousands of near-duplicate phantom URLs from a handful of real pages. Crawlers dutifully request every combination, and almost none of them deserve an index slot.
<a href> in the markup. A crawler cannot tell a canonical category page from the 900th sort-order permutation of it; both are just links. Render facet controls as buttons that update state, and reserve anchors for URLs you would genuinely want indexed.Canonicalize parameter variants to their clean base URL, disallow the parameter paths in robots.txt, and avoid exposing every filter as a crawlable link. Our robots.txt directives guide covers the disallow patterns that stop this without hiding the pages you want crawled.
The largest non-200 bucket is a locked door — but as the client-type table showed, it is mostly closing on non-AI automated traffic, not on AI crawlers. Some of that is deliberate. A meaningful slice is accidental: a CDN rule, a firewall setting, or a bot-management toggle that catches legitimate crawlers in its net. The difference between a deliberate 403 and an accidental one is the difference between a policy and a leak.
Audit which user agents you're refusing and confirm it matches intent. Allow the search and citation crawlers you want indexing you, and reserve blocks for traffic you've actually decided to turn away. If you're making that decision deliberately, you need to see the traffic first, which is what AI-crawler analytics are for.
A 429 says "too many requests, slow down." Here too the burden is uneven: non-AI bots absorb an 8.8% rate-limit rate against 1.0% for verified AI bots. Rate-limiting Googlebot delays your own indexing; rate-limiting aggressive scrapers is fine. The problem is a blanket limit that can't tell them apart.
Raise capacity or tune your rate limits so verified crawlers get through while abusive traffic still gets throttled.
Only 1.2% of requests end in a 304 Not Modified, and that number is quietly one of the worst in the table. Every crawl of unchanged content that returns a full 200 instead of a 304 spends bandwidth and crawl budget re-downloading bytes the client already has.
ETag at the CDN or derive it from a build hash, so a deploy that changes nothing about a given page still mints a fresh identifier. The crawler's If-None-Match then never matches and it re-downloads content that is byte-identical to what it already had. Deriving validators from page content rather than build identity is what turns those requests into cheap 304s — and it's worth checking before you go hunting for redirect chains.If you're deciding which bots to allow, the tempting metric is the crawl-to-refer ratio: how many pages an operator takes for every visitor it sends back. Radar publishes it per operator. The trap is that it is wildly unstable for exactly the operators people use it to judge.
| Operator | Crawl:refer (28d to 19 Jul) | Prior 28d | Movement |
|---|---|---|---|
| Anthropic | 2,302:1 | 4,825:1 | Halved |
| Mistral | 1,653:1 | 80:1 | Up ~20× |
| Perplexity | 225:1 | 192:1 | Up 17% |
| OpenAI | 214:1 | 937:1 | Down 77% |
| Microsoft | 34.6:1 | 35.0:1 | Flat |
| Yandex | 26.1:1 | 25.5:1 | Flat |
| Baidu | 11.6:1 | 10.7:1 | Flat |
| ByteDance | 11.0:1 | 9.6:1 | Flat |
| 4.7:1 | 5.2:1 | Flat | |
| DuckDuckGo | 2.5:1 | 1.9:1 | Flat |
Two patterns. Search engines are stable — Google, Microsoft, Yandex, Baidu and DuckDuckGo all moved by less than a point across the two windows. AI operators are not: OpenAI's ratio fell by more than three-quarters while Mistral's rose twentyfold in the same month.
That instability is the finding. A blocking policy built on one month's ratio for a given AI operator is calibrated to noise. If you want to use this metric, track it over several windows before acting on it, and weight the stable search-engine rows more heavily than the volatile AI ones.
Robots.txt is where sites declare intent, and Radar's parse of robots.txt files across its domain population shows AI crawlers dominating the rules — even though, per the client-type data, they are also the crawlers most likely to be let through.
| User agent named in robots.txt | Directives |
|---|---|
| GPTBot | 735 |
| ClaudeBot | 647 |
| Google-Extended | 617 |
| CCBot | 604 |
| Bytespider | 524 |
| meta-externalagent | 457 |
| Amazonbot | 448 |
| PerplexityBot | 442 |
Every agent in the top eight is an AI crawler. Classic search bots barely feature. The web is writing far more rules about AI crawlers than about anything else, while the response data shows those same crawlers getting a 73.0% success rate. Declared policy and served reality are not the same thing — which is a good reason to check what your edge actually returns rather than trusting what your robots.txt says.
You can't cut what you can't see, and a site crawl alone won't show it. Measuring your own crawl waste takes three sources read together.
Search Console crawl stats. In Settings, open the Crawl stats report and read requests by response and by file type. This is the fastest way to see your host's real 200-versus-error split, straight from Googlebot. Google's own crawl budget documentation explains what each response class means for large sites.
Server-log analysis. Your logs are the ground truth. They record every request, including the guessed URLs, blocked bots, and legacy redirects that a link-following crawl never touches. This is the only place the full waste picture exists — and the only place you can segment by client the way the table above does.
A site crawl. Crawling your own link graph gives you the clean baseline: the pages you intend bots to reach and their status. Run a free SEO audit to surface redirect chains and broken internal links, and use a live crawl test to confirm what a bot receives at your URLs today.
Crawl budget waste is the portion of a crawler's requests to your site that return no indexable content: 403 blocks, 404s, redirect hops, rate-limited 429s, and duplicate parameter URLs. Globally, Cloudflare Radar puts usable 200 responses at 45.9% of all requests in its Web Crawlers dataset.
No — the opposite, in the current data. Verified AI bots receive a 200 response 73.0% of the time and a 403 only 13.5% of the time, while non-AI bots (SEO tools, scrapers, monitors) see 33.3% and 29.2% respectively. Note this covers verified AI crawlers; unverified scrapers presenting browser user-agents are classified elsewhere.
A crawl budget is the number of URLs a search engine will fetch from your site in a given period. Google frames it as crawl capacity (how much your server can handle) times crawl demand (how much Google wants your pages). Sites under about 1,000 pages rarely hit the limit.
Cut the non-200 buckets in order of size: audit accidental 403 blocks first, flatten redirect chains to one hop, return clean 404 or 410 responses for dead URLs, and stop faceted-navigation parameters from generating phantom URLs. Then point every internal link and sitemap entry at the final canonical URL.
There's no single official formula. Google's model is roughly crawl budget equals crawl capacity limit times crawl demand. The number that matters in practice is your waste rate: the share of crawler requests returning something other than a 200 or 304, which you read from server logs or Search Console.
Use three sources together. Search Console's crawl stats report breaks requests down by response and file type. Raw server logs are the only place blocked and guessed URLs appear. A site crawl maps your internal link graph. Compare the 200 rate across all three to find the leak.
Crawl depth is the number of clicks from your homepage to a given page. Pages buried deep get crawled less often and can drop out of the index. Shallow, well-linked architecture keeps important URLs within a few hops and spends crawl budget where it earns rankings.
Crawl demand is how badly a search engine wants to crawl your URLs, driven mainly by their popularity and how stale its stored copy is. High-demand pages get re-crawled often; low-demand pages wait. Waste erodes effective demand by teaching crawlers your URLs frequently return nothing.
Every request spent on a blocked or dead URL is attention not spent on pages you want indexed. On large sites the effect is measurable: according to mydigipal, 42% of B2B SaaS product pages are under-crawled or never indexed, often because waste soaks up the crawl before bots reach them.
For sites under about 1,000 pages, crawl budget is rarely the bottleneck. Google usually crawls them fully. Waste still matters when it takes the form of 403s or redirect chains that block or slow indexing, but the pages-per-day ceiling mostly bites on sites above 10,000 URLs.
Cloudflare Radar (external spine). Response-status, client-type, bot-category, industry, vertical, crawl-to-refer, and robots.txt figures come from Cloudflare Radar over the 28-day window of 21 June to 19 July 2026, with month-over-month comparisons drawn from the immediately preceding 28-day window (24 May to 21 June 2026) via the API's control-range parameter. Endpoints: radar/bots/crawlers/summary/{response_status, response_status_category, client_type, vertical, industry, crawl_refer_ratio}, radar/bots/summary/bot_category, radar/ai/bots/summary/{user_agent, crawl_purpose}, and radar/robots_txt/top/user_agents/directive (radar.cloudflare.com). Client-type segmentation uses the clientType filter (ai_bot, non_ai_bot, human, mixed_purpose). Shares are percentages of requests except crawl-to-refer, which is an unbounded ratio, and robots.txt rows, which are raw directive counts from a single-day snapshot. Re-verify before any future republish — these values shift daily, and the July re-run moved the 403 rate 1.20 points from the June reading.
A note on the dataset's population. Radar's Web Crawlers dataset is segmented by client type, and human traffic accounts for 42.5% of it. The headline 45.9% is therefore a blend across all client types, not a bots-only figure. The client-type table is the more precise reading of who gets served what.
SEOmator crawl (first-party). On 15 July 2026, SEOmator's site crawler attempted 12 large, well-maintained public sites across news, retail, SaaS, government, education, and entertainment, up to 60 pages each via internal-link discovery. Ten crawled successfully for 389 internal URLs, of which 386 (99.2%) returned 200. Two of them, Etsy and IMDb, blocked the crawler at the root with a 403 and a 202 before exposing a single internal link.
Limitations. The 99.2% figure is not a web-wide success rate. A clean internal link graph on a well-run site understates real crawl waste, which concentrates on small, legacy, and faceted e-commerce sites and in the non-link request firehose that only server logs and Search Console capture. Radar's AI-bot classification covers verified crawlers only; unverified scrapers are counted elsewhere and their treatment is not visible in these figures. Status codes reflect what was measured in the stated windows and vary by region, bot-management rules, and personalization.
SEOmator Rank Tracker
Technical fixes are invisible until positions move. Track the keywords a fix was meant to affect and watch whether it actually landed.
SEOmator Rank Tracker
Benchmark your site against real 2026 data: Cloudflare Radar crawl ratios, Web Almanac Core Web Vitals pass rates, and post-AI-Overview CTR — every figure sourced and dated.
Mistral crawls 3,389 pages for every referral it sends back — now the most extractive AI bot on the web, ahead of Anthropic (2,237:1) and OpenAI (217:1). I cross-checked Cloudflare Radar's global network against SEOmator's own 500+ site panel to rebuild the crawl-to-refer ratio for July 2026.
Discover the top 10 technical SEO issues affecting websites in 2025, with data-driven solutions. Learn how to fix crawlability problems, improve page speed, resolve duplicate content, and implement schema markup to boost your search rankings.
