Crawl Waste Report 2026: Only 45.8% of Crawler Requests Return a 200

Only 45.8% of crawler requests to the web return a usable 200, according to Cloudflare Radar. The rest is blocks, errors, and redirect hops. Yet SEOmator's July 2026 crawl of 10 major sites saw 99.2% return 200. Crawl budget waste hides in your server logs, not your clean link graph.

Key findings from the 2026 data:

• Only 45.8% of global crawler requests return a 200; efficient 304 re-validations add a healthy 1.2%.
• 36% of requests are blocked or error out (403, 404, 429, 503); 403 alone is the single largest non-200 bucket at 20.5%.
• Redirect hops (301 + 302) burn 12.5% of all crawler requests.
• AI bots now generate about 35.7% of crawler traffic, out-crawling classic search engines at 26.3%.
• SEOmator's crawl of 10 major sites hit 99.2% 200s — the waste never touched the internal link graph.

What is crawl budget waste?

Crawl budget waste is the share of a search engine's crawl activity spent on URLs that return no usable, indexable content: blocked pages, dead links, redirect chains, and near-duplicate parameter URLs. Every one of those requests spends part of your crawl budget without adding a single page to the index.

Crawl budget itself is the number of URLs a bot will fetch from your site in a given window. Google splits it into two parts: crawl capacity (how hard it can hit your server without degrading it) and crawl demand (how much it actually wants your pages). Most people talk about the budget. Almost nobody quantifies the waste, which is odd, because the waste is the part you can cut.

The waste shows up as common HTTP status codes that aren't 200: a 403 the bot was refused, a 404 it guessed wrong on, a 301 that sends it on another round trip. On a small site this rounds to zero. On anything past a few thousand URLs, and especially on sites with deep crawl depth, it decides how fast your real pages get indexed.

Where do crawler requests actually go in 2026?

Across the whole web, fewer than half of crawler requests come back with content. Here's the full response-status distribution from Cloudflare Radar for the 28 days ending 15 July 2026.

Three buckets tell the story. Usable content (200) is 45.8%. Blocked or errored responses (403, 404, 429, 503) add up to 36%. Redirect overhead (301 plus 302) eats another 12.5%. So for every 100 pages a crawler asks for, roughly 46 come back with something to read, 36 come back with a wall, and 13 send it somewhere else first.

The number barely moves. Compared with the prior 28 days, the 200 rate shifted by a single hundredth of a point. What changed underneath it was the shape of the waste: 404s and 429s each climbed about three-quarters of a point while 403 blocking eased by 1.58 points.

Why does your own site crawl look so clean?

Point a crawler at a site's internal links and it almost never finds the waste. In SEOmator's July 2026 crawl, a crawler that followed each site's own navigation and in-body links landed on a valid page 99.2% of the time. Radar's 45.8% and this 99.2% describe the same web from two different doors.

The gap comes from where each number is measured. A site crawl walks the link graph a site chose to expose. Those links point at real, canonical pages by design, so a well-run site returns almost all 200s. The request firehose Radar measures includes everything else: URLs guessed from old sitemaps, inbound links from other sites to pages you deleted, bots hammering paths you blocked, and years of redirects left behind by past migrations.

That's why this waste is so easy to miss. It doesn't live in the link graph a crawler discovers on its own. It lives in your server logs and in Search Console's crawl stats, where every real request is recorded, including the tens of thousands your navigation never points to. Two of the twelve sites we tried never reached the internal-link stage at all: Etsy returned a 403 and IMDb returned a 202 at the root, the same "blocked at the door" pattern Radar measures globally as 20.5% of every crawler request.

One honest caveat. Ten large, well-maintained sites are not a census of the web. A clean internal link graph on a well-run site understates real crawl waste, which piles up on small, legacy, and faceted e-commerce sites. Read the 99.2% only as a contrast, never as a web-wide success rate.

Who is burning your crawl budget in 2026?

The bots crawling your site have changed, and that change explains the rising blocks. Cloudflare Radar's bot-category data for the same window shows AI-related crawlers now generating more requests than search engines do.

Add the three AI rows and you get about 35.7% of crawler traffic, comfortably ahead of the 26.3% classic search crawlers like Googlebot generate. That single shift is the engine behind the 20.5% 403 rate. As AI crawlers multiplied, publishers started refusing them, and a refusal is a 403. The economics behind that decision are stark once you measure what each bot takes versus what it sends back, which is the argument in our crawl-to-refer ratio report.

Blocking a training bot is a defensible architecture decision. The trouble is that bot-management rules are blunt instruments, and an over-broad rule blocks the crawlers you actually want alongside the ones you don't. If you're going to allow or refuse AI traffic deliberately, you need to see it first, which is what AI-crawler analytics are for.

The biggest sources of crawl waste (and how to cut each)

Radar's non-200 buckets aren't random noise. Five of them account for almost all the waste, and each has a distinct root cause on the front end. Here's what drives each, and the fix I'd reach for first.

Redirect chains and legacy 301/302s (12.5% of requests)

Every redirect is a wasted round trip: the bot asks for a URL, gets told to go elsewhere, then asks again. Chains multiply that cost. A page that 301s to a URL that 302s to a third address burns three requests to deliver one page. Site migrations, CMS replatforms, and HTTP-to-HTTPS moves each leave a layer of these behind, and they stack.

Flatten every chain to a single hop, then repoint your internal links and sitemap at the final destination. Start by learning to detect a 3xx redirect chain, then clean up the underlying 301 redirect and canonicalization rules so new ones don't accumulate.

404s from guessed and dead URLs (7.7%)

A 404 usually means a page was removed but is still linked, whether from an old sitemap, an external site, or your own footer. Malformed parameters and stale internal links generate the rest. The share of 404s rose about three-quarters of a point in the last month, so this bucket is growing, not shrinking.

Fix the internal links that point at dead pages, return 410 Gone for content you deliberately retired, and keep your XML sitemap free of URLs that no longer resolve.

Faceted navigation and parameter explosion

This is where front-end architecture quietly generates waste. A filter UI that appends ?color=, ?size=, and ?sort= to the URL, or a client-side router that mints ?ref= and session permutations, can spawn thousands of near-duplicate phantom URLs from a handful of real pages. Crawlers dutifully request every combination, and almost none of them deserve an index slot.

Canonicalize parameter variants to their clean base URL, disallow the parameter paths in robots.txt, and avoid exposing every filter as a crawlable link. Our robots.txt directives guide covers the disallow patterns that stop this without hiding the pages you want crawled.

403s that block the bots you want (20.5%)

The largest non-200 bucket is a locked door. Some of it is intentional AI-bot blocking. A meaningful slice is accidental: a CDN rule, a firewall setting, or a bot-management toggle that catches legitimate crawlers in its net. The difference between a deliberate 403 and an accidental one is the difference between a policy and a leak.

Audit which user agents you're refusing and confirm it matches intent. Allow the search and citation crawlers you want indexing you, and reserve blocks for traffic you've actually decided to turn away.

429 rate-limiting under crawl load (6.5%)

A 429 says "too many requests, slow down." With AI bots now out-crawling search engines, servers see more concurrent crawling than they were provisioned for, and they throttle. Rate-limiting Googlebot delays your own indexing; rate-limiting aggressive scrapers is fine. The problem is a blanket limit that can't tell them apart.

Raise capacity or tune your rate limits so verified search crawlers get through while abusive traffic still gets throttled.

How do you measure your own crawl waste?

You can't cut what you can't see, and a site crawl alone won't show it. Measuring your own crawl waste takes three sources read together.

Search Console Crawl stats. In Settings, open the Crawl stats report and read requests by response and by file type. This is the fastest way to see your host's real 200-versus-error split, straight from Googlebot. Google's own crawl budget documentation explains what each response class means for large sites.

Server-log analysis. Your logs are the ground truth. They record every request, including the guessed URLs, blocked bots, and legacy redirects that a link-following crawl never touches. This is the only place the full waste picture exists.

A site crawl. Crawling your own link graph gives you the clean baseline: the pages you intend bots to reach and their status. Run a free SEO audit to surface redirect chains and broken internal links, and use a live crawl test to confirm what a bot receives at your URLs today.

Frequently asked questions about crawl budget waste

What is crawl budget waste?

Crawl budget waste is the portion of a crawler's requests to your site that return no indexable content: 403 blocks, 404s, redirect hops, rate-limited 429s, and duplicate parameter URLs. Globally, Cloudflare Radar puts usable 200 responses at just 45.8% of all crawler requests.

What is a crawl budget?

A crawl budget is the number of URLs a search engine will fetch from your site in a given period. Google frames it as crawl capacity (how much your server can handle) times crawl demand (how much Google wants your pages). Sites under about 1,000 pages rarely hit the limit.

How do you reduce crawl budget waste?

Cut the non-200 buckets in order of size: fix accidental 403 blocks first, flatten redirect chains to one hop, return clean 404 or 410 responses for dead URLs, and stop faceted-navigation parameters from generating phantom URLs. Then point every internal link and sitemap entry at the final canonical URL.

What is the crawl budget formula?

There's no single official formula. Google's model is roughly crawl budget equals crawl capacity limit times crawl demand. The number that matters in practice is your waste rate: the share of crawler requests returning something other than a 200 or 304, which you read from server logs or Search Console.

How do you check crawl budget?

Use three sources together. Search Console's Crawl stats report breaks requests down by response and file type. Raw server logs are the only place blocked and guessed URLs appear. A site crawl maps your internal link graph. Compare the 200 rate across all three to find the leak.

What is crawl depth?

Crawl depth is the number of clicks from your homepage to a given page. Pages buried deep get crawled less often and can drop out of the index. Shallow, well-linked architecture keeps important URLs within a few hops and spends crawl budget where it earns rankings.

What is crawl demand?

Crawl demand is how badly a search engine wants to crawl your URLs, driven mainly by their popularity and how stale its stored copy is. High-demand pages get re-crawled often; low-demand pages wait. Waste erodes effective demand by teaching crawlers your URLs frequently return nothing.

How does crawl waste affect indexing?

Every request spent on a blocked or dead URL is attention not spent on pages you want indexed. On large sites the effect is measurable: according to mydigipal, 42% of B2B SaaS product pages are under-crawled or never indexed, often because waste soaks up the crawl before bots reach them.

Does crawl waste matter for small sites?

For sites under about 1,000 pages, crawl budget is rarely the bottleneck. Google usually crawls them fully. Waste still matters when it takes the form of 403s or redirect chains that block or slow indexing, but the pages-per-day ceiling mostly bites on sites above 10,000 URLs.

Methodology and sources

Cloudflare Radar (external spine). The crawler response-status distribution and bot-category shares come from Cloudflare Radar's Web Crawlers dataset over the 28-day window of 17 June to 15 July 2026. Source: Cloudflare Radar — radar/bots/crawlers/summary/response_status (radar.cloudflare.com), 17 June – 15 July 2026. Re-verify these figures before any future republish.

SEOmator crawl (first-party). On 15 July 2026, SEOmator's site crawler attempted 12 large, well-maintained public sites across news, retail, SaaS, government, education, and entertainment, up to 60 pages each via internal-link discovery. Ten crawled successfully for 389 internal URLs, of which 386 (99.2%) returned 200. Two of them, Etsy and IMDb, blocked the crawler at the root with a 403 and a 202 before exposing a single internal link.

Limitations. The 99.2% figure is not a web-wide success rate. A clean internal link graph on a well-run site understates real crawl waste, which concentrates on small, legacy, and faceted e-commerce sites and in the non-link request firehose that only server logs and Search Console capture. Status codes reflect what the crawler received on 15 July 2026 and vary by region, bot-management rules, and personalization.