Free Website Crawl Test: See Your Page Without JavaScript

Most crawl checkers answer one question: is this URL allowed to be indexed? That is a permission question, and it is answered by two small pieces of text — a rule in your robots.txt and a meta robots tag in your HTML. It is worth checking, it takes milliseconds, and it is almost never the reason a page is missing from Google.

The second question is harder and far more often the culprit: what does Googlebot actually receive when it requests your URL?

Those are different failures with different fixes. A page can be perfectly permitted and still be indexed as a near-empty document, because the content a visitor sees was assembled in their browser by JavaScript that Googlebot's first request never ran. This test runs both checks side by side:

• Permission — your robots.txt is parsed with Google's longest-match rule, and the page's meta robots value is read, producing separate crawlable/indexable verdicts for Google and Bing.
• Reality — the URL is fetched twice. Once as raw HTML with Googlebot's smartphone user agent, and once through headless Chrome with JavaScript enabled. The same signals are counted in both copies, and the difference is reported.

Everything below is about the second half, because that is the half people rarely have a way to see.

Google does not crawl and render in one step. It runs two passes, separated by an unknown amount of time.

Pass one — crawl and index the HTML. Googlebot requests your URL and receives whatever your server sends. It parses that response, extracts the text, headings, links, meta tags and structured data it finds in that response, and indexes them. It also queues every link it found for crawling. This pass is fast and it is the one that always happens.

Pass two — render. The page is handed to a separate service that loads it in a headless browser, executes JavaScript, and produces a rendered DOM. Google then re-indexes the page from that DOM. This pass is deferred: it sits in a queue, it has its own timeouts and resource budget, and it is prioritised like any other finite resource.

For most pages on most sites the second pass arrives. But three things follow from the architecture, and they are what make raw HTML matter:

1. There is a gap. Between the two passes — which can be minutes, or considerably longer — the only version of your page in Google's index is the raw HTML one.
2. The queue is not guaranteed. Pages that time out, error during rendering, or sit low in crawl priority can be indexed on their first pass alone. There is no notification when this happens.
3. Links found only in the rendered DOM are discovered later. If a category page's links to its products exist only after hydration, those product URLs are not queued during pass one. On a large site, that delay compounds.

This is why "Google renders JavaScript" is true and still not a reason to relax. Rendering is a second chance, not the first one.

The comparison table has one row per signal and one column per pass. The left column is what pass one receives; the right is what exists after JavaScript. Here is what each row is telling you.

Title, meta description, canonical. These belong in the server HTML without exception. A title set by JavaScript may be missed on the first pass, and a canonical that only exists post-render is a well-known cause of the wrong URL being indexed. If any of these three are present on the right and missing on the left, fix that before anything else — they are usually a one-line change.

H1 and headings. A missing H1 in the raw HTML alongside a present one after JavaScript is the classic single-page-app signature. It means your document arrives with no stated topic.

Words of text. The headline percentage on this page is computed from this row: how much of the rendered page's text is absent from the raw HTML. Small gaps are normal and not worth acting on — a cookie banner, a lazily-loaded testimonial, a chat widget. Large gaps mean the page's substance is not in the response Google indexes first.

Internal links. Treat this as the most important row after the meta tags. Text that arrives late is indexed late. Links that arrive late may mean the pages behind them are never discovered at all. Zero internal links in the raw HTML on a page that has dozens after rendering is a site-architecture problem wearing a rendering costume.

JSON-LD blocks. Structured data injected by JavaScript is read on the render pass, if the render pass arrives. Rich-result eligibility that depends on it is eligibility you are gambling on.

Below the table, the text preview prints the actual visible text extracted from your server's response. This is the least abstract output on the page: it is, literally, the version of your page Google indexes first. If it is empty, that is the finding.

Across the 100,000+ sites our audit engine has crawled in 2026, content that is present in the rendered DOM but absent from the raw HTML is the eighth most common failure we log — it appears on 31.4% of audited sites (H1 2026). It sits in the same tier as missing alt text and duplicate title tags: not exotic, not confined to experimental stacks, and frequently present on sites whose teams are confident they render on the server.

Five patterns account for most of what we see.

The homepage is fine and nothing else is

Teams pre-render the homepage because it is the page they demo, the page they test, and the page every tool defaults to. Paginated listings, filtered categories, search results and account-adjacent pages fall back to client rendering — and those routes are where the majority of a site's indexable URLs live. Always test a deep page.

Navigation is JavaScript, not anchors

Menus and pagination built from click handlers and router.push() produce no href for Googlebot to follow in pass one. The pages are reachable for humans and invisible to the first crawl. The fix is unglamorous: real <a href> elements, with the JavaScript enhancing them rather than replacing them.

A framework upgrade silently flipped a route

Rendering behaviour is a property of configuration, not of your framework. A major-version upgrade, a component moved from server to client, or a new edge-caching rule can move a route from server-rendered to client-only without any visible change in a browser. Nobody notices, because the page still looks correct to everyone who looks at it.

The content is there, but it took too long

A page that renders correctly after twelve seconds of API calls is a page whose render can time out. Google's renderer, like ours, gives up eventually. Slow rendering is a rendering failure with a delay attached.

The server treats crawlers differently

Some servers refuse requests carrying Googlebot's user agent — usually as bot filtering nobody remembered was enabled. If this test reports that the Googlebot user agent was refused, that is worth immediate investigation: the real Googlebot verifies itself by reverse DNS, but a blanket block on the user-agent string catches it too.

In order of impact:

1. Move the critical elements into the server response. Title, meta description, canonical, H1, and the page's primary body text. Every major framework supports server-side rendering, static generation or prerendering for exactly this. You do not need to convert the whole application — you need these elements in the first response.
2. Make internal links real anchors. <a href="/products/widget"> in the HTML, whatever the click behaviour does on top of it.
3. Keep structured data server-side. JSON-LD is text in a script tag; there is rarely a reason for it to be assembled in the browser.
4. Put noindex decisions in the server response, and mean them. A noindex in the raw HTML is obeyed on the first pass. Removing it with JavaScript afterwards does not undo it — the decision was made before the render queue was reached.
5. Re-test the specific URLs you changed. Not the homepage. The routes you touched.

Fixing the raw HTML is more durable than trying to make the render pass faster, because it removes the dependency rather than racing it.

Three honest limits are worth stating.

Our renderer is not Google's. This test uses headless Chrome and waits for the network to go quiet before taking its snapshot. Google's Web Rendering Service has its own timeouts, its own resource budget, and a queue nobody outside Google can observe. A large gap here is strong evidence that a page is at risk; it is not proof that a specific URL is unindexed.

It cannot tell you what Google actually stored. Only Search Console's URL Inspection tool can do that, and only for properties you have verified. If this test shows a large gap and Search Console reports the page as indexed with full content, the render pass reached it — you were lucky rather than safe, but the page is fine today.

It tests one URL. A single result generalises to templates, not to a whole site. If a product page fails, assume every product page fails; but confirm it with a full site audit rather than assuming.

For the neighbouring checks, the robots.txt tester explains a blocked verdict in detail, the sitemap finder shows what you are actually submitting for crawling, and the mobile-friendly test covers the other half of mobile-first indexing. If Search Console is already showing the symptom, Crawled — currently not indexed is the report that most often traces back to what this test measures.