AI crawlers read a web page very differently from Googlebot, and almost all of that difference comes down to JavaScript. When part of your content only appears once the browser has run some code, Google usually ends up seeing it, while the crawlers behind ChatGPT or Claude leave with a half-empty page. I still come across plenty of sites that polish their copy to get cited by AI without ever checking whether that copy reaches the bots that collect it. Here is what we know about how those crawlers behave, a two-minute test you can run on your own site, and what to fix depending on how your site is built.
TL;DR
A study by Vercel and the agency MERJ found that GPTBot, ClaudeBot and PerplexityBot do not execute the JavaScript on the pages they visit.
Googlebot runs JavaScript in a second rendering step, which is why content can show up on Google and stay invisible to ChatGPT.
The simplest test is to search for a sentence from your page in the raw source code, the view you get with Ctrl+U.
A standard WordPress site sends its text in the HTML, but tabs, reviews and blocks loaded by scripts deserve a check.
Get an AI summary of this article
What AI crawlers actually receive when they visit a page
The most solid reference on this topic is still “The rise of the AI crawler”, published in December 2024 by the hosting platform Vercel and the agency MERJ, with Giacomo Zecchini and Malte Ubl among its authors. They analyzed real crawler traffic across Vercel’s network. Over one month, GPTBot, OpenAI’s crawler, made 569 million requests there and Anthropic’s made 370 million, against 4.5 billion for Googlebot.
The core finding fits in one sentence: none of the major AI crawlers they studied execute JavaScript. GPTBot does download JavaScript files, in 11.5% of its requests, and ClaudeBot in almost 24%, yet neither runs them. PerplexityBot behaves the same way. In practice, these AI crawlers take the HTML the server sends on the first load, and anything added afterwards by a script never reaches them.
Two exceptions came out of the study. AppleBot uses a browser-based crawler that renders JavaScript much like Googlebot does. Gemini relies on Googlebot’s infrastructure and gets the same full rendering. The AI crawlers people use most read your page without JavaScript.
An honest caveat about the date. These measurements are almost two years old, and the companies behind these crawlers may have upgraded their tools since. Hypothesis, not verified: as far as I know, the public documentation from OpenAI and Anthropic still says nothing about their collection crawlers rendering JavaScript, which is why I treat the finding as still valid until proven otherwise. Assuming your JavaScript goes unread remains the safest position either way.
Why Google sees your page while ChatGPT sees half of it
Google built a separate path for JavaScript. Its documentation on JavaScript SEO basics describes three phases: crawling, rendering and indexing. During rendering, a headless version of Chrome runs the page’s JavaScript, and Google indexes that final result. The step can take a few seconds or much longer, but it does happen.
The line I find most telling on that page is one Google slips in almost in passing. It recommends server-side rendering because it makes a site faster for users and crawlers, and because not all bots can run JavaScript. In Google’s own words, then, a site that depends on JavaScript to display its content works for Google and works a lot less well for the rest of the web.
That gap shows up directly in AI answers. ChatGPT’s index is built from what its crawlers actually read, as explained in the piece on how the ChatGPT index picks the pages it cites. A page whose text arrives through a script can rank on Google and still be missing from ChatGPT’s answers, with nothing in Search Console to warn you.
Sites are not all equally exposed. Apps built entirely in the browser, with React, Vue or Angular and no server rendering, take the biggest hit, because their starting HTML often holds little more than an empty container. Traditional sites are rarely affected as a whole, but they often lose specific pieces: a tab that loads its text on click, a customer review block injected by a widget, an FAQ generated by a script, a pricing table that appears after the fact.
One workaround keeps coming up: publishing a separate markdown version of every page, made for AI crawlers. The piece on markdown for AI crawlers, which mostly draws SEO tools, reports that on John Mueller’s test sites, the only bots claiming to accept markdown were SEO tools, and that a Profound experiment on 381 pages found no significant difference between markdown and HTML. Honestly, the better investment is making sure the normal page already carries its text in the HTML, since that is the page crawlers visit first.
The test that shows what an AI crawler reads on your site
The quickest test needs no tool at all. Open a page on your site, view its source with Ctrl+U on Windows or Option+Cmd+U on a Mac, and search for an exact sentence from your text with Ctrl+F. The source view shows the HTML exactly as the server sends it, before any script runs. If the sentence is there, AI crawlers get it. If it shows on screen but is missing from the source, it is loaded by JavaScript.
Be careful not to mix this up with the browser inspector, the right-click and “Inspect” option. The inspector displays the page after JavaScript has run, so it shows everything, including the blocks that AI crawlers miss. Plenty of people run the test there the first time and come away with a false sense of security.
I ran the exercise on SEO Noobies, requesting a recent article with GPTBot’s user agent, the way OpenAI’s crawler would. The server answered normally with a 217 KB page containing fourteen scripts, and the opening sentence of the article was right there in the raw HTML, along with about 1,800 words of text readable without running anything. For a WordPress site with a classic theme, that is the expected result, since pages are built on the server before they are sent.
Run the test on three kinds of pages: the homepage, a service or product page, and a blog post. Commercial pages are usually where the surprises hide, because they pile up modules added over the years, like sliders, price comparison tools or review blocks. A page that passes on the blog post and fails on the product page tells you exactly where to focus.
If you are comfortable with a terminal, there is a slightly more faithful version of the same test. A single curl command can request the page while announcing itself with GPTBot’s user agent, then search the response for your sentence. It adds one thing the source view cannot show: whether your server, your security plugin or your CDN treats that crawler differently from a normal visitor. A site that returns an error or a challenge page to GPTBot has a different problem from JavaScript, but the outcome for AI visibility is exactly the same.
Discover the SEO Noobies agency
What I would fix depending on how your site is built
On a standard WordPress site, the foundation is usually sound, and the job is to track down the modules that load their text through scripts. A review widget can often be swapped for a version that writes reviews straight into the page. Content tucked behind a tab or an accordion stays readable as long as it is present in the HTML on load, which the source code test confirms in seconds.
On a site built with a JavaScript framework, the fix is server-side rendering or prerendering, which generates complete HTML pages ahead of time. That is exactly what the Vercel and MERJ study recommends for any important content. This work needs a conversation with whoever built the site, and it is far cheaper to scope before a redesign than after one.
One last check belongs in the same pass: readable text is useless if the crawler is not allowed in. Your robots.txt has to let through the bots you want citing your site, and each one has its own habits, as the article on ChatGPT-User reading pages your robots.txt blocks showed. HTML readability and AI crawler access are best checked together, in a single sweep.
Getting an outside check on what AI tools read
This check is part of the SEO Noobies agency’s “I don’t have the time!” package at €799: alongside the technical audit, AI crawler access to the site is reviewed and fixed wherever that is possible without rebuilding the site. For a JavaScript framework site that needs real server-rendering work, the diagnosis becomes the starting point for a quote.
Keep learning on SEO Noobies.
