AI crawlers from ChatGPT and Claude do not run JavaScript. What they get from your pages, a two-minute test, and what to fix depending on your site.
Tag
Showing: 1 - 10 of 10 RESULTS AI crawlers
AI Training Opt-Out on Cloudflare Keeps You on Google
Cloudflare now separates AI training from search at Google and Apple. What the Disallow AI Training setting does and what is still worth checking.
Markdown for AI Crawlers Only Draws SEO Tools
John Mueller only sees SEO tools asking for markdown on his test sites, and the one controlled experiment published finds no significant gap.
Microsoft Clarity Ties AI Scraping to Your Visits
Microsoft Clarity now ties AI bot scraping volume to the referral traffic it returns. The published example shows six thousand hits for a single visit.
ChatGPT-User Reads Pages Your Robots.txt Blocks
ChatGPT-User fetches pages on demand and partly sidesteps robots.txt, and OpenAI’s own documentation states it in plain language.
Your AI Citations Show Up Without a Paid Tool
A free method to track your AI citations: manual testing in ChatGPT and Perplexity, crawler names in your server logs, and assistant referral traffic.
How the ChatGPT Index Picks the Pages It Cites
The ChatGPT index is not fed by Bing. How it picks the pages it cites, what the shared read cache does, and what actually matters for a young site.
Cloudflare Blocks Googlebot When You Block AI
Cloudflare returns a 403 to Googlebot when AI bot blocking is on. What changes on September 15 and which settings to check right now.
Robots.txt: The Cloudflare Directive Google Ignores
Robots.txt now carries a Cloudflare Content Signals directive Google fully ignores. What this file actually does, and what it never did.
llms.txt: 97% never read (Ahrefs 137k-site study)
Ahrefs analyzed 137,000 sites shipping an llms.txt file: 97% have never been read by an AI crawler. Read from a 10-year SEO practitioner.