Saying no to AI training without dropping out of Google used to be an impossible choice for sites running behind Cloudflare. The same crawlers read the web for search and for feeding models, so blocking one use meant blocking the other. On September 15, 2026, Cloudflare shipped a new setting that finally splits those two uses at Google and Apple, and it automatically replaces the older blocking options. If you ticked a protection box over the past few months, this is mostly good news. It still pays to understand what the setting actually does, what it leaves untouched, and what remains worth checking on your own domain.
TL;DR
Cloudflare launched Disallow AI Training, a setting that writes a no-training preference into your robots.txt while Googlebot, Applebot and Bingbot keep crawling for search.
Sites that had picked Block or Block on pages with ads are moved to the new setting automatically.
At Google, the opt-out goes through the Google-Extended token, which Google says has no effect on inclusion or ranking in Search.
Microsoft does not honor a site-level no-training preference yet, with support targeted for early 2027.
Get an AI summary of this article
Opting Out of AI Training Without Closing the Door on Search
Cloudflare’s announcement is titled Have it both ways: stay discoverable in search while disallowing AI training, and the title sums up the promise fairly well. The setting is called Disallow AI Training. Once it is on, it publishes a no-training preference in your robots.txt for the crawlers involved, and it lets the search crawlers from Google, Apple and Microsoft through.
To see why this matters, go back to what happened over the summer. In early August we covered how Cloudflare was serving a 403 error to Googlebot when AI blocking was switched on. The cause was the way mixed-use crawlers work. Googlebot, Applebot and Bingbot read your pages to rank them in search, and part of what they collect can also feed generative systems. Refusing training meant refusing indexing along with it.
Cloudflare admits as much in its post. The Block and Block on pages with ads options did not apply to mixed-use crawlers, because blocking them could hurt discoverability in search. Put plainly, the protection people thought they had either skipped the crawlers that matter most to a small site, or covered them at the cost of that site’s traffic. For a young site still looking for its audience, that second outcome is the worse of the two, because crawling problems take weeks to show up in rankings and even longer to repair once spotted.
The new setting treats the two families differently. Crawlers that exist only for training, and Cloudflare names the ones from Amazon, Anthropic, Meta and OpenAI, can be blocked outright. Mixed-use crawlers stay allowed to crawl, and they receive a no-training rule in robots.txt that their operators have committed to respect. A written preference now replaces the network block.
One detail matters for existing customers. Cloudflare says previous Training selections of Block or Block on pages with ads will migrate to Disallow AI Training. If you had picked either one, there is nothing to redo to get out of the situation described in August. The setting is available to every customer on every plan, in the security settings of each domain.
Discover the SEO Noobies agency
Google-Extended and Applebot-Extended, the Two Tokens Doing the Sorting
Under the hood, everything relies on a mechanism Google and Apple already offered. Each of them created a special token to target in robots.txt, separate from its search crawler. Google’s is called Google-Extended. Apple’s is Applebot-Extended. Cloudflare’s setting simply writes a Disallow rule for those tokens and keeps it up to date for you, which spares you from editing the file by hand every time an operator changes its crawler names.
The question site owners worry about most gets its answer on Google’s side. The official documentation for Google’s crawlers explains that Google-Extended lets publishers decide whether crawled content may be used to train future generations of Gemini models. It adds that the token does not affect a site’s inclusion in Google Search and is not used as a ranking signal. Refusing Gemini training costs you no rankings at all.
That said, it is worth reading what the page leaves out. It talks about Gemini training and the apps built on it, and never mentions AI Overviews or AI Mode in Search. Nothing there supports the idea that this setting removes you from the generated summaries at the top of Google, and nothing supports the opposite either. Treat those as two separate topics until Google publishes something more specific.
Microsoft is a different story. According to Cloudflare, Microsoft is currently building the mechanism to respect a site-level no-training preference in robots.txt, targeted for early 2027. Until then the setting has no equivalent effect on training uses tied to Bingbot, even though the crawler keeps reading your pages for search.
These tokens also have nothing to do with the assistant crawlers that fetch a page live while someone is asking a question. Those follow their own names in robots.txt, and mixing them up with training crawlers remains the most common mistake. We walked through that trap when looking at the ChatGPT crawler you should not block by accident in robots.txt.
Cloudflare also introduces a label worth knowing for future announcements. It calls mixed-use crawler operators accountable when they meet four conditions: a robots.txt opt-out, an opt-out for AI summaries, visibility into how pages are used for training, and an assurance that opting out of training will not affect search rankings. Apple, Google and Microsoft are described as meeting those conditions or having committed to meet them within a set time frame.
What to Check If Your Site Already Runs on Cloudflare
Start by opening the security settings for your domain in the Cloudflare dashboard and look at the line covering AI crawler training. If it reads Disallow AI Training, the migration went through. If it still shows something else while you believed training was blocked, fix it now instead of waiting for a crawling problem to reveal it.
Next, open your robots.txt in a browser by typing your domain followed by /robots.txt. The rules Cloudflare writes should sit next to your own. Make sure no line targets plain Googlebot, because a Disallow on the search crawler has a very real effect on your presence in Google, with nothing to do with training.
The third check is the one that matters most if your traffic dipped this summer. In Search Console, see whether crawling has returned to normal since mid-September and whether some pages are still flagged as blocked or in error. The full walkthrough is in our guide on how to check indexing in Google Search Console. A corrected setting won’t reindex any page by itself.
One last point for domains joining the platform from now on, because the defaults are changing too. On a new domain that shows ads, the AI training category starts on Disallow AI Training, while a site without ads stays on Allow. The agent category, meaning crawlers that act live on someone’s behalf, starts on Block on pages with ads for ad-supported sites. If you are launching a site in the coming weeks, open both lines before going live, so the setting reflects what you actually want rather than a value picked for you.
It also helps to keep this in proportion. Cloudflare says 17% of sites on its network choose some mechanism to block AI training, and less than 1% block search bots. For most small sites, the new setting will change nothing day to day. It mainly concerns people who made a defensive choice without measuring what it did to their visibility.
The underlying question is one every site has to settle on its own. Refusing model training protects your content from being used with nothing in return, and this setting now allows that without giving up search at Google and Apple. Still, visibility inside generated answers depends on other crawlers and other rules. If your goal is to get cited by assistants, go through your directives crawler by crawler before shutting everything down on principle.
Keep learning on SEO Noobies.
