ChatGPT-User is the bot that goes and fetches a page the moment someone asks the assistant a question, and OpenAI’s own documentation says robots.txt may not stop it. That line sits in the official crawler page, in plain language, with no real room for interpretation. The reasoning behind it fits in a sentence: a real person triggered the visit, so it is not treated as an automated crawl. Plenty of site owners have pasted blocking rules into their file believing they had shut the door on generative AI for good. Here is what the documentation actually says, what it changes for a small site, and which settings still carry weight.
TL;DR
OpenAI documents four separate bots, and ChatGPT-User is the one that fetches a page on demand rather than crawling the web on a schedule.
The documentation states that because those visits are user-initiated, robots.txt rules may not apply to them.
Blocking that bot does not remove your site from ChatGPT search, and allowing it does not earn you a citation.
The setting that decides whether you show up in answers is the one covering OAI-SearchBot.
Summarize This Article with AI
ChatGPTWhat OpenAI Documents About ChatGPT-User
Four bots carry the OpenAI name, and mixing them up is the most common mistake once you start digging. GPTBot gathers material used to train the models. OAI-SearchBot feeds the search index that decides whether your pages can surface in answers. OAI-AdsBot handles the advertising side. ChatGPT-User does something narrower than all three: it retrieves one specific page at the moment a question needs it.
One distinction deserves a moment before going further, because it trips up almost everyone. A robots.txt file governs access to a page, never its appearance in a search product. That is precisely the mix-up unpacked in our piece on why blocking a page in robots.txt is not the same thing as a noindex tag. The same logic applies here, with an extra layer of AI-specific nuance stacked on top.
The sentence causing all the noise is short. It says that because these actions are initiated by a user, robots.txt rules may not apply. It goes on to note that this bot plays no part in deciding whether content shows up in search, and points site owners toward OAI-SearchBot for anything to do with opting out of results. The file still gets read, it simply does not settle this particular visit, which is a long way from a guarantee.
That wording lives in the official page describing how each OpenAI bot behaves toward robots.txt. Reading it end to end is worth the four minutes, if only to notice that every bot gets a different recommendation and two of them pull in opposite directions.
Why A Visit Someone Asked For Sits Outside The Rule
The reasoning here is not arbitrary, and it helps to follow it. The robots exclusion protocol was built to handle programs that sweep the web continuously, with no particular purpose in mind, burning server capacity as they go. Someone pasting a link into ChatGPT and asking for a summary occupies different ground. They are behaving like a person opening a browser tab, except the rendering happens through an intermediary.
This reading is far from unique to OpenAI. Several players in the space draw the same line between a systematic crawler and an agent answering a one-off request. The argument is less about whether the principle holds than about how far it stretches, since nothing technically prevents a very large volume of fetches from being labelled user-initiated.
On the hosting side, this stopped being theoretical a while ago. We covered the case where an AI blocking rule set through Cloudflare ended up getting in Googlebot’s way too, with genuine consequences for ordinary indexing. Filtering bots reaches well past a text file sitting at the root of a domain.
Worth saying plainly: robots.txt was never a firewall. It is a convention that serious operators honour because honouring it serves them, and that everyone else ignores without penalty. What changed is that a major player now documents, in public, a category of visits where it reserves the right not to follow the rule.
A practical consequence follows, and it catches people out. ChatGPT-User leaves a very different fingerprint in your server logs than a conventional crawler does. Where GPTBot sweeps through in regular waves across dozens of URLs, a user-triggered fetch touches one page, once, at the moment somebody asks their question. Spotting it in your stats means hunting for an isolated spike rather than a pattern, which goes a long way toward explaining why so many site owners have never noticed it at all.
What A Small Site Should Check Today
Start by opening your robots.txt and reading which names are actually in there. Sites that copied a list off a forum routinely block GPTBot and ChatGPT-User while leaving OAI-SearchBot wide open, or do the exact reverse without realising it. Those two configurations produce opposite outcomes, one touching training and on-demand reading, the other your presence in answers.
Then comes the real question: what are you trying to achieve? A site that lives on its audience and wants to be cited by assistants gains from letting the search bot through, even if that means accepting that pages get fetched on demand. A site selling paid content reasons differently and protects itself at the server level rather than through an advisory file. A deliberate choice beats a copied blocklist whose effects nobody has measured.
For anyone chasing citations, the useful work sits elsewhere entirely. The criteria that genuinely drive page selection are laid out in our breakdown of how the ChatGPT index picks the pages it cites, and they look a great deal like solid conventional SEO: readability, freshness, a direct answer to a question someone actually asked.
One option gets overlooked more often than it should, and it is frequently the right one: block nothing at all. A content site chasing visibility gains nothing by shutting the door on ChatGPT-User, since every page fetched is a page that might be summarised with a link pointing back to you. Blocking earns its keep when you sell access to that content, or when server load turns into a genuine bill. Outside those two situations, a copied Disallow line costs you a channel and hands you nothing in return.
A final word of caution about how far to take all this. The documentation says the rules may not apply, not that they are always ignored. That gap matters, and it rules out concluding that the file has become pointless. It still carries the instructions aimed at every other bot, and those account for the overwhelming share of automated traffic hitting a small site.
None of this calls for an emergency rebuild. Checking what your file contains, learning which bot each line refers to, and deciding on purpose what you let in covers today comfortably. The rest depends on how these rules evolve, and they are evolving quickly.
Keep learning on SEO Noobies.
