An AI agent is an assistant that browses the web on your behalf: it opens pages, compares products, fills a cart and sometimes goes all the way to checkout. On September 20, Amazon cut off access to its store for Muse, the AI agent Meta had launched a few days earlier. The most useful detail for anyone running a website is the tool Amazon reached for: its Conditions of Use, because its robots.txt file had no grip on that AI agent at all. Analyst Slobodan Manic took the episode apart, and it says a lot about what this small file can still do against assistants that browse for their users. Here is what happened, why robots.txt stayed silent, and what you can check on your own site.
TL;DR
Since September 20, Amazon has been blocking Muse, Meta’s AI agent, citing the Conditions of Use its customers agreed to.
Amazon says the agent browses without identifying itself and appears to store customers’ login credentials.
robots.txt works with declared bot names, and Meta’s documentation lists no name at all for Muse.
Open your robots.txt, list the AI bots you block, and check that each block is a decision you actually made.
Get an AI summary of this article
Amazon Shuts Its Store to Meta’s AI Agent
Muse is the AI agent Meta introduced in early September to help people shop or book appointments online. On the evening of Sunday, September 20, users who sent it to amazon.com landed on a warning page: “Continued access by an unauthorized AI agent violates Amazon’s Conditions of Use, to which our customers have agreed.”
Amazon gave three reasons, laid out in Slobodan Manic’s analysis of the block: Meta never told Amazon that Muse would come to its store, the agent does not identify itself while browsing, and it appears to capture and keep customer credentials. The company added that an app buying on a customer’s behalf should operate openly and respect a retailer’s decision about whether to take part. Meta had not responded publicly when that analysis came out.
The fight goes beyond Muse. Amazon had already taken Perplexity to court over its shopping agent, and the underlying question stays the same: can a website decide which AI agent gets to act on it? For an e-commerce giant, the answer runs through lawyers. For a small site, it starts with robots.txt, which is exactly where the story gets interesting.
Discover the SEO Noobies agency
Why robots.txt Had No Grip on This AI Agent
A quick refresher on how the file works. A robots.txt rule starts by naming a bot through its user agent, the label a piece of software sends every time it asks a site for a page. The rule then says what that bot may fetch. Without a declared name, no rule can target an AI agent or any other piece of software.
And Meta’s official crawler documentation lists five bots, none of them called Muse. There is Meta-ExternalAgent for model training, Meta-WebIndexer for Meta AI search, Meta-ExternalAds for advertising, FacebookExternalHit for link previews, and Meta-ExternalFetcher, which fetches pages at a user’s request. Slobodan Manic points out that Amazon already blocks three of them in its robots.txt, and it made no difference with Muse.
One line of that documentation deserves a second read. Meta states that Meta-ExternalFetcher may bypass robots.txt rules because it performs fetches requested by the user. Other companies apply the same logic: SEO Noobies already showed that ChatGPT’s user bot reads pages your robots.txt blocks when someone asks it to. An AI agent acting for a person sits in a category of its own, one that robots.txt covers poorly, and every new AI agent from a major player raises the same question for site owners.
Google says it plainly in its introduction to robots.txt: the instructions in the file cannot force a crawler to follow a site’s rules. Reputable crawlers obey them, others may ignore them. robots.txt works like a signpost, and the Muse episode shows what happens when the visitor does not even have a name to read on it.
What to Check in Your Own robots.txt
For a small business site or a blog, blocking an AI agent is rarely the priority. The opposite risk is often bigger: a robots.txt copied from a template found online, blocking AI bots nobody decided to block, and quietly removing the site from ChatGPT or Meta AI answers. Start by opening your file, typing your domain followed by /robots.txt in the browser, and listing every bot name it mentions.
For each AI bot you block, ask a simple question: does this block help the business? Blocking a training bot like Meta-ExternalAgent is a defensible editorial choice. Blocking a bot that feeds a search product, like Meta-WebIndexer, means vanishing from an engine where your customers may already be asking about your trade. The right setting depends on what you do, and a shopping AI agent cannot be managed from this file anyway.
If your concern is genuinely sensitive pages, such as a customer area, quotes or partner pricing, Google points to other methods like password protection. Private content belongs behind a login. To take a page out of Google’s results, the noindex tag does the job, as covered in the piece on why blocking a page in robots.txt is not noindex. robots.txt then keeps its original role, which is guiding the bots willing to read it.
There is a GEO angle worth keeping in mind as well. Being cited in an AI answer starts with a bot that is allowed to read your pages, so every search-oriented bot you block is one engine where your business can no longer show up. Reviewing the file once a quarter, the same way you would check your Google Business Profile, keeps those choices deliberate instead of inherited from a template nobody remembers installing.
The Muse story plays out between two giants, yet it leaves a useful lesson for every site: robots.txt speaks to bots that announce their name, and an anonymous AI agent walks right past its instructions. Everything else falls to terms of use, logins and server settings.
Keep learning on SEO Noobies.
