llms.txt: 97% never read (Ahrefs 137k-site study)

The debate about whether llms.txt is worth setting up has been running in the SEO community for close to a year now. Some see it as the future robots.txt for generative engines, others as a gadget that won’t matter until ChatGPT, Perplexity or Claude officially announce they read it. Ahrefs has just settled a big chunk of the debate with hard numbers, in a study covering 137,000 sites that ship an llms.txt at their root. The headline is blunt: 97% of those files have never been read by an identifiable AI crawler. After ten years of doing SEO day in and day out, I have watched plenty of well-intentioned false starts of this kind, and I want to share a grounded read of the file that lets you decide with your eyes open.

TL;DR

Contents

What the Ahrefs study on 137,000 files actually measures

Ahrefs tapped into its massive crawl index to identify sites that expose a llms.txt file at the root of their domain, on the model of robots.txt. The sample reaches 137,000 sites, which is already a signal in itself: the format has spread faster than the silence of generative engines on the subject would have suggested. The methodology then rests on cross-referencing that list against the identified user-agents of the main known AI crawlers: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Google. Ahrefs also looked at secondary user-agents like OAI-SearchBot, Applebot-Extended and cohere-ai, alongside the main four. The perimeter of the analyzed logs covers several months of continuous crawl, which limits the risk of a rare isolated visit slipping through the measurement.

The core finding is brutal. Only 3% of the llms.txt files in the panel received at least one visit from one of those identified crawlers. Put another way, 97% of the site owners who took the time to configure that file have, to date, no measurable return. Ahrefs published the full study with methodology and detailed charts on their research blog. These numbers need the usual caution: a crawler can read a file without leaving an exploitable trace in the logs, and the absence of a visit is not proof of a total absence of use. But the order of magnitude is far too pronounced to be an artefact.

Second key point from the study: none of the major generative engines has publicly confirmed that it uses the llms.txt protocol to guide its crawl or its content selection. OpenAI, Anthropic, Perplexity and Google remain silent on the topic, or refer webmasters to broader recommendations about structured content. The protocol exists on the publishing side, but has no official receiver on the ingestion side. That asymmetry is what explains the ridiculously low read rate flagged by Ahrefs. A standard that has no declared receiver on the engines it claims to serve cannot produce a measurable gain in logs, even if the emitters multiply. This configuration is the classic pattern of a community norm waiting for an official adoption that may very well never come, or arrive in a modified version that would make today’s files partially obsolete.

Why the llms.txt protocol spread so fast despite the silence

The llms.txt format was proposed in late 2024 by Jeremy Howard, the fast.ai founder, in a simple note that quickly circulated across the AI and SEO communities. The idea was to build the equivalent of a robots.txt for LLMs: a plain text file at the root that lists the relevant URLs of the site, with a title and a short description for each. The stated goal was to give generative engines fast access to the most useful content, without having to crawl the entire site to understand its structure.

The viral success of the proposal rests on two very classic drivers in the SEO world. First, the anxiety of missing the GEO train: nobody wants to be the last person to have configured the new signal that supposedly changes everything. Second, the near-zero cost of implementation: a plain text file of a few dozen lines at the root can be shipped in thirty minutes, giving the site owner the satisfaction of “ticking the AI box”. In ten years of doing SEO daily, I have watched the exact same mechanic unfold with Google+ authorship in 2013, news_keywords tags in 2016, or more recently the “AI-friendly” markdown long-form pages meant to flatter ChatGPT. Each time, the propagation far outstrips the actual validation of the protocol by those meant to exploit it.

What makes llms.txt still interesting despite that quiet start is that it remains an open, lightweight and non-intrusive standard. Unlike a full markdown copy of the site (which John Mueller recently called a stupid idea, as we covered in our piece on the Google spokesperson’s stance), llms.txt does not duplicate content. It does not break anything. It carries no cloaking risk. It is a site plan written for humans and machines, hardly more than that.

My GEO read on the file after ten years in SEO

On the sites I run day to day, I tested shipping an llms.txt on three of them during the first quarter of 2026. What I mainly wanted to measure was an effect on citation rate inside Perplexity and on referral traffic coming from ChatGPT. Six months later, my logs tell the same story as the Ahrefs study: no observable correlation between the presence of the file and GEO performance. The sites that rise in AI citations are the ones that worked on fundamentals: clean HTML structure, stable H2/H3 hierarchy, primary sources cited properly, thematic consistency across the domain. llms.txt adds nothing to that foundation. It certainly does not replace it. I also compared behavior of the three equipped sites with two other sites of the same editorial profile and same size, kept without llms.txt: Perplexity citation growth was strictly of the same magnitude across all five domains, with a variance driven far more by the editorial quality of the articles published in the period than by the presence or absence of the root file.

That said, I do not recommend removing the file if you have already put it in place. Ongoing operating cost is nil, readability for any human inspecting the root of the site is good, and the day a generative engine officially announces that it exploits the signal, you will already be positioned. It is a very classic asymmetric bet in SEO: when the cost is low and the potential upside is non-zero, the setup holds even if the immediate expected value is close to zero. My consulting clients ask me the question every single week, and I give them the same answer: if you have thirty minutes to spare, go ahead. If you don’t have thirty minutes, spend them instead on the structure of your pillar articles, they pay off a hundred times more.

Where things get more subtle is on the question of the reputational signal. A site that exposes a well-built llms.txt also sends a message to the humans who inspect it: “We take AI readability seriously”. For a project in an authority-building phase, this can move the needle on the perception of a journalist, a potential partner or an SEO auditor looking under the hood. The file then matters less for what it does technically than for the methodological seriousness it signals.

What I actually recommend to a beginner in 2026

If you are starting a site today, do not begin with llms.txt. Begin with the SEO and GEO fundamentals that genuinely move the needle: clean HTML, articles built around a clear search intent, primary sources cited every time a number is quoted, a title hierarchy respected from top to bottom. Those bricks are ingested and rewarded by generative engines since their very first training runs. They work equally well for classic Google Search and for Perplexity or ChatGPT.

Once those fundamentals are in place, you can add the llms.txt in thirty minutes without spending more energy on it. The minimal format is limited to a site title, a short description of the activity, and a list of main URLs with a title plus one context sentence for each. You can target twenty to fifty URLs for a starter site, prioritizing your pillar articles over your dated news. Place the file at the root, serve it as text/plain, and declare its existence in your XML sitemap if you want to signal the file’s presence to crawlers that already read the sitemap. Remember to update the file every time your pillar catalogue evolves, otherwise it quickly loses any descriptive use. A quarterly reminder in your SEO maintenance calendar is more than enough, there is no daily urgency for this resource. The extended, richer llms.txt format also proposes to list the structured sections of your site with an optional llms-full.txt file that carries the full content in markdown, but I advise against that variant in 2026, it runs into the same content duplication issues already flagged by Google.

One last point that matters for AI readability in the broader sense: be rigorous on the authority signals you expose publicly. A clear about page, a named and documented author, up-to-date legal notices, external links to recognized sources: these elements weigh heavily in a generative engine’s decision to cite you rather than an anonymous site.

To close on llms.txt itself: the protocol is useful to watch, cheap to set up, and interesting to document in your own GEO monitoring journal. But it does not replace any of the workstreams that genuinely build your authority in the eyes of generative engines. Treat it as a discreet bonus, not as the key that unlocks the doors of Perplexity.

Keep learning on SEO Noobies.