Ahrefs: 97% of llms.txt files get zero LLM traffic
Ahrefs' 137,000-domain study finds llms.txt files are almost universally ignored by LLM crawlers, making the protocol a poor proxy for AI citation strategy.
Key takeaways
- 97% of llms.txt files across 137,000 domains receive zero LLM crawler traffic, per Ahrefs.
- LLM crawlers support the spec in principle but do not consult llms.txt as a standard step in their retrieval pipeline.
- llms.txt cannot substitute for the source authority and third-party citation patterns that actually drive LLM visibility.
- The file retains narrow value for platforms that explicitly support it, but this does not generalise to AI search at scale.
- Teams should redirect effort toward durable citation signals: topical depth, external references, and structured authority.
The llms.txt convention arrived with considerable enthusiasm. A clean, readable file in the root directory, the thinking went, would help AI crawlers understand what a site contains and cite it accordingly. Ahrefs has now tested that theory against 137,000 domains, and the result is not encouraging.
Per Ahrefs, 97% of llms.txt files across those sites receive zero LLM traffic. Not marginal traffic. None. The file sits in the root directory, well-formed and ignored, while the crawlers do something else entirely.
The mechanism worth understanding
The failure is not technical. LLM crawlers can read llms.txt; several explicitly support the spec. The problem is that supporting a format and actively consulting it before every retrieval decision are different behaviours. Most crawlers index content through their standard pipeline first and reach for supplementary guidance files only occasionally, if at all. An llms.txt file that sees no bot traffic is not broken. It is simply irrelevant to how those bots actually operate.
This matters because the llms.txt proposal was never just a developer curiosity. A non-trivial number of content and SEO teams at large organisations treated it as an actionable step toward better AI visibility: a way to signal authoritative content to models, exclude low-quality pages from retrieval, or structure a site's knowledge for LLM consumption. The Ahrefs data suggests that wager has not paid off.
The comparison to robots.txt is instructive but should not be pushed too far. Robots.txt works because search engine crawlers were designed, from the beginning, to check it before indexing. That convention is baked into crawler architecture and enforced by platform norms. LLM retrieval pipelines were built with different assumptions, and most of them do not yet have an equivalent hardwired respect for llms.txt. The file is a convention without the infrastructure that makes conventions binding.
What this means for brands betting on structured signals
For communications and digital teams at multilateral institutions, industrial groups, and financial services firms, the practical implication is specific. Many of these organisations have invested in structured content signals over the past eighteen months precisely because AI-generated answers increasingly drive how their policy positions, products, and institutional expertise are surfaced to decision-makers. If llms.txt was part of that strategy, it should be deprioritised.
The organisations most exposed are those that implemented llms.txt as a substitute for more durable citation-building work: producing content that is already being referenced by outlets and sources that LLMs demonstrably cite, building topical authority legible to model training pipelines, and earning mentions in the structured knowledge sources models rely on most heavily. A file that 97% of crawlers never open cannot compensate for the absence of those fundamentals.
There is a narrower case where llms.txt retains some value: platforms that have explicitly confirmed they consult it, such as certain API-access tools or developer-facing retrieval systems. For organisations whose content is primarily consumed through those channels, maintaining a well-structured file is low-cost and not irrational. But that is a different claim from the broader one, that llms.txt meaningfully improves AI visibility at scale.