Study: AI agents favour sites they can actually fetch and read

AX is the new AEO. That is the paper's thesis, and it arrives with a sample size that makes it hard to dismiss as vendor theatre: 37,927 agent journeys across 1,056 real businesses, run through four independent harnesses, per the arXiv paper "AX is the New AEO." The businesses were matched on fame, on prior model knowledge, and on two proxies for answer-engine optimisation, then split by whether an AI agent could actually fetch and read their site. The finding: AEO gets a business surfaced. AX decides whether it gets recommended.

The mechanics nobody priced in

The AEO playbook, barely two years old, already looks like it was built for a system that no longer exists. In 2023 large language models answered from frozen training data, so the advice was to seed that data: get mentioned in enough places that the next training run absorbs you. By 2024 the advice shifted to live retrieval: scatter citations across forums, listicles, and third-party reviews so a search-augmented model is likelier to surface your name in an answer.

Both generations of advice assumed the decisive event was surfacing. The arXiv paper's contribution is to show that surfacing is now the easy part, and the cheap part. A modern agent handling a buyer question does not stop at a search-results page. It opens results, reads them, and often loops back through several more rounds of search and fetch before it answers. That drill-down step, not the initial ranking, is where a business's fate gets decided. And drill-down lives or dies on one unglamorous variable: can the agent's fetcher actually retrieve and parse the page.

This is a infrastructure question dressed up as a content question. Sites that gate content behind JavaScript rendering, aggressive bot-blocking, login walls, or malformed HTML are invisible to an agent at exactly the moment the agent is trying hardest to verify a claim. A business can rank well in the surfacing step, via strong AEO signals, prior model familiarity, or general fame, and still lose the recommendation because the agent's fetch call returned a blank page or a 403.

Where this bites hardest

The sectors most exposed are the ones with the oldest, most bureaucratically encrusted web estates: financial-services firms with compliance-driven CMS platforms, multilateral institutions running content management systems designed for human editors rather than machine fetchers, and industrial groups whose product documentation sits behind PDF walls or region-gated portals. These are precisely the organisations TCE's clients represent, and precisely the ones least likely to have audited their sites for machine readability rather than human readability.

The perverse result is that a UN agency or a global insurer can invest heavily in the current AEO orthodoxy: press mentions, forum presence, structured citations, and still lose the recommendation to a smaller competitor whose site simply loads cleanly for a bot. Fame and prior model knowledge, the paper's own control variables, do not compensate for a broken fetch. That is the uncomfortable part. AEO effort and AX failure are not substitutes; they operate at different stages of the same journey, and a business can win one and still lose the sale.