1 in 5 'successful' AI fetches return no real content

A static HTTP client fetched 10,307 URLs over three days and logged a "200 OK" on most of them. Per the arXiv paper "When 200 OK Is Not an Answer," 20.9% of those supposedly successful fetches contained no usable main content at all: app shells waiting on JavaScript, bot-detection walls, block pages, error pages dressed up as normal ones, or partial fragments masquerading as full articles. The server said yes. The page said nothing. One in five times, nobody checked the difference.

This matters because "200 OK" is the load-bearing assumption behind almost every automated system that claims to verify what's on the web, including the crawlers and retrieval pipelines that feed answer engines like Perplexity, ChatGPT's browsing mode, and the agentic tools now fetching pages at question time rather than relying solely on a pre-built index. The paper's authors built their test set deliberately: 1,507 sites across five strata, a control group of random non-existent paths, and 4,795 URLs drawn from citations in ExpertQA, a benchmark of expert-verified claims. If a fifth of "live" citation sources turn out to be blank shells or bot-challenge pages, then a meaningful share of what AI systems believe they have read, they have not actually read.

The gap between "fetched" and "understood"

The taxonomy the researchers built is the useful part. A 200 response can mean: the real content, an app shell with no content rendered yet, a bot-detection challenge, a block page, an error page served with the wrong status code, or a partial or truncated version of the page. Static clients, the lightweight scrapers most retrieval systems use because rendering every page in a headless browser is expensive, cannot tell these apart. They see a status code and move on. The paper's human-reviewed sample, cross-checked against an LLM-based labeling assistant, retained every usability label after discussion on all 60 manually audited items, which suggests the 20.9% figure is not an artifact of sloppy automated labeling. It is a real measurement of a real blind spot.

For brands, the implication runs in a direction most content teams have not considered. The conversation around AI visibility has focused almost entirely on what gets cited: which domains, which pages, which phrasing wins a spot in a generated answer. This paper points at a prior, more basic failure. Before a page can be cited accurately, it has to be fetched successfully, and "successfully" has been defined by a status code that turns out to measure almost nothing about whether the content arrived intact. A financial services firm whose investor-relations page sits behind a cookie-consent wall that renders as an empty shell to a static fetcher is not being misquoted. It is being silently skipped, or worse, having stale cached content attributed to it.

Where this bites hardest

The sectors with the most to lose are the ones whose web infrastructure was built for compliance and control, not for scraping ease. Multilateral institutions and UN agencies routinely gate reports behind PDF portals, session-based downloads, or geofencing that returns a 200 with a block message instead of the document. Industrial groups and financial institutions layer bot-detection and WAF challenges on investor and regulatory pages precisely because they are high-value targets for scraping, which means the same defenses that stop malicious bots also return empty-shell pages to the legitimate retrieval systems feeding answer engines. Philanthropic and policy organizations publishing primary research behind JavaScript-heavy content-management systems face the app-shell problem directly: the page loads fine for a human in a browser and returns structurally empty for a static client racing to answer a user's question in real time.

The fix is not complicated, but it is not nothing either. Server-side rendering, or at minimum ensuring that primary content, key claims, data points, named findings, renders in the initial HTML response rather than after JavaScript execution, is the difference between being cited and being invisible while technically "live." Checking whether a bot-challenge or consent wall fires for non-authenticated, non-cookied requests, the exact conditions a retrieval crawler operates under, is a five-minute audit most communications teams have never run.

The uncomfortable part of this finding is not that 20.9% of fetches fail. It is that nobody measuring "citation accuracy" or "source reliability" in AI answers has been accounting for silent fetch failure at all. Every study of what gets cited assumes the citation was built on content that actually arrived. This paper's contribution is showing that assumption fails one time in five, and that the measurement infrastructure built on top of it, including most audits of brand visibility in LLM answers, has been quietly blind to its own error bars.