OpenAI's fetch bot reaches sites that disallowed it

ChatGPT's page-fetching bot, ChatGPTBot, is crawling websites that have explicitly blocked it via robots.txt. Search Engine Journal reports that new data confirms the bot is reaching disallowed sites, and OpenAI's own documentation provides the mechanism: the company distinguishes between crawling for training and fetching for real-time retrieval, and robots.txt may not govern the latter.

The distinction matters more than it first appears. ChatGPTBot is the crawler associated with training data collection, and most robots.txt blocks were written with that function in mind. OpenAI's documentation describes a separate process for real-time web fetching, used when a user's query triggers a live search. That fetch function may operate outside the standard robots.txt protocol. In short, a brand that believed it had opted out of ChatGPT's reach may not have done so at all.

The two bots brands have been treating as one

Web administrators have spent the past two years updating robots.txt files to block ChatGPTBot, the well-documented training crawler. The assumption was reasonable: robots.txt has governed crawler access since 1994, and every major search engine honours it. OpenAI encouraged that assumption by publishing its crawler's user-agent strings and directing site owners to use them.

The fetch bot used for real-time retrieval operates under different logic. It serves a live user request rather than a background indexing run, and OpenAI's position appears to be that the retrieval context exempts it from the opt-out mechanism that governs training. Whether that interpretation is legally or ethically defensible is a separate question; the operational reality is that the block is not working as intended.

For brands in financial services, multilateral institutions, and major industrial groups, this has a concrete implication. Many of these organisations have legal or compliance reasons to control how their content appears in AI-generated answers. A bank's regulatory disclosures, a UN agency's embargo policies, an industrial group's investor communications: these are not content categories where "your robots.txt may not apply" is an acceptable answer. Content governance frameworks built on the assumption that robots.txt controls AI access now have a gap in them.

Citation without consent

The visibility question cuts both ways. A brand that wants to appear in ChatGPT's answers benefits if the fetch bot can reach its pages regardless of technical blockers. A brand that wants to stay out, or to control which content surfaces, finds that its primary technical lever has been weakened.

The data showing ChatGPTBot reaching disallowed sites suggests this is not theoretical. It is already happening. Brands monitoring their server logs should expect to see fetch activity from OpenAI's infrastructure even where robots.txt instructions explicitly prohibit it.

The practical response has two parts. First, review server logs to confirm whether OpenAI's fetch bot is accessing content that the robots.txt file was meant to exclude. Second, escalate the question from an SEO or web-ops concern to a legal and communications one. Robots.txt was a technical gentleman's agreement; it carries no statutory force. If controlling AI retrieval of sensitive content matters to your organisation, the mechanism to enforce it is not a text file in the root directory.

OpenAI's move, whether deliberate policy or an underexplained technical distinction, shifts the terms of engagement between publishers and AI platforms. The reasonable expectation that a disallow directive would be honoured has been broken. Brands that have relied on it to manage their AI-era content footprint should treat that assumption as unverified until they check their logs.