Cloudflare splits AI crawlers into three tiers from September
Cloudflare's default-block on Training and Agent crawlers reshapes LLM access to brand content from September 15.
Key takeaways
- From Sept 15, Cloudflare blocks Training and Agent bots by default on ad-supported pages, while Search crawlers remain permitted.
- Search crawlers feed real-time LLM retrieval; blocking them would remove a brand from ChatGPT and Perplexity citations.
- Enterprises with custom Cloudflare configurations must audit settings before September 15 — defaults do not override bespoke rules.
- Agent crawlers, now a distinct tier, pose the highest compliance risk for financial services and regulated institutions.
- Any content strategy treating AI visibility as a single toggle is now structurally out of date.
Cloudflare protects roughly 20% of all websites. What it decides to block, the internet's AI pipelines cannot easily reach. The Decoder reports that from September 15, 2026, Cloudflare will replace its all-or-nothing AI bot controls with a three-tier system, separating crawlers by purpose: Search, Training, and Agent. On ad-supported pages, Training and Agent bots will be blocked by default. Site owners who do nothing will, for the first time, permit Search crawlers while shutting out model trainers and autonomous agents simultaneously.
That distinction is not cosmetic. Search crawlers feed the retrieval pipelines that power ChatGPT, Perplexity, and Google's AI Overviews. Training crawlers feed the next generation of base models. Agent crawlers are the newest category: bots acting on behalf of users in real time, fetching, summarising, and transacting. Until now, a site owner choosing to block AI bots had to choose all or nothing. Cloudflare's new default collapses that blunt instrument into something more surgical.
The default is the decision
Most site administrators will not change a default. This is not cynicism; it is the empirical record of every major platform policy that has ever shipped with a pre-ticked box. Cloudflare's choice to auto-block Training and Agent crawlers on ad-supported pages while permitting Search crawlers will therefore function as policy for the majority of the web.
For brand visibility in LLM answers, the implication is direct. Search crawlers permitted by default means that content on Cloudflare-protected sites remains eligible for retrieval-augmented generation, the mechanism through which ChatGPT and Perplexity surface cited sources in their answers. A brand's white papers, technical documentation, and press releases sitting behind a Cloudflare proxy will still, under this default, be findable by the models that answer industry queries in real time. That is a better outcome than the blanket block many site owners previously activated without fully understanding what they were surrendering.
Training crawlers blocked by default is a separate matter with longer time horizons. It reduces the probability that a brand's proprietary content trains the next version of a frontier model without consent or compensation. For multilateral institutions and policy bodies, where authoritative documents carry legal and normative weight, this default offers a form of provenance protection that a robots.txt file never fully guaranteed.
Where the risk now concentrates
The Agent tier is where the stakes are least understood. Agent crawlers do not index for later retrieval; they act now, on behalf of a user, often invisibly. A financial services firm that allows Agent crawlers to traverse its site is, in effect, permitting third-party AI systems to read, extract, and act on its content in real time, potentially in ways that bypass the compliance and disclosure controls those firms spend considerably to maintain. Cloudflare blocking Agents by default on ad-supported pages does not cover every enterprise configuration; it covers the common case.