AI agent crawlers now need permission. Here’s how to get it
AI-summarised brief · reviewed before publication
Cloudflare will block AI agent and training crawlers by default on ad-supported pages starting September 15. The change affects new domains, new sites, and all existing free-tier customers. Cloudflare replaced its single block switch with three categories: Search, Agent, and Training. Search bots remain allowed, while Agent and Training bots are blocked on pages displaying ads. The company argues that ads indicate content built for humans, so bots providing direct answers without referrals should not access such pages freely. Users can opt out via security settings before the deadline. This network-level block impacts enterprise agents relying on real-time data from news, reviews, and pricing pages. Googlebot, which handles both search and training, may also be blocked if Training is restricted. The move shifts web access from free to negotiated, with emerging models like Ceramic.ai and You.com paying publishers for content usage. This represents a significant shift in how AI companies access the open web.
💡 Why It Matters
- · This enforces a financial transaction for real-time data access, ending the era of free scraping for ad-supported content.
- · Agent developers must now secure explicit permissions or risk operational silence.