Three days. That's the runway before Cloudflare's new AI crawler defaults take effect across every site they proxy, and I want to be precise about what "defaults" means here, because the details matter.

On July 19, Cloudflare announced it was splitting AI crawler controls into three distinct categories: Search, Training, and Agent. Different rules for each. September 15 is when the new default configuration applies to any site that hasn't made an explicit choice. If you haven't touched the dashboard settings, your site's behavior changes that day. Whether the new behavior matches what you actually want is a separate question from whether you've thought about it.

Here's the split:

AI Search crawlers — bots that index your content to cite it in model-powered answers. ChatGPT search, Gemini AI Mode, Perplexity, Claude.ai web retrieval. Default after September 15: allowed. Cloudflare's reasoning: being cited in an AI answer drives referral traffic back to you. That argument is partially right. The referral math holds for high-authority, brand-name sites. For long-tail content, it's murkier.

AI Training crawlers — bots that ingest your content to train new foundation models. Default: blocked. The commercial-data-without-compensation problem that's been simmering since the first copyright litigation wave.

AI Agent crawlers — autonomous agents that take actions on your pages. Submitting forms, clicking links, executing multi-step workflows against your site. Default: blocked. The liability concern is obvious. Less obvious: this category also covers legitimate use cases — accessibility agents, user-authorized automation, integration workflows your own customers are running.

So the defaults are: Search yes, Training no, Agent no. If those match your preferences, you need to do nothing. If they don't, you have three days.

The HTTP 402 training tier

This is the part that's easy to skim past. Cloudflare built a commercial opt-in tier into the Training category: if you want AI labs to train on your content, you can flip that on and receive micropayments via stablecoin. Cloudflare's implementation uses USDC on Base, settling daily into a wallet you connect through the billing interface. The HTTP 402 status code is how a compliant crawler identifies itself as payment-eligible.

I have real uncertainty about whether this will work at scale. The payment flow requires labs to integrate with Cloudflare's rails. As of September 12, Anthropic and OpenAI have confirmed integration; Google is listed as 'coming soon'. That gap matters: if your content is primarily valuable for Gemini training, opting into the commercial Training tier currently earns you nothing from your most significant potential buyer. The mechanism is right in principle. The adoption pipeline has holes.

Source spread

• Cloudflare Blog — "Content Independence Day: AI options" [builder] — the July 19 announcement. Has the three-tier structure, HTTP 402 design rationale, and September 15 timeline. Dense but readable; the pricing table is buried at the end.
• Cloudflare Developers — AI Crawler Controls docs [builder] — current per-lab integration status; the source for which crawlers actually support the 402 payment flow today vs. "coming soon."
• TechCrunch — "Cloudflare's AI paywall hits a practical problem: most labs haven't integrated" [skeptic] — mid-August coverage of the slow integration timeline; Google's absence on the payment rails called out specifically.
• The Verge — "Who controls what AI sees?" [neutral] — the clearest plain-language walkthrough of what each category actually means in practice. Read this first if the Cloudflare docs feel dense.

Pros & cons

What's actually good about this:

• The three-tier taxonomy is the right structure. "Allow AI crawlers" as a single binary was always too coarse. A site that's fine with appearing in AI search results may be very much not fine with its content being training data. Cloudflare built the distinction in, and it's overdue.
• The defaults are conservative on the categories that carry real liability. Sites that do nothing get protected from training ingestion and agent actions without having to understand any of this infrastructure.
• If the stablecoin rails mature and labs actually integrate, this is the first mechanism that could turn AI content access into a real licensing market. The outcome — content creators compensated for training data — is the right one, even if the current state is uneven.

What deserves skepticism:

• Search crawlers allowed by default is worth sitting with. Gemini AI Mode, ChatGPT search, and Perplexity are in that category. They cite your pages — but they also answer questions without generating a click-through. The traffic benefit Cloudflare is citing doesn't reliably materialize; there's a well-documented zero-click pattern in AI-powered search that the default doesn't account for.
• The Google integration gap. As of today, Google's training crawlers don't support the HTTP 402 payment flow. If your content's most significant commercial consumer is Google DeepMind, the commercial Training opt-in earns you nothing from them until that changes.
• "Agent crawlers" is underspecified in the current docs. Cloudflare lists accessibility agents and user-authorized automation as planned exceptions — but that exception mechanism isn't live in the dashboard yet. Blocking all Agent crawlers catches legitimate use cases alongside predatory scrapers, and the workaround (per-bot allowlisting) requires knowing the bot's user-agent string, which isn't published for most agents.

What builders need to know

• The path is Cloudflare Security → Bots → AI Crawler Controls. That's where your current configuration is visible and where you change it. The September 15 switch only affects sites that have never made an explicit selection — if you've touched these settings before, your config is preserved.
• If you have user-authorized agent workflows, log a support ticket with Cloudflare now. The dashboard exception mechanism for Agent crawlers isn't live yet. The workaround is per-bot allowlisting by user-agent string — available but tedious if you have multiple agent integrations.
• The commercial Training opt-in is worth reading. Cloudflare is publishing a rate schedule of $0.50–$4.00 per 1,000 pages indexed, depending on the lab. Anthropic and OpenAI are currently integrated. Google is not. If your content is specialized and high-quality — technical documentation, structured knowledge, long-form analysis — the economic case for opting in is real, even at current partial integration.
• The robots.txt token spec is available for non-Cloudflare stacks. X-Robots-Tag: ai-training: none as a response header; X-AI-Training: disallow in your robots.txt file. The major crawlers that support the HTTP 402 flow also respect these tokens, so this isn't Cloudflare-only behavior.
• Don't over-interpret the September 15 deadline. The defaults change; your explicit settings don't. If you've already configured AI crawler behavior through a bot management product, confirm it maps to the new three-tier taxonomy — but you're not starting from zero, and the sky isn't falling on September 16.

Further reading

• Cloudflare Blog — Content Independence Day: AI options — the July 19 announcement with design rationale and the full pricing table
• Cloudflare Developers — AI Crawler Controls documentation — current per-lab integration status and the commercial tier pricing schedule
• TechCrunch — Cloudflare's HTTP 402 integration gap — which labs are and aren't live on the payment rails as of mid-August
• The Verge — Who controls what AI sees? — plain-language walkthrough of the three categories and what each one protects