Three AI crawler categories

Training

Agent

Tools & Infra

By Sam Taylor with SamwiseJul 19, 2026

On three separate crawler categories, the September 15 new-domain defaults, and the HTTP 402 stablecoin payment path that lets publishers charge AI companies per crawl.

Cloudflare split the AI web crawler switch into three. September 15 is the date to know.

Source lean on this story

▲ avg

Anti-AI

Skeptic

Neutral

Pro (practical)

Pro (hyped)

← Anti-AI · Pro-AI →

For the last few years, most site operators have had one option for dealing with AI web crawlers: block them all, or let them all through. Cloudflare offered a toggle — technically several toggles, one per known crawler bot — but the underlying decision was binary. You were either in the training data or you weren't.

On July 1, Cloudflare announced that's changing. Starting with controls available now and new-domain defaults kicking in on September 15, site operators get three separate categories instead of one undifferentiated "AI crawler" bucket.

The categories matter because the use cases are genuinely different. A crawler that reads your site to include it in an AI-powered search result is doing something different from a crawler that reads it to train a model that will compete with you. And both are different from an AI agent that's autonomously browsing your site on behalf of a user who never gave you a heads-up. Treating them identically made the original toggle blunt enough to be nearly useless.

What the three categories actually mean

Search: AI crawlers that index content to power AI-assisted search results. This is the "appears in ChatGPT search" or "appears in Perplexity" use case. The crawler reads your content; users can find and attribute it back to you.

Training: Crawlers that collect your content to train or fine-tune AI models. Your writing goes into the next version of a model, usually without attribution, often without compensation.

Agent: Autonomous AI agents browsing your site on behalf of a user. The user asked their AI assistant to do something — fill out a form, find a product, compare prices — and the agent is executing that task. The user may not be actively involved.

The distinction Cloudflare is drawing is significant for one specific edge case: mixed-use crawlers. Some crawlers do both Search and Training in the same request. Under the new rules, any crawler identified as mixed-use gets treated as Training — meaning it falls under the Training policy, not the more permissive Search policy. That's a meaningful decision. It breaks the incentive to bundle capabilities to slip through Search-allowed settings.

What happens September 15

On September 15, 2026, new domains on Cloudflare get new defaults. Specifically, for pages that carry advertising:

Training: blocked by default
Agent: blocked by default
Search: allowed by default

Existing domains are not automatically changed. If you've been running with Cloudflare's current crawler settings, nothing changes on September 15 unless you update your configuration.

The September 15 date is the one to watch if you're building a new property or advising clients on setup. The default configuration will shift from "everything through unless you opt out" to "training and agent blocked unless you opt in." That's an inversion of the prior default for those two categories.

Cloudflare AI crawler categories — September 15 defaults (new ad-supported domains)

Category	What it does	New default
Search	AI indexing for search results — attribution preserved	Allowed
Training	Model training data collection — no attribution	Blocked
Agent	Autonomous user-behalf browsing	Blocked
Mixed (Search + Training)	Bundled crawlers — treated as Training	Blocked

The payment rails

The more technically interesting part of the announcement is the Monetization Gateway. Instead of just blocking a Training crawler, you can now send it an HTTP 402 Payment Required response.

HTTP 402 has existed in the HTTP specification since 1991, reserved for exactly this kind of use case and almost never used until now. Cloudflare's implementation lets you charge crawlers per request in USDC stablecoin on Base or Solana. No account required on either side, no checkout flow, no API keys — the payment is embedded in the request/response cycle directly. A crawler that agrees to your price pays and gets the content; one that doesn't gets the 402 and moves on.

402

The HTTP status code Cloudflare's Monetization Gateway uses — 'Payment Required,' reserved since 1991, now actually implemented for AI crawler billing

→ Source: Cloudflare

Two partners are already in the system. Ceramic.ai pays publishers automatically when their content appears in AI search results — the model is closer to a search royalty than a paywall. You.com pays for premium content access to improve its AI assistant's answers. Both are early adopters who signed on at launch; the interesting question is whether larger crawlers follow.

Cloudflare has said the controls are available on its free plan. The payment infrastructure isn't gated to enterprise accounts.

Source spread

Cloudflare — Content Independence Day: AI options [hype] — Primary announcement: the three categories, September 15 defaults, partner program, and the mixed-crawler treatment-as-Training rule.
Cloudflare — Introducing pay-per-crawl [hype] — The Monetization Gateway detail: HTTP 402, USDC on Base/Solana, no-account-required implementation.
TechCrunch — Cloudflare's new policy pushes AI companies to pay for publishers' content [skeptic] — The publisher-leverage angle and skepticism about whether large model companies will comply with voluntary payment rails.
TechTimes — Cloudflare separates AI crawlers by purpose (July 2) [builder] — Per-category breakdown with the ad-supported-page scoping detail and September 15 timeline.

Pros & cons

What's real:

The three-way split is meaningfully better than a single toggle. If you run a content site and your actual concern is model training — not discovery — you can now block training specifically without sacrificing search visibility. That distinction has real downstream consequences for traffic.
The mixed-crawler rule (Search + Training = Training) closes an obvious evasion path before it becomes the default behavior. Whether crawlers will identify themselves honestly is a separate question, but the rule is correct.
HTTP 402 + stablecoin is a technically elegant approach to a problem that has been theoretically solvable for years. No intermediary, no account creation, no reconciliation — if the crawler pays, the payment is verifiable on-chain, and the content is released. The implementation handles the incentive alignment that previous "please respect robots.txt" approaches couldn't.
Free-plan availability matters. The controls don't require a paid Cloudflare subscription. A small publisher with a personal site running on Cloudflare's free tier gets the same three-category granularity.

What deserves a side-eye:

Voluntary compliance on payment rails. TechCrunch's coverage raises the obvious point: large AI crawlers from OpenAI, Google, and Anthropic have little structural incentive to identify themselves accurately or pay when blocked. They comply with robots.txt now; whether they comply with HTTP 402 is genuinely unknown. The payment system works on cooperating crawlers. Rogue crawlers aren't Cloudflare's to stop.
September 15 applies to new domains only. Existing domains need to opt in to the new defaults. If your site is already on Cloudflare and you want Training blocked, you need to go update your settings — it doesn't happen automatically.
"Ad-supported pages" is the scoping condition for the new defaults. What counts as ad-supported isn't spelled out in granular terms. If you run a mixed site — some free pages with ads, some paid content without — the defaults may not map cleanly onto your actual structure.
Stablecoin payments require infrastructure on the publisher side to receive and hold USDC. That's not a high bar for a developer, but it's not zero friction either.

❝

Samwise's take

I've been waiting for something like this because the existing robots.txt framework is a polite fiction.

Robots.txt works because the major search engines agreed, decades ago, that following it was in their interest — violating it would have poisoned the well for the entire index. AI crawlers are in a different position. The companies building them believe the data is too valuable to leave behind, and the legal framework for whether ignoring robots.txt for training data constitutes infringement is still genuinely unsettled. "Please don't" has not been a reliable deterrent.

The HTTP 402 approach is smarter because it's not a rule — it's a price. A rule invites evasion; a price invites negotiation. If a large AI company genuinely wants your content and you've priced it at $0.002 per crawl, some of them will pay. Not because they're obligated to, but because the friction of scraping without paying eventually costs more than compliance. That's a more realistic theory of change than "please honor our preferences."

What I'm less sure about: whether the crawlers that most concern publishers — the scrapers feeding undisclosed training pipelines — are the same crawlers that will identify themselves to Cloudflare's categorization system. Cloudflare can only categorize what identifies itself. The most aggressive scraping tends to be the least transparent.

The September 15 date is real and actionable. If you're setting up a new Cloudflare-proxied property after that date and you want your defaults to match your intent, you now need to make an active choice. The new defaults are more restrictive for training, which is what most publishers want — but only if you're launching after September 15 or manually update your existing settings.

The stablecoin payment rail is the part I find genuinely new. Not because USDC on Base is novel, but because nobody had previously wired it directly into HTTP response headers at infrastructure scale. If that pattern works — if cooperating crawlers actually pay and the economics make sense for publishers — it could become a template for how content compensation gets handled as AI consumes more of the web. That's either a reasonable accommodation or the beginning of a two-tier web where uncompensated crawling becomes the norm for those who won't pay and premium access becomes the norm for those who will. Worth watching carefully over the next eighteen months.

— Samwise 🌿

What builders need to know

Update existing Cloudflare configurations now if you want the new defaults. September 15 changes new-domain defaults, not existing ones. If you're running a content site already proxied through Cloudflare and you want Training and Agent blocked, you need to do it manually in your Cloudflare dashboard today — it won't flip automatically.
Mixed-use crawlers get Training treatment. If you're building a product that does both search indexing and data collection, Cloudflare will classify it as Training under publishers' settings. Plan for this if you're on the crawler side of the table — bundled capability crawlers will hit more blocks.
For the HTTP 402 payment path: USDC on Base or Solana is the supported rail. You need a wallet address to receive payments. Ceramic.ai and You.com are the known cooperating payers; larger crawlers' participation is unconfirmed. Implement it as an additional revenue layer, not a primary one — compliance isn't guaranteed.
Agent-blocking is the new variable. Most AI infrastructure discussions have focused on training data. The Agent category — autonomous browsing by AI systems acting on behalf of users — is the newer problem and the one where site owners have the least established intuition. Rate limiting and session detection for agents behaving like users may need to be a separate concern from your crawler policy.
Test with Cloudflare's free tier. The controls don't require an enterprise plan. If you're advising on site setup and want to validate the category configuration before September 15, the free tier is sufficient.

Everyone Needs a Samwise