Apr 7, 2026

Cloudflare Pay Per Crawl: Making AI Pay to Read Your Website

When AI crawlers scrape your blog day after day, feeding your content into model training and user-facing answers without a single cent flowing back to you, the asymmetry is glaring. Cloudflare is now trying to flip this dynamic using an HTTP status code that's been collecting dust for over 25 years: 402 Payment Required.

The Current State: AI Crawlers Are Draining the Open Web

According to Cloudflare's data, AI crawlers send over 50 billion requests per day to its network, accounting for 4.2% of all HTML request traffic. Imperva's report goes further, showing that in 2024, bot traffic surpassed human traffic for the first time, reaching 51% of all web traffic.

But the most striking finding isn't the traffic volume — it's the massive gap between what AI companies "take" and what they "give back."

Crawl-to-Click Gap

Cloudflare's data reveals a staggering asymmetry: Anthropic crawls 38,066 pages for every 1 human visitor it sends back; OpenAI's ratio is 1,091:1; even the relatively friendlier Perplexity sits at 195:1. By comparison, Google's ratio is 5.4:1 — it sends back one visitor for roughly every 5 pages crawled.

This is exactly what Cloudflare CEO Matthew Prince meant when he said: "If the web is to survive the AI era, we need to give publishers the control they deserve and build a new economic model that works for everyone."

The Power Map of the AI Crawler Ecosystem

Over the past year, the competitive landscape among AI crawlers has shifted dramatically.

AI Crawler Market Share

The most notable change is ByteDance's Bytespider plummeting from 42% to 7% (an 83% drop), while OpenAI's GPTBot surged from 5% to 30% (a 500% increase). Meta's ExternalAgent rocketed from zero to 19%. This reflects the fierce competition among major AI companies for training data.

AI Bot Traffic Purpose

Even more telling: 80% of AI crawler traffic is for model training, with only 17% for AI search and 3% for user interaction. In other words, the vast majority of crawlers aren't helping your readers find your content — they're carting it away.

What Is AI Crawl Control?

Cloudflare's AI Crawl Control (formerly AI Audit) went GA on August 28, 2025, offering website owners three options to control each AI crawler's access:

Allow: Free access
Charge: Collect fees via Pay Per Crawl
Block: Deny access

A major policy shift: starting July 1, 2025 (what Cloudflare calls "Content Independence Day"), all newly onboarded domains on Cloudflare default to blocking AI crawlers. Website owners must actively opt in to allow them. Given that Cloudflare powers 20% of all websites globally, this is a far-reaching decision.

Since the policy took effect, Cloudflare has blocked over 416 billion AI crawler requests. More than 1 million customers have chosen to block AI crawlers.

Pay Per Crawl: The Resurrection of HTTP 402

HTTP 402 Payment Required was defined in 1999 and has been "reserved for future use" ever since — dormant for over 25 years. Cloudflare's Pay Per Crawl finally gives it a real purpose.

The Technical Protocol Flow

Pay Per Crawl supports two payment flows:

Discovery-First (Passive Discovery) Flow:

1. Crawler → GET /article (with signed headers)
2. Server → HTTP 402 + crawler-price: USD 0.01
3. Crawler → GET /article + crawler-exact-price: USD 0.01
4. Server → HTTP 200 + crawler-charged: USD 0.01 + content

Intent-First (Active Bidding) Flow:

1. Crawler → GET /article + crawler-max-price: USD 0.05
2. Server → HTTP 200 + crawler-charged: USD 0.01 + content
   (returns 402 if price exceeds budget)

Custom HTTP Headers

Header	Direction	Example	Purpose
`crawler-price`	Response (402)	`USD 0.01`	Server declares price
`crawler-max-price`	Request	`USD 0.05`	Crawler declares max bid
`crawler-exact-price`	Request	`USD 0.01`	Crawler agrees to exact price
`crawler-charged`	Response (200)	`USD 0.01`	Confirms amount charged
`crawler-error`	Response (402)	`InvalidCrawlerExactPrice`	Error code

Web Bot Auth: Ed25519 Cryptographic Identity Verification

To prevent crawlers from spoofing their identity, Pay Per Crawl uses a verification mechanism based on RFC 9421 HTTP Message Signatures. Each crawler must:

Generate an Ed25519 key pair
Publish the public key in JWK format at /.well-known/http-message-signatures-directory
Include three headers with every request: Signature-Agent (pointing to the key directory), Signature-Input (signature metadata), and Signature (Ed25519 signature)

Payment headers (crawler-exact-price or crawler-max-price) must be included in the signature input to prevent tampering and replay attacks.

Notably, Cloudflare has submitted these technologies as an IETF draft (draft-meunier-web-bot-auth-architecture), with Google as a co-author — hinting at industry consensus on standardization.

Connection to x402

Cloudflare also co-launched the x402 protocol with Coinbase, enabling HTTP 402 payments using stablecoins (USDC). While Pay Per Crawl currently uses traditional billing (credit card, daily settlement), x402 could become an alternative payment channel in the future, enabling decentralized real-time micropayments.

How Much Can You Actually Earn?

This is the most practical question. Let's answer it with data.

Revenue Tiers

At $0.01 USD per crawl (the current minimum), assuming 1-2% of traffic comes from AI crawlers:

Site Scale	Monthly Pageviews	AI Crawler Visits	Est. Monthly Revenue
Personal blog	50K	500-1,000	$5-$ 10
Mid-size blog	500K	5K-10K	$50-$ 100
Large blog	2M	20K-40K	$200-$ 400
Major publisher	100M	1M-2M	$10K-$ 20K

The reality is harsh: for most personal blogs, Pay Per Crawl revenue amounts to roughly the price of a cup of coffee. But for major publishers, it's a non-trivial income stream.

More importantly: many crawlers may simply stop crawling when they receive a 402 instead of paying up. Stack Overflow observed exactly this phenomenon after enabling Pay Per Crawl.

The Big Picture: The AI Content Licensing Market

To understand what Pay Per Crawl really means, you need to zoom out.

Licensing Deals

Content licensing deals between AI companies and publishers currently total approximately $2.92 billion**, with annual licensing spend in 2024 around **$ 817 million. OpenAI accounts for 52.9% of deal volume.

Some notable deals:

News Corp + OpenAI: $250M over 5 years (~$ 50M/year)
Reddit + Google: $60M/year
Reddit + OpenAI: ~$70M/year
Shutterstock + multiple AI companies: $104M in 2023 revenue

But these deals are reserved for top-tier publishers. Average bloggers and small publishers have zero negotiating leverage. Pay Per Crawl's real value lies in democratization — enabling any website on Cloudflare to participate in this economic system, not just publishers with legal teams at their disposal.

Meanwhile, pressure on publishers is intensifying. Google search referral traffic dropped 27% globally in 2025, with news site organic traffic falling from 2.3 billion to 1.7 billion monthly visits — a loss of 600 million visits. AI Overview has a click-through rate of just 8%, compared to 15% for traditional search results.

Major AI Crawlers at a Glance

Crawler Name	Operator	Purpose	Respects robots.txt?
GPTBot	OpenAI	Training	Yes
ChatGPT-User	OpenAI	Inference/Browsing	Yes
ClaudeBot	Anthropic	Training	Yes
Meta-ExternalAgent	Meta	Training (Llama)	Delayed compliance
Bytespider	ByteDance	Training	No (crawls but ignores)
PerplexityBot	Perplexity	AI Search	Yes
CCBot	Common Crawl	Open corpus	Yes
Amazonbot	Amazon	AI services	Yes
Applebot-Extended	Apple	Apple Intelligence	Yes
DeepSeekBot	DeepSeek	Training	No (unidentifiable UA)

Notably, about 5.7% of AI crawlers use spoofed User Agents, and the new generation of browser-based AI agents (such as ChatGPT Atlas/Operator, Google Mariner) use standard Chrome UAs, making them nearly impossible to detect through traditional methods. This makes Web Bot Auth's cryptographic identity verification all the more critical.

How to Set It Up (Implementation Guide)

Step 1: Add Your Domain to Cloudflare

Click "Add Site" in the Cloudflare Dashboard
Enter your domain
Select the Free plan (AI Crawl Control is available on the free tier)
Under AI crawlers settings, select "Do not block (allow crawlers)"
Update your domain registrar's nameservers to the Cloudflare NS records provided

Step 2: Review AI Crawler Data

Once your domain is active, go to the AI Crawl Control panel:

Crawlers tab: See which AI crawlers are visiting your site
Analytics tab: View crawl volume trends

Step 3: Apply for Pay Per Crawl Beta

Go to Cloudflare's Pay Per Crawl application page
Once approved, enable Pay Per Crawl under AI Crawl Control > Settings
Set your per-crawl price (minimum $0.01 USD)
Connect a Stripe account to receive payments
Set the crawlers you want to charge to "Charge" in the Crawlers tab

Free Paths

The following paths are never charged:

/robots.txt
/sitemap.xml
/security.txt
/.well-known/security.txt
/crawlers.json

Practical Recommendations

For personal blogs (like this one):

Start by enabling AI Crawl Control (free) to observe the data
Understand how many AI crawlers are visiting your site
Don't rush to apply for Pay Per Crawl — revenue for small sites is limited
Consider strategically allowing AI search crawlers (Perplexity, OAI-SearchBot) for traffic referrals
Block pure training crawlers (unless you're willing to let them use your content for free)

For mid-to-large publishers:

Apply for Pay Per Crawl Beta immediately
Simultaneously explore direct licensing deals (higher revenue potential)
Use AI Crawl Control data as negotiation leverage

Conclusion

Pay Per Crawl won't make bloggers rich on its own, but it represents a significant paradigm shift: content on the web is no longer a free-for-all resource taken for granted. HTTP 402 waking up after 25 years of dormancy may be one of the most significant structural changes the AI era has brought to the open web.

Cloudflare is already sending over 1 billion 402 responses per day. Regardless of your site's size, at the very least turn on AI Crawl Control and look at the data — you might be surprised by how many AI crawlers are quietly reading your content.

References: