# AI Traffic Is a Trade, Not a Channel: How to Compute What Your Crawl Budget Actually Earns

> Every AI traffic study counts what comes in. None of them count what goes out. Cloudflare measured one platform taking 70,900 pages for every visitor it returned. This is the other half of the ledger — how to compute revenue per 1,000 AI crawls on your own site, why the answer differs 300× between engines, and what to do when it comes back negative.

Author: Vincent Ruan (https://www.vince-ruan.com/)
Publisher: Attrifast — https://attrifast.com
Last updated: 2026-08-16
Canonical: https://attrifast.com/blog/ai-crawl-to-revenue

**TL;DR**

- Every published AI-traffic study measures **arrivals**. Semrush's 50,000-site
  study is the best of them and it reports AI at **0.14% of visits, growing 66%**
  in 2025. Both numbers are true and neither tells you whether to invest.
- The other half of the ledger is what AI **takes**. Cloudflare defines a
  crawl-to-refer ratio — HTML fetches divided by referred visits — and published
  one platform sitting at **70,900:1** in a single week of June 2025.
- A ratio is still not a decision. **4,000:1 is fine if those referrals convert
  at 10×**, and ruinous if they don't. The missing denominator is money.
- The metric this article proposes is **RP1K — revenue per 1,000 AI crawls**,
  computed per engine from three numbers you already have: crawler hits,
  attributed revenue, and which engine sent it.
- Improving it is not "publish more". Across 47,097 citations, **72% of cited
  pages looked recent by last-update date and only 42% by publish date** — and
  the pages that held their citation for four straight months had a *higher*
  median age than the ones that spiked for one.

## Everyone is counting one side of a two-sided exchange

Here is the state of AI-traffic measurement in 2026, and it is genuinely good
work. Semrush ran the numbers across more than 50,000 websites and 17
industries, worldwide, desktop and mobile, for all of 2025. AI traffic grew
66.02% — from 462 million to 767 million monthly visits — and still landed at
0.14% of total visits against 64.69% direct and 16.04% organic.[1]

**Share of all web visits by channel, December 2025**

| Category | Value |
|---|---|
| Direct | 64.69% |
| Organic search | 16.04% |
| AI (all engines) | 0.14% |
| Google AI Mode | 0.01% |

*Source: Semrush Traffic & Market Toolkit — 50,000+ websites, 17 industries, worldwide desktop + mobile, Jan–Dec 2025*

That study is careful and its numbers are probably the best public estimate we
have. My argument is not with the data. It is with the shape of the question.

Every one of these studies — Semrush's, Similarweb's, the ones built on
clickstream panels — measures **arrivals**. Visits in. That framing treats AI
like a channel: a pipe that delivers people, to be sized against other pipes.

Growth rates make the framing worse, not better. Here is AI's 66% against the
other channels in the same study:

**Pages crawled per referral sent back — 2025 growth rates say nothing about this**

| Category | Value |
|---|---|
| Paid search | 75.84% growth in 2025 |
| AI | 66.02% growth in 2025 |
| Display | 62.76% growth in 2025 |
| Referral | 53.39% growth in 2025 |
| Organic search | 2.38% growth in 2025 |

*Source: Semrush channel study (2025 growth) vs Cloudflare Radar crawl-to-refer, week of 19–26 June 2025*

Paid search grew faster than AI last year. Nobody wrote a think-piece about it,
because paid search's base is enormous and everyone already knows what to do
with it. AI's 66% is only remarkable because the base is 0.14%, and a percentage
computed on a base that small can move an order of magnitude on noise.

The real problem is different, though, and it is structural: **a visit count can
only ever measure what you were given.** It cannot measure what you gave.

## The other half of the ledger

Cloudflare sits in front of a large share of the web, which means they can see
both directions at once. They defined the metric that makes the exchange
visible:

> dividing the total number of requests from relevant user agents associated
> with a given search or AI platform where the response was of
> `Content-type: text/html` by the total number of requests for HTML content
> where the `Referer` header contained a hostname associated with a given
> search or AI platform[2]

In plain terms: **how many pages did this platform take, per human it sent
back.** For the week of 19–26 June 2025 Cloudflare published Anthropic at
70,900:1 and Mistral at 0.1:1 — a spread of nearly six orders of magnitude
between two platforms in the same week.[2]

Two things about those numbers, because they get quoted badly.

**They move, a lot.** Cloudflare says so explicitly, and the figures you will
see in secondary coverage — Anthropic at 4,580:1, OpenAI at 848:1 — are later
readings from Cloudflare Radar, not from that post. Radar publishes the ratio
live. Take the current number from there, not from an article, including this
one.

**A high ratio is not automatically theft.** A crawler that fetches your pages
for training will never send a referral, by construction, because the model
answers in the chat window. That is a real complaint, but it is a complaint
about the business model, not a measurement error. The measurement error is
assuming the ratio alone tells you what to do.

## Why the ratio still isn't a decision

Take two sites with an identical crawl-to-refer ratio of 4,000:1.

The first sells a $19/month tool. Its AI referrals convert at roughly the site
average and each one is worth about what any other visitor is worth. At 4,000:1
the trade is bad and getting worse, and the honest move is to stop optimising
for it.

The second sells $40,000 implementation contracts. Its AI referrals arrive
having already read a comparison page and a docs page, and close at four times
the rate of its organic traffic. At the same 4,000:1, the trade is one of the
best it has.

Same ratio. Opposite decisions. The ratio is a **rate of exchange with no
prices attached** — and you cannot evaluate an exchange rate without knowing
what you are buying.

## RP1K: revenue per 1,000 AI crawls

The fix is to finish the fraction. Put money in the numerator:

```
                 revenue attributed to engine E, period P
RP1K(E) = 1000 × ─────────────────────────────────────────
                  HTML crawls by engine E's bots, period P
```

Three inputs, all of which you can get today:

1. **Crawls by engine.** Your server or CDN logs, filtered to the bot
   user-agents and — this matters — **verified against the vendor's published IP
   ranges.** Anyone can send `User-Agent: GPTBot`. OpenAI, Anthropic, Perplexity,
   Google and Common Crawl all publish the ranges their bots crawl from, and an
   unverified hit belongs in a different bucket than a verified one.
2. **Referred sessions by engine.** Referrer hostname plus the UTM values each
   engine appends. This is the step most analytics get wrong, which is why AI
   traffic so often lands in Direct.
3. **Revenue attributed to those sessions.** Your payment processor, joined to
   the session that preceded it.

A worked example, on a 40-page B2B site over 30 days. These are illustrative
numbers — substitute your own:

| Engine | Verified HTML crawls | Referred sessions | Attributed revenue | RP1K |
|---|---|---|---|---|
| Perplexity | 4,100 | 22 | $36.50 | **$8.90** |
| ChatGPT | 31,600 | 41 | $44.90 | **$1.42** |
| Anthropic | 96,400 | 3 | $2.90 | **$0.03** |

**Worked example: revenue per 1,000 AI crawls on one 40-page site**

| Category | Value |
|---|---|
| Perplexity | $8.90 |
| ChatGPT | $1.42 |
| Anthropic | $0.03 |

*Source: Illustrative calculation from the formula in this article — substitute your own crawler log, referral and Stripe figures*

The spread here is roughly 300×, and it inverts the crawl volume: the bot that
took the most pages returned the least money. That inversion is the entire
point of computing this. A crawl-budget conversation based on hit counts would
have concluded that Anthropic is the most important AI relationship on the site.
It is the least.

Two properties make RP1K worth the trouble:

- **It is denominated in the thing you actually spend.** Crawl budget is real
  cost — bandwidth, origin CPU, and on some stacks a per-request bill.
- **It survives the zero.** If an engine sends no revenue, RP1K is `0`, which is
  a legible answer. A crawl-to-refer ratio with zero referrals is a division by
  zero, and gets quietly dropped from summaries.

### Reading the answer

| RP1K | What it means | What to do |
|---|---|---|
| Above your organic RP1K | This engine is your best-converting channel per unit of content given | Feed it: keep the pages it cites current, add the formats it retrieves |
| $0.50 – organic parity | A normal, working channel | Maintain. Do not build a programme around it yet |
| Near zero, with citations | It reads you and cites you, but the answer ends the journey | Optimise for the **mention**, not the click — brand copy, not landing pages |
| Near zero, no citations | Pure extraction | Rate-limit it in robots.txt and spend the budget elsewhere |

That last row is a real option and it is worth saying plainly: robots.txt
directives are honoured by the major AI crawlers, and OpenAI and Perplexity both
document a delay of roughly a day before a policy change takes effect. Blocking
is not free — you also lose the citations — which is exactly why you want the
number before you decide.

## Raising the numerator is not "publish more"

The instinct, once RP1K is low, is to publish. The best evidence available says
that is the wrong lever.

Seer Interactive analysed 7,683 pages carrying 47,097 citations across ChatGPT,
Gemini and Perplexity between March and June 2026, dating each page from its
structured signals — schema, sitemaps, headers.[3] Their headline
finding is that 75% of cited pages were updated within the last year.

The finding underneath it is the useful one. Measured by **last-update** date,
72% of cited pages looked recent. Measured by **publish** date, only 42% did.
Over a quarter of the "fresh" pages had first been published more than two years
earlier.

**The same cited pages, counted two ways**

| Category | Value |
|---|---|
| By last-update date | 72% counted as recent |
| By original publish date | 42% counted as recent |

*Source: Seer Interactive, 7,683 pages carrying 47,097 citations across ChatGPT, Gemini and Perplexity, March–June 2026*

Read that as an instruction and it says: **the citation went to the page that
was maintained, not the page that was new.** A two-year-old comparison page,
kept current, beat the post published last week. For anyone allocating a content
budget, those are completely different plans.

The second finding cuts against the freshness advice even harder. Pages cited in
all four months of the study had a median time-since-update of 0.47 years. Pages
that appeared in one month only — the spikes — had a median of 0.16 years.

**Pages that hold a citation are not the freshest ones**

| Category | Value |
|---|---|
| Cited in all four months | 0.47 years since last update |
| Cited in one month only | 0.16 years since last update |

*Source: Seer Interactive, March–June 2026 — median time since last update, cited pages*

The freshest pages won citations and lost them. The durable citations sat on
older, settled pages. If your RP1K is low because citations do not persist, the
answer is not a faster publishing cadence — it is fewer pages, maintained
longer.

### The engines disagree, and the disagreement is actionable

**Age of cited pages, by engine**

| Category | Updated in the last year | Updated in the last two years |
|---|---|---|
| Gemini | 78% | 90% |
| ChatGPT | 73% | 87% |
| Perplexity | 65% | 83% |

*Source: Seer Interactive, 47,097 citations, March–June 2026 — share of cited pages updated within each window*

Gemini is the most recency-biased of the three (78% within a year), Perplexity
the least (65%).[3] Combine that with your own RP1K table: if
Perplexity is your best earner *and* it is the engine least fussed about
freshness, a refresh programme aimed at Perplexity is largely wasted effort.
Spend it on the engine whose citations you are losing.

## What this does not tell you

Being clear about the limits, because a metric this easy to compute is easy to
over-read.

**It is not causal.** A page can be crawled by GPTBot and cited from Bing's
index without the crawl having caused the citation. RP1K measures a trade, not a
mechanism.

**Attribution windows distort it.** AI-referred buyers frequently take longer
than a standard 7-day last-click window allows, so a short window pushes RP1K
down for every engine — unevenly. Compute it on a 30-day window as well and
compare the two; the gap between them is itself a finding.

**The denominator includes crawls you would never have wanted.** Training
crawls, retries and 404 sweeps all land in it. That is deliberate — they cost
you the same bandwidth — but it means RP1K is a measure of the *relationship*,
not of any single page's performance.

**Small sites will have unstable numbers.** Below a few hundred referred
sessions a month, one enterprise deal moves RP1K by an order of magnitude.
Report it as a range across three months, never as a single figure.

## Start with the log you already have

Nothing above needs a new vendor. The first version of this table comes out of
your existing access logs and your existing payment data, and takes an
afternoon:

1. Pull 30 days of HTML requests, group by verified bot vendor.
2. Pull the same 30 days of sessions whose referrer or UTM maps to an AI engine.
3. Join those sessions to payments.
4. Divide. Do it again next month.

The number that matters is not the first reading. It is the **direction** across
three of them, per engine — because that is what tells you whether the trade you
are making with each platform is getting better or worse while everyone else is
still quoting a growth rate on 0.14%.

## FAQ

### What is a good crawl-to-refer ratio?

There is no universal threshold, because the ratio ignores value. The only
figures I can point at from Cloudflare's own post are Anthropic at 70,900:1 and
Mistral at 0.1:1 in one week of June 2025[2] — a six-order-of-
magnitude spread that should tell you how little a single reading is worth. A
4,000:1 ratio on visitors worth $200 each beats a 50:1 ratio on visitors worth
nothing. Compute RP1K instead and compare it against
your own organic RP1K, which is the only benchmark that reflects your prices.

### Do AI crawlers respect robots.txt?

The major declared crawlers do, and the vendors document it. OpenAI and
Perplexity both describe a delay of roughly 24 hours before a policy change is
reflected. The bigger practical problem is spoofing: `User-Agent: GPTBot` is a
string anyone can send. Verify against the published IP ranges — OpenAI,
Anthropic, Perplexity, Google and Common Crawl all publish them — before you
count a hit as real, or your denominator is inflated by traffic that was never
theirs.

### Why is my AI traffic showing up as Direct?

Because several AI surfaces send visitors without a referrer, or strip it on the
hop. That misfiling deflates the referral side of every ratio on this page. Fix
the identification first — otherwise RP1K will look worse than reality for every
engine, and worse unevenly. We covered the fixes in
[Dark AI Traffic](/blog/dark-ai-traffic-ga4).

### Should I block AI crawlers with a bad RP1K?

Only after checking the middle case. An engine can read you, cite you by name,
and send almost no clicks — that is a working brand channel with a broken
measurement story, not extraction, and blocking it costs you the mention. The
distinction is visible: run a visibility scan and see whether you are being
cited. If you are cited and RP1K is near zero, optimise the mention. If you are
neither cited nor sending traffic, blocking is a defensible use of your
bandwidth.

### How is this different from measuring AI traffic normally?

Normal AI-traffic measurement answers "how many people arrived". RP1K answers
"what did we earn for what we gave". The first is a channel report. The second
is a trade balance, and it is the one that tells you where to spend a content
budget — because content is the thing you are spending on the other side of it.

## Sources

1. [We analyzed billions of web visits: How AI is reshaping traffic channels](https://www.semrush.com/blog/traffic-channel-mix-study/) — Semrush, 2026
2. [The crawl before the fall… of referrals: understanding AI](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/) — Cloudflare, 2025
3. [Study: Content Recency](https://www.seerinteractive.com/insights/study-content-recencys-impact-on-ai-visibility-in-2026) — Seer Interactive, 2026
4. [Semrush Releases Expanded 2026 AI Visibility Index, Analyzing 126 Million AI Search Prompts](https://www.semrush.com/news/463141-semrush-releases-expanded-2026-ai-visibility-index-analyzing-126-million-ai-search-prompts/) — Semrush, 2026
5. [Google users are less likely to click on links when an AI summary appears in the results](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/) — Pew Research Center, 2025