AI Traffic Is a Trade, Not a Channel: How to Compute What Your Crawl Budget Actually Earns

Everyone is counting one side of a two-sided exchange

Here is the state of AI-traffic measurement in 2026, and it is genuinely good work. Semrush ran the numbers across more than 50,000 websites and 17 industries, worldwide, desktop and mobile, for all of 2025. AI traffic grew 66.02% — from 462 million to 767 million monthly visits — and still landed at 0.14% of total visits against 64.69% direct and 16.04% organic.^[1]

Source: Semrush Traffic & Market Toolkit — 50,000+ websites, 17 industries, worldwide desktop + mobile, Jan–Dec 2025

That study is careful and its numbers are probably the best public estimate we have. My argument is not with the data. It is with the shape of the question.

Every one of these studies — Semrush's, Similarweb's, the ones built on clickstream panels — measures arrivals. Visits in. That framing treats AI like a channel: a pipe that delivers people, to be sized against other pipes.

Growth rates make the framing worse, not better. Here is AI's 66% against the other channels in the same study:

Source: Semrush channel study, Jan–Dec 2025 — 50,000+ websites

Paid search grew faster than AI last year. Nobody wrote a think-piece about it, because paid search's base is enormous and everyone already knows what to do with it. AI's 66% is only remarkable because the base is 0.14%, and a percentage computed on a base that small can move an order of magnitude on noise.

The real problem is different, though, and it is structural: a visit count can only ever measure what you were given. It cannot measure what you gave.

The other half of the ledger

Cloudflare sits in front of a large share of the web, which means they can see both directions at once. They defined the metric that makes the exchange visible:

dividing the total number of requests from relevant user agents associated with a given search or AI platform where the response was of Content-type: text/html by the total number of requests for HTML content where the Referer header contained a hostname associated with a given search or AI platform^[2]

In plain terms: how many pages did this platform take, per human it sent back. For the week of 19–26 June 2025 Cloudflare published Anthropic at 70,900:1 and Mistral at 0.1:1 — a spread of nearly six orders of magnitude between two platforms in the same week.^[2]

Two things about those numbers, because they get quoted badly.

They move, a lot. Cloudflare says so explicitly, and the figures you will see in secondary coverage — Anthropic at 4,580:1, OpenAI at 848:1 — are later readings from Cloudflare Radar, not from that post. Radar publishes the ratio live. Take the current number from there, not from an article, including this one.

A high ratio is not automatically theft. A crawler that fetches your pages for training will never send a referral, by construction, because the model answers in the chat window. That is a real complaint, but it is a complaint about the business model, not a measurement error. The measurement error is assuming the ratio alone tells you what to do.

Why the ratio still isn't a decision

Take two sites with an identical crawl-to-refer ratio of 4,000:1.

The first sells a $19/month tool. Its AI referrals convert at roughly the site average and each one is worth about what any other visitor is worth. At 4,000:1 the trade is bad and getting worse, and the honest move is to stop optimising for it.

The second sells $40,000 implementation contracts. Its AI referrals arrive having already read a comparison page and a docs page, and close at four times the rate of its organic traffic. At the same 4,000:1, the trade is one of the best it has.

Same ratio. Opposite decisions. The ratio is a rate of exchange with no prices attached — and you cannot evaluate an exchange rate without knowing what you are buying.

RP1K: revenue per 1,000 AI crawls

The fix is to finish the fraction. Put money in the numerator:

                 revenue attributed to engine E, period P
RP1K(E) = 1000 × ─────────────────────────────────────────
                  HTML crawls by engine E's bots, period P

Three inputs, all of which you can get today:

1. Crawls by engine. Your server or CDN logs, filtered to the bot user-agents and — this matters — verified against the vendor's published IP ranges. Anyone can send User-Agent: GPTBot. OpenAI, Anthropic, Perplexity, Google and Common Crawl all publish the ranges their bots crawl from, and an unverified hit belongs in a different bucket than a verified one.
2. Referred sessions by engine. Referrer hostname plus the UTM values each engine appends. This is the step most analytics get wrong, which is why AI traffic so often lands in Direct.
3. Revenue attributed to those sessions. Your payment processor, joined to the session that preceded it.

A worked example, on a 40-page B2B site over 30 days. These are illustrative numbers — substitute your own:

Source: Illustrative calculation from the formula in this article — substitute your own crawler log, referral and Stripe figures

The spread here is roughly 300×, and it inverts the crawl volume: the bot that took the most pages returned the least money. That inversion is the entire point of computing this. A crawl-budget conversation based on hit counts would have concluded that Anthropic is the most important AI relationship on the site. It is the least.

Two properties make RP1K worth the trouble:

• It is denominated in the thing you actually spend. Crawl budget is real cost — bandwidth, origin CPU, and on some stacks a per-request bill.
• It survives the zero. If an engine sends no revenue, RP1K is 0, which is a legible answer. A crawl-to-refer ratio with zero referrals is a division by zero, and gets quietly dropped from summaries.

Reading the answer

That last row is a real option and it is worth saying plainly: robots.txt directives are honoured by the major AI crawlers, and OpenAI and Perplexity both document a delay of roughly a day before a policy change takes effect. Blocking is not free — you also lose the citations — which is exactly why you want the number before you decide.

Raising the numerator is not "publish more"

The instinct, once RP1K is low, is to publish. The best evidence available says that is the wrong lever.

Seer Interactive analysed 7,683 pages carrying 47,097 citations across ChatGPT, Gemini and Perplexity between March and June 2026, dating each page from its structured signals — schema, sitemaps, headers.^[3] Their headline finding is that 75% of cited pages were updated within the last year.

The finding underneath it is the useful one. Measured by last-update date, 72% of cited pages looked recent. Measured by publish date, only 42% did. Over a quarter of the "fresh" pages had first been published more than two years earlier.

Source: Seer Interactive, 7,683 pages carrying 47,097 citations across ChatGPT, Gemini and Perplexity, March–June 2026

Read that as an instruction and it says: the citation went to the page that was maintained, not the page that was new. A two-year-old comparison page, kept current, beat the post published last week. For anyone allocating a content budget, those are completely different plans.

The second finding cuts against the freshness advice even harder. Pages cited in all four months of the study had a median time-since-update of 0.47 years. Pages that appeared in one month only — the spikes — had a median of 0.16 years.

Source: Seer Interactive, March–June 2026 — median time since last update, cited pages

The freshest pages won citations and lost them. The durable citations sat on older, settled pages. If your RP1K is low because citations do not persist, the answer is not a faster publishing cadence — it is fewer pages, maintained longer.

The engines disagree, and the disagreement is actionable

Source: Seer Interactive, 47,097 citations, March–June 2026 — share of cited pages updated within each window

Gemini is the most recency-biased of the three (78% within a year), Perplexity the least (65%).^[3] Combine that with your own RP1K table: if Perplexity is your best earner and it is the engine least fussed about freshness, a refresh programme aimed at Perplexity is largely wasted effort. Spend it on the engine whose citations you are losing.

What this does not tell you

Being clear about the limits, because a metric this easy to compute is easy to over-read.

It is not causal. A page can be crawled by GPTBot and cited from Bing's index without the crawl having caused the citation. RP1K measures a trade, not a mechanism.

Attribution windows distort it. AI-referred buyers frequently take longer than a standard 7-day last-click window allows, so a short window pushes RP1K down for every engine — unevenly. Compute it on a 30-day window as well and compare the two; the gap between them is itself a finding.

The denominator includes crawls you would never have wanted. Training crawls, retries and 404 sweeps all land in it. That is deliberate — they cost you the same bandwidth — but it means RP1K is a measure of the relationship, not of any single page's performance.

Small sites will have unstable numbers. Below a few hundred referred sessions a month, one enterprise deal moves RP1K by an order of magnitude. Report it as a range across three months, never as a single figure.

Start with the log you already have

Nothing above needs a new vendor. The first version of this table comes out of your existing access logs and your existing payment data, and takes an afternoon:

1. Pull 30 days of HTML requests, group by verified bot vendor.
2. Pull the same 30 days of sessions whose referrer or UTM maps to an AI engine.
3. Join those sessions to payments.
4. Divide. Do it again next month.

The number that matters is not the first reading. It is the direction across three of them, per engine — because that is what tells you whether the trade you are making with each platform is getting better or worse while everyone else is still quoting a growth rate on 0.14%.

See which AI engines actually send you paying customers

Attrifast splits ChatGPT, Perplexity, Claude and Gemini into their own revenue lines — joined to real Stripe payments, not estimates.

• ✓One script tag and a Stripe key — live in minutes
• ✓Cookieless, so no consent banner for analytics
• ✓Every AI referral matched to the payment it produced

Start your free trial →

7-day free trial · $0 due today · then $9.99/mo · cancel anytime

FAQ

What is a good crawl-to-refer ratio?

There is no universal threshold, because the ratio ignores value. The only figures I can point at from Cloudflare's own post are Anthropic at 70,900:1 and Mistral at 0.1:1 in one week of June 2025^[2] — a six-order-of- magnitude spread that should tell you how little a single reading is worth. A 4,000:1 ratio on visitors worth $200 each beats a 50:1 ratio on visitors worth nothing. Compute RP1K instead and compare it against your own organic RP1K, which is the only benchmark that reflects your prices.

Do AI crawlers respect robots.txt?

The major declared crawlers do, and the vendors document it. OpenAI and Perplexity both describe a delay of roughly 24 hours before a policy change is reflected. The bigger practical problem is spoofing: User-Agent: GPTBot is a string anyone can send. Verify against the published IP ranges — OpenAI, Anthropic, Perplexity, Google and Common Crawl all publish them — before you count a hit as real, or your denominator is inflated by traffic that was never theirs.

Why is my AI traffic showing up as Direct?

Because several AI surfaces send visitors without a referrer, or strip it on the hop. That misfiling deflates the referral side of every ratio on this page. Fix the identification first — otherwise RP1K will look worse than reality for every engine, and worse unevenly. We covered the fixes in Dark AI Traffic.

Should I block AI crawlers with a bad RP1K?

Only after checking the middle case. An engine can read you, cite you by name, and send almost no clicks — that is a working brand channel with a broken measurement story, not extraction, and blocking it costs you the mention. The distinction is visible: run a visibility scan and see whether you are being cited. If you are cited and RP1K is near zero, optimise the mention. If you are neither cited nor sending traffic, blocking is a defensible use of your bandwidth.

How is this different from measuring AI traffic normally?

Normal AI-traffic measurement answers "how many people arrived". RP1K answers "what did we earn for what we gave". The first is a channel report. The second is a trade balance, and it is the one that tells you where to spend a content budget — because content is the thing you are spending on the other side of it.