# AI crawler tracking

> See which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and more — visit your site, which pages they read, and whether each hit is genuine. Bots don't run JavaScript, so this works from your server or edge, not the tracker.

Last updated: 2026-07-22
Canonical: https://attrifast.com/docs/ai-crawler-tracking

AI crawlers fetch your pages so AI engines can cite them — but they don't execute JavaScript, so the regular tracker never sees them. Crawler tracking closes that gap: you forward bot requests from your server or edge, and the dashboard's **AI Crawlers** panel shows which bots visited, which pages they read, how that trends over time, and how much of your site each engine has actually seen.

Every forwarded hit is also **verified**. For vendors that publish their crawl IP ranges (OpenAI, Anthropic, Perplexity, Google, Common Crawl), Attrifast checks the hit's source IP against the vendor's published list — refreshed twice daily — and labels it `verified`, `spoofed` (claims a verifiable vendor, IP outside its ranges), or `unverified` (vendor publishes nothing). Anyone can `curl -A GPTBot` your site; verification is how the numbers stay honest. The full bot list lives in the [AI crawler directory](/crawlers).

## Option A: middleware snippet (Next.js / Vercel)

One file, no dependencies. Add a `middleware.ts` at your project root (or merge into your existing one):

```ts
import { NextRequest, NextResponse } from 'next/server';

const AI_BOTS =
  /gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|google-extended|ccbot|bytespider|meta-externalagent|amazonbot|applebot-extended/i;

export function middleware(req: NextRequest) {
  const ua = req.headers.get('user-agent') || '';
  if (AI_BOTS.test(ua)) {
    // First x-forwarded-for entry = the bot's real IP (set by your host's
    // proxy). Needed for verification; hashed server-side, never stored raw.
    const ip = (req.headers.get('x-forwarded-for') || '').split(',')[0].trim();
    fetch('https://api.attrifast.com/api/collect/crawler', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({
        trackingId: 'YOUR_TRACKING_ID',
        userAgent: ua,
        path: req.nextUrl.pathname,
        ip,
        source: 'middleware',
      }),
    }).catch(() => {});
  }
  return NextResponse.next();
}
```

Replace `YOUR_TRACKING_ID` with the tracking ID from your dashboard's install panel. The call is fire-and-forget — a slow or failed POST never delays the bot's response. Non-bot traffic never leaves your server.

## Option B: nginx log shipper (any stack)

If you serve through nginx (or can read its access log), run the shipper script — it tails the log, keeps only AI-crawler lines, and POSTs them in batches. Human traffic is dropped locally and never leaves the box.

```bash
curl -O https://attrifast.com/crawler-log-shipper.sh && chmod +x crawler-log-shipper.sh
TRACKING_ID=YOUR_TRACKING_ID nohup ./crawler-log-shipper.sh \
  >/var/log/crawler-shipper.log 2>&1 &
```

Defaults: reads `/var/log/nginx/access.log` (combined format), batches 50 hits or 10 seconds, survives logrotate, and needs `gawk`. For a permanent install, the script's header includes a ready-to-paste systemd unit. Log-shipped hits carry their original timestamps, so you can also replay a historical log once to backfill.

## What gets stored (and what doesn't)

- **Stored per hit:** bot name, path, HTTP status (when available), timestamp, source (`middleware` or `log-upload`), the verification verdict, and a salted SHA-256 hash of the IP. **The raw IP is never persisted.**
- **Dropped server-side:** any request whose User-Agent isn't a known AI crawler — you can forward liberally.
- **Limits:** 500 hits per request, 100,000 per site per day. Duplicate (bot, path, second) tuples within a batch are de-duplicated automatically.

## Reading the panel

The AI Crawlers panel groups activity by vendor (OpenAI, Anthropic, Perplexity, Google) with a daily trend, a per-bot table with verified-hit percentages, and **crawl coverage** — how much of your sitemap each engine has fetched. Training crawls (GPTBot, CCBot) build the corpus your brand appears in; search and user-fetch crawls (OAI-SearchBot, ChatGPT-User, PerplexityBot) are the ones that precede cited, clickable answers — rising hits there usually lead rising AI referral traffic, which the [revenue attribution guide](/docs/revenue-attribution-guide) shows you how to turn into revenue numbers.