TRACKING

AI crawler tracking

See which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and more — visit your site, which pages they read, and whether each hit is genuine. Bots don't run JavaScript, so this works from your server or edge, not the tracker.

AI crawlers fetch your pages so AI engines can cite them — but they don't execute JavaScript, so the regular tracker never sees them. Crawler tracking closes that gap: you forward bot requests from your server or edge, and the dashboard's AI Crawlers panel shows which bots visited, which pages they read, how that trends over time, and how much of your site each engine has actually seen.

Every forwarded hit is also verified. For vendors that publish their crawl IP ranges (OpenAI, Anthropic, Perplexity, Google, Common Crawl), Attrifast checks the hit's source IP against the vendor's published list — refreshed twice daily — and labels it verified, spoofed (claims a verifiable vendor, IP outside its ranges), or unverified (vendor publishes nothing). Anyone can curl -A GPTBot your site; verification is how the numbers stay honest. The full bot list lives in the AI crawler directory.

Option A: middleware snippet (Next.js / Vercel)

One file, no dependencies. Add a middleware.ts at your project root (or merge into your existing one):

import { NextRequest, NextResponse } from 'next/server';

const AI_BOTS =
  /gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|google-extended|ccbot|bytespider|meta-externalagent|amazonbot|applebot-extended/i;

export function middleware(req: NextRequest) {
  const ua = req.headers.get('user-agent') || '';
  if (AI_BOTS.test(ua)) {
    // First x-forwarded-for entry = the bot's real IP (set by your host's
    // proxy). Needed for verification; hashed server-side, never stored raw.
    const ip = (req.headers.get('x-forwarded-for') || '').split(',')[0].trim();
    fetch('https://api.attrifast.com/api/collect/crawler', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({
        trackingId: 'YOUR_TRACKING_ID',
        userAgent: ua,
        path: req.nextUrl.pathname,
        ip,
        source: 'middleware',
      }),
    }).catch(() => {});
  }
  return NextResponse.next();
}

Replace YOUR_TRACKING_ID with the tracking ID from your dashboard's install panel. The call is fire-and-forget — a slow or failed POST never delays the bot's response. Non-bot traffic never leaves your server.

Option B: nginx log shipper (any stack)

If you serve through nginx (or can read its access log), run the shipper script — it tails the log, keeps only AI-crawler lines, and POSTs them in batches. Human traffic is dropped locally and never leaves the box.

curl -O https://attrifast.com/crawler-log-shipper.sh && chmod +x crawler-log-shipper.sh
TRACKING_ID=YOUR_TRACKING_ID nohup ./crawler-log-shipper.sh \
  >/var/log/crawler-shipper.log 2>&1 &

Defaults: reads /var/log/nginx/access.log (combined format), batches 50 hits or 10 seconds, survives logrotate, and needs gawk. For a permanent install, the script's header includes a ready-to-paste systemd unit. Log-shipped hits carry their original timestamps, so you can also replay a historical log once to backfill.

What gets stored (and what doesn't)

• Stored per hit: bot name, path, HTTP status (when available), timestamp, source (middleware or log-upload), the verification verdict, and a salted SHA-256 hash of the IP. The raw IP is never persisted.
• Dropped server-side: any request whose User-Agent isn't a known AI crawler — you can forward liberally.
• Limits: 500 hits per request, 100,000 per site per day. Duplicate (bot, path, second) tuples within a batch are de-duplicated automatically.

Reading the panel

The AI Crawlers panel groups activity by vendor (OpenAI, Anthropic, Perplexity, Google) with a daily trend, a per-bot table with verified-hit percentages, and crawl coverage — how much of your sitemap each engine has fetched. Training crawls (GPTBot, CCBot) build the corpus your brand appears in; search and user-fetch crawls (OAI-SearchBot, ChatGPT-User, PerplexityBot) are the ones that precede cited, clickable answers — rising hits there usually lead rising AI referral traffic, which the revenue attribution guide shows you how to turn into revenue numbers.