We Re-Ran Our Own AI Visibility Scan After Finding a Bug In It: The Raw Data (2026)

Part of the AI Search Hub, AEO Hub, and the generative engine optimization guide.

Most posts about AI visibility tools are written from the outside, by someone comparing dashboards. This one is written from the inside, and it now has an uncomfortable second act. I build Attrifast, and one of the things it does is run an AI Visibility scan: it asks the major AI engines a set of buyer-stage questions about your category, parses each answer, and reports whether you were cited, how you compare to competitors, and which sources the engines pulled from instead of you.

In July I pointed it at my own domain and published the raw output, including the parts that made us look bad. Two weeks later, while debugging another account whose scan also came back at zero, I found that our scanner had been systematically under-counting citations — ours, our competitors', everyone's. This post now carries both runs: the original numbers, the bug, and the re-run that corrected them. For the category-level buyer's guide, see AI visibility tools: what they are and how to pick one; this is the case study that shows one running end to end, including the part where it was wrong.

The correction, up front

Here is what was wrong with the July run, in the order it mattered.

Two of four engines never searched. Claude and ChatGPT were offered a web-search tool rather than forced to use one. Left to their own judgment on a conceptual prompt, both answer from parametric memory and retrieve nothing. An answer with no retrieval can only name brands the model already memorised — which is precisely the bias a visibility scan exists to measure, not to inherit. Probed against the live gateway on 3 August, one of the real scan prompts returned 0 sources on automatic tool choice and 10 on forced for Claude; ChatGPT went 0 to 8 on another.

Every Gemini citation was discarded. Google's grounding does not hand back real URLs. It returns opaque vertexaisearch.cloud.google.com redirect links, and our pipeline filtered that host in three places downstream as citation plumbing. Gemini contributed 115 source URLs to the re-run. In July, all of its equivalents were dropped on the floor.

The prompts themselves were generated blind. Auto-generation read the domain, the site name and the top pageview paths — never the page content — so it inferred a category rather than reading one. It also carried an explicit rule against generating questions that name the site itself, which deletes the single highest-signal prompt shape there is.

None of this was a detection bug. Searching all forty stored July answers for our brand string returns zero hits, exactly as the score said. The engines genuinely never named us. But the evidence base under that conclusion was roughly two-thirds of what it should have been.

What we re-ran, and how

To keep the comparison clean, the re-run changed exactly one thing: the pipeline. Same domain, same ten prompts, same four engines, same day.

Gemini stays on automatic deliberately — forcing its tool measurably returns fewer sources — and Perplexity has no tool to force. One methodological caveat worth stating plainly: the re-run was executed through a script mirroring the shipped engine configuration, and its redirect resolver is a simplified version of the one in the product. It resolved 95 of 115 Gemini redirects; the remaining 20 stayed opaque and are excluded from the domain tables below rather than counted as a domain in their own right. Treat that 83% as this run's figure, not a product specification.

The prompts we asked

The ten prompts our scanner generated were broad, category-level questions:

Hold that list in mind. It explains everything about the result.

The headline number: 0/100, twice

Across 40 answers on the fixed pipeline, our domain was cited zero times. Score: 0/100 — unchanged from July, now with half again as much evidence behind it.

I find the second zero more useful than the first. A zero produced by a pipeline that captured 320 sources is ambiguous; you cannot separate a real absence from a retrieval failure. A zero produced by a pipeline that captured 470 sources across 252 domains, with every engine retrieving and no errored calls, is a finding. The engines are not failing to see us. On these questions, they are choosing other names.

Source: Attrifast AI Visibility re-run, attrifast.com, 3 August 2026

The per-engine spread deserves a moment. Perplexity alone contributed 196 of the 470 sources — it is built as a citation engine and behaves like one. Gemini returned 115, Claude 99, and ChatGPT just 60 even with search forced. Only two answers in the entire run came back with no sources at all, one from ChatGPT and one from Gemini. If your visibility tool reports a single blended score, it is averaging four engines whose retrieval volume differs by more than 3×.

The 75% counter-test: your score is mostly your prompt set

Here is the part that reframes everything above. On the same day, through the same fixed pipeline, I ran three commercial-intent prompts — the questions our actual buyers type — against all four engines.

Nothing changed except the question. Broad category prompts: 0%. Commercial-intent prompts: 75%. A 75-point swing, same domain, same engines, same hour.

Source: Attrifast AI Visibility re-run, attrifast.com, 3 August 2026

Perplexity and Gemini cited us on every commercial prompt. Claude on two of three. ChatGPT on one of three, consistent with it being the stingiest retriever in the run.

The takeaway for anyone measuring a young brand: broad category prompts flatter incumbents and bury challengers. On questions as generic as "what are the top AEO tools for 2026," the engines reach for the brands with the most third-party coverage — Semrush, the established SEO suites, the names on every listicle. If your prompt set is all "best X tool" questions, your score is largely measuring your competitors' backlink history. That is a legitimate thing to measure. It is simply not the same thing as whether AI recommends you to a buyer.

This has a pricing consequence buyers should notice. Most monitoring tools meter you by tracked prompt, and 25 prompts at the entry tier is typical. How those 25 get chosen will move your reported score further than anything you do to your own site this quarter.

Share-of-voice: the correction hit everyone

This is where a scan earns its keep, and where the re-run changed the story most. The score tells you about you; share-of-voice tells you about the race.

Source: Attrifast AI Visibility scans, attrifast.com, July 2026 and 3 August 2026

The numbers, unrounded:

Peec.ai went from 3% to 25%, an eightfold correction. In July it looked like a marginal presence cited by a single engine; it is in fact cited by three and running neck-and-neck with Otterly. Otterly more than doubled and now appears across all four.

That is the honest cost of the bug, and it is worth being precise about who it hurt. Under-measurement did not flatter us — we were zero either way. It flattered our reading of the race. A founder looking at the July chart would have concluded there was one competitor worth worrying about and a comfortable gap behind them. The corrected chart says two competitors are established across nearly every engine, and the gap behind them is us. For the full method behind this, see how to analyze your competitors' AI visibility and AI share of voice in 2026.

The most useful output: the domains AI reads instead of you

If share-of-voice is the diagnosis, the cited-sources list is the prescription. Across the 40 answers the engines pulled 470 source URLs from 252 distinct domains, and the scanner ranks them by how many of your prompts they showed up for. This is, functionally, the list of places you need to get mentioned.

Source: Attrifast AI Visibility re-run, attrifast.com, 3 August 2026

The top of the list, with July's figure alongside where the domain appeared in both runs:

Two things stand out. Semrush is the only domain unchanged at 70% across both runs, and the only one cited by all four engines. Everything else moved when retrieval was fixed; Semrush was already saturating the category conversation so completely that a two-thirds increase in captured sources did not raise its coverage at all. If you want one number for what category dominance looks like to an AI engine, that is it.

Second, look at what these are: listicles, forum threads, review sites, and comparison posts — third-party pages, not the vendors' own homepages. Reddit is now the single most-cited domain in our category at 80% prompt coverage, and it gets there through one engine doing the citing. This is the most important thing an AI visibility scan taught me about my own business: getting cited by AI is mostly an off-site problem. The engines trust the sites that already round up "best X tools," so the path to citation runs through earning a spot on those roundups, seeding honest comparisons, and showing up in the Reddit and LinkedIn conversations where buyers actually ask. That is the co-citation game, and it is covered in depth in AI citations vs backlinks. A monitoring tool that stops at "you scored 0" is useless; one that hands you the ranked list of where to go is a work plan.

What this says about every score in the category

I am not going to pretend this failure was unique to us. It is worth generalising, because the failure modes are structural and completely invisible from a dashboard.

Every AI visibility product sits on the same three decisions, and none of them are usually disclosed:

1. Is web search forced or offered? If offered, you are partly measuring the model's training-data memory of your brand. That number moves on a model release, not on anything you did.
2. What happens to Gemini's redirect citations? They arrive opaque. Resolving them costs a network round-trip per URL, so the cheap implementation is to drop them — and dropping them silently removes an entire engine's evidence from your score.
3. Who chose the prompts, and how? Our own generator inferred a category from URL paths without reading a page. A prompt set assembled that way can measure a company against a category it is not in.

The practical version, if you are evaluating tools: ask for the per-engine source count on your own scan. Not the score — the raw number of cited URLs each engine returned. An engine reporting near-zero sources while the others return dozens is not evidence that you are invisible. It is evidence that engine never searched.

The number no scan on this list can give you: revenue

Here is the honest limit of everything above, including our own scan. A visibility score tells you whether AI cited you. It cannot tell you whether that citation paid you. Every tool in this category shares the same blind spot, and it is structural, not a feature gap.

The reason is plumbing. AI clients strip the Referer header, so when ChatGPT sends someone to your site, the visit shows up in GA4 as Direct/(none) — indistinguishable from someone typing your URL. Independent measurement and our own data put 65–82% of ChatGPT visits in that Direct bucket [1][2]. So any "AI revenue" figure derived from a GA4 integration is built on a systematic undercount, and any monitoring tool quoting one is estimating.

The only AI revenue number you can defend is one tied to a transaction: an AI-referred session traced through to a paid Stripe invoice or Shopify order. That requires a first-party attribution layer that captures the AI referral before the header is stripped and joins it to the payment webhook — which is the actual job Attrifast does, and why we treat the visibility scan as the top of the funnel, not the whole thing. A high citation share with flat revenue-per-visitor usually means you are being cited for informational queries that do not convert — a false positive a score-only tool will never catch [3]. For the full argument, see does GEO actually drive revenue and AI visibility metrics and KPIs.

What I changed after reading my own data

A scan is only worth running if it changes what you do. Here is the July list with what actually happened to each item.

1. Fix the scanner before trusting the prompt set. Done, and it turned out to be the bigger problem. Search is now forced on Claude and ChatGPT, Gemini's redirect citations are resolved to real domains before matching, and prompt generation reads actual page content instead of inferring a category from URL paths.
2. Fix the prompt set. In progress. The re-run proves the lever is real — 0% to 75% on question choice alone — so the tracked set is moving to roughly half commercial-intent, some branded comparisons, and a few broad prompts kept deliberately as a category barometer rather than as the score.
3. Work the cited-sources list, top-down. Unchanged, and now better targeted. Reddit at 80%, Semrush at 70% across all four engines, the AEO listicles, the LinkedIn threads — that ranked list is a placement backlog, and it is more actionable than any generic "build backlinks" advice because the engines told me exactly which domains they already trust for my category.
4. Watch the trend, not the day. With a caveat I did not appreciate in July: a trend line is only a trend if the pipeline underneath it holds still. Ours did not, so the honest move is to treat 3 August as a new baseline rather than pretend the two runs are points on one curve.
5. Keep measuring revenue separately. The score is a leading indicator. The lagging indicator that actually matters — did AI-referred visitors pay — lives in the attribution layer, and that is the number I report to myself weekly.

If you want to run the same scan on your own category and see your share-of-voice, your competitor gap, and your own version of that placement list, that is a few clicks inside Attrifast.

FAQ

What does an AI visibility scan actually measure?

An AI visibility scan sends a set of buyer-stage prompts to multiple AI engines — in our case ChatGPT, Claude, Gemini, and Perplexity — parses each generated answer, and records whether your domain was cited, which competitors were cited, and which external sources the engine pulled from. The core outputs are a 0–100 visibility score (the share of answers that cited you, excluding failed calls), a per-engine breakdown, competitor share-of-voice, and a ranked list of the domains the engines cited instead of you. In our own re-run, 40 answers across 4 engines produced 470 cited sources across 252 distinct domains, a 0% score for us, and a 28% share-of-voice for one competitor — all from a single run.

Why did your own AI visibility score come back as zero?

Because the ten prompts our scanner auto-generated were broad category questions like 'best tool for tracking AI search visibility' and 'top AEO tools for 2026', and on questions that generic the four engines named larger, older brands, not us. The zero survived a full re-run on a fixed pipeline that captured 47% more sources, so it is a real result rather than a measurement failure. But it measures the prompt set, not the brand: on three commercial-intent prompts run the same day through the same pipeline, 9 of 12 answers cited attrifast.com — a 75% score against 0% on the broad set.

Can an AI visibility tool under-report your score?

Yes, and ours did. Two of the four engines were answering from parametric memory instead of searching, because their web-search tool was offered rather than forced — an answer with no retrieval can only name brands the model already memorised. Separately, every Gemini citation arrived as an opaque vertexaisearch.cloud.google.com redirect and was being discarded downstream as plumbing. Fixing both raised captured sources from 320 to 470 on the identical prompt set. Before trusting any score, ask how many sources each engine returned and whether search was forced; a suspiciously clean zero is often a retrieval bug.

What is AI share-of-voice and how is it calculated?

AI share-of-voice is the percentage of answers in a scan that cite a given brand, calculated as that brand's cited answers divided by the total valid (non-errored) answers. In our 40-answer re-run, Otterly.ai was cited in 11 answers for a 28% share-of-voice and Peec.ai in 10 for 25%, while DataFast and Attrifast both sat at 0%. The same prompts measured on our pre-fix pipeline gave Otterly 13% and Peec 3% — the under-measurement was hitting every brand in the scan, not just ours.

Which sources do AI engines cite when they answer category questions?

In our re-run the top cited domains were reddit.com (appeared for 80% of prompts), semrush.com (70%), youtube.com (70%), linkedin.com (60%), and a long tail of SEO and review sites including blog.hubspot.com, techradar.com, rankability.com, seranking.com and useomnia.com at 50% each. This matters because getting cited by AI is often less about your own page and more about earning a mention on the third-party listicles, forums, and review sites the engines already trust. A visibility scan's cited-sources list doubles as a placement to-do list.

Do AI visibility scores predict revenue?

No. A visibility score tells you whether AI engines mention you; it says nothing about whether the resulting visit paid you. This is the structural blind spot of every monitoring-only tool: AI clients strip the Referer header, so most ChatGPT and Perplexity visits land in GA4's Direct/(none) bucket and are never attributed to the engine that sent them. The only revenue number you can trust is one tied to a transaction — an AI-referred session traced to a paid Stripe invoice or Shopify order. That attribution layer is separate from, and more valuable than, the visibility score itself.

How often should you run an AI visibility scan?

Because a generative answer is probabilistic — the same prompt can return different answers on different runs — a single scan is a snapshot, not a trend. For a stable read you want to scan on a schedule and watch the direction of your score and share-of-voice over weeks, not obsess over one number. We rate-limit scans to protect against runaway API cost, and pace the calls inside each scan so we stay under the AI Gateway's throughput limit. The signal you act on is the trend line and the competitor gap, not any single day's score.