AI visibility tracking: the number is easy to get wrong
Every week another tool launches promising to score your visibility in AI answers, and most of them will hand you a number that is wrong — not by a little, and not in a way you can see. I spent years building measurement into marketing P&Ls, and the rule I trust most is simple: a number you cannot trust is worse than no number, because it moves budget with false confidence.
AI visibility is the most trap-laden thing I have measured. The traps do not throw errors; they return clean figures that read as truth. Here are the four that matter, and the discipline that survives each one.
Trap 1 — the number moves when nothing changes
Ask the same questions twice on the same morning, with everything identical, and in our capture logs about 4% of them disagree on whether an AI answer even appears — a day apart, 5 to 7%. Google decides at serve time — experiment bucket, load, the user's own state. Nothing about you changed; the number did.
This is the difference between weather and climate, and most trackers report the weather. A dashboard announcing "AI visibility up 12% this week" is, more often than not, inside the noise.
The discipline: presence is a rate over repeated captures, never a property of a single keyword. Measure at least three captures a side, and treat a move as real only when it clears the wobble within each side — in our logs about 4% on a same-morning re-ask, 5 to 7% a day apart. State that band beside the number every time. And prefer domain-level citation share for trend, because it averages over many answers and holds far steadier than presence.
Trap 2 — the surfaces are not one surface
Google AI Overviews, ChatGPT and Gemini do not cite the same world, and pooling them does not average the truth — it reverses it.
They cite different worlds. In a UK health market:
- Google AI Overviews lead with YouTube — the single most-cited domain, and close to invisible in the chatbots;
- ChatGPT leans on Reddit — well over half of its community citations there;
- Gemini is the only one of the three that reaches for X;
- People Also Ask is a fourth surface again, and the near-universal one — it appears on almost every result that carries an AI Overview, and on plenty that do not.
A channel that looks dead on one surface leads on another. So a single "AI visibility" score, pooled across surfaces, is not a summary — it is a mistake with a decimal point.
The discipline: measure each surface on its own column and never pool them. A number without a surface named beside it should not leave the building.
Trap 3 — cited is not the same as consulted
In our health-market capture sets, roughly half the sources under an AI answer were not cited to the reader at all — they are grounding the model quietly reached for, not links the answer pointed anyone towards. Counting the two together doubles your apparent visibility and hides whether you actually reached a human.
There is a related tax: the platforms cite themselves. On the Gemini surface, Google's own units were the top "earned" source until we excluded them — leave them in and you are benchmarking against Google, not your rivals.
The discipline: separate what the answer pointed the reader to from what the model merely consulted, and strip the platform's self-citations before you rank anyone. And count on the answer grain — did the model name this domain in this answer — not the URL grain, which simply rewards whoever deep-links the most pages.
Trap 4 — your question list is not the market
This is the quiet one, and the most common. Hand-pick fifty questions you think matter, run them, and you have measured your own question list — not your market. Weight it toward your own brand name and the earned number flatters you, because a model knowing your own story is not the same as it recommending you on a neutral question.
The discipline: the question set is the instrument, so it has to be built like one — enumerated from the market's own structure rather than chosen by hand, weighted so the generic (non-brand) questions carry the earned metric, and then frozen. The moment you edit the questions between runs, the time series breaks and you are back to measuring the instrument. Earned visibility is share on the generic questions; the branded ones answer a different question entirely.
Some take-aways
A trustworthy AI-visibility number is not hard because the maths is hard. It is hard because four things go wrong quietly:
- the number wobbles — about 4% on a re-ask, with nothing changed;
- the channels disagree — never pool them;
- half the sources never reached the reader;
- and the easy way to build the test is the one that measures yourself.
Get those four right and the number means something. Get any one wrong and it reads exactly the same — which is the danger.
None of this is an argument against tracking — quite the opposite. It is an argument for tracking it properly, on the discipline above, so the movement you report is real. That is the first half of the picture. The second is why a source gets cited in the first place — which is where your Google rank turns out to matter far more than "SEO is dead" would have you believe.
If you want the framework this measurement feeds, start with the AI-visibility ladder; for the whole-market view, see what market intelligence actually is, or explore the platform.
Method note: figures are Theia measurements across a live UK private-healthcare market, reported as a rate over repeated captures. Surface-level results are never pooled. We publish the result and the reasoning, never the method that produces them.
Frequently asked
- Are AI visibility tools accurate?
- Most mislead in ways you can't see. They pool ChatGPT, Gemini and Google's AI Overviews into one score when the three cite different worlds; they report a presence number that flips about 4% on a same-morning re-ask as if it were signal; and they count sources the model only consulted, never the ones the answer actually showed the reader. An accurate read names the surface, treats presence as a rate over repeated captures, and separates what the answer pointed to from what it merely used.
- Why do my AI visibility numbers keep changing?
- Because presence is decided at serve time. Ask the same questions twice on the same morning, everything identical, and about 4% of them flip on whether an AI answer even appears — a day apart, 5 to 7%. Treat presence as a rate over repeated captures, never a fixed property: take at least three captures a side and count a move only when it clears the spread within each side.
- Which AI surface should I measure — ChatGPT, Gemini or Google AI Overviews?
- All three, separately, and never pooled — they cite different worlds. In a UK health market, YouTube is the single most-cited domain in Google's AI Overviews and close to invisible in the chatbots, while Reddit supplies well over half of ChatGPT's community citations. A channel that looks dead on one surface leads on another, so a number without a surface named beside it is meaningless.