Why Does ChatGPT Give Different Answers to the Same Question Every Time?

ChatGPT gives different answers to the same question because the model samples its output probabilistically, because your request gets batched with strangers' requests on a GPU whose arithmetic is not batch-invariant, and because the product layer on top (memory, personalization, live web retrieval, silently updated system prompts) changes what the model even sees. None of that makes your AI visibility unmeasurable. It makes a single check worthless and a repeated, controlled sample the only honest unit of measurement. Stop asking "do I show up in ChatGPT." Start asking "in what share of answers do I show up, over how many asks, across which engines."

Most founders discover this the annoying way. You type "best [your category] tool for [use case]" on Monday and see three competitors. You type it again Friday and see four different ones, one of which is you, in fifth place. Your colleague runs it and gets a lineup neither of you has seen. The natural conclusion is that the whole channel is noise. That conclusion is wrong, and it is expensive.

What actually causes the variation?

Four separate mechanisms stack, and they are worth separating because only some of them are yours to control.

Sampling. The model predicts a probability distribution over the next token and then draws from it. Temperature controls how flat that distribution is. Lower temperature concentrates on the likeliest token, which reduces variance but does not eliminate it.

Batch-size variance. This is the one most people miss. Even at temperature zero, outputs are not reproducible, because inference kernels are not batch-invariant: your prompt is processed alongside however many other requests hit the server in that instant, and floating-point operations are non-associative, so (a+b)+c does not always equal a+(b+c). Different server load produces slightly different numbers, and tiny numerical differences cascade into a different word, then a different sentence, then a different shortlist. Your answer depends partly on how busy the service was.

Personalization. ChatGPT's memory, your chat history, your account context, and your inferred location all shape the response. You and your colleague are not running the same test. You are running two different tests that happen to share a sentence.

The moving parts above the model. System prompts get updated without notice. Models get silently swapped or versioned. Retrieval pulls live pages that changed yesterday. Mixture-of-experts routing adds another layer of run-to-run variation.

Sort those into two piles. Sampling and batch variance are physics: you cannot remove them, you can only average over them. Personalization you can mostly neutralize with a clean testing setup. The platform layer you can neither control nor average away, which is why you watch it as a trend line and date every reading you take.

Does variability mean AI visibility cannot be tracked?

It means a snapshot is not evidence. One ask is one draw from a distribution, which is roughly as informative as one coin flip is about a coin.

The fix is the same one pollsters figured out decades ago: do not try to eliminate the randomness, sample it. Ask the same question repeatedly over time, hold the conditions as steady as you can, and report a rate with the sample size attached. "We were named in 6 of the last 20 ChatGPT answers to this question, average position 3" is a real measurement. "I checked and we weren't there" is an anecdote.

What makes a mention rate trustworthy?

Five pieces of discipline separate a number you can show a board from a number that will embarrass you.

• Fix the question text. The wording stays stable across runs so you are measuring the engine's movement, not your own rephrasing.
• Control the personalization you cannot remove. Prefix each ask with a consistent buyer persona, and a location line when location matters, so answers reflect what a real prospect in your market would see instead of what your logged-in account would see. The persona must never name your own company, or it biases the exact number it is measuring.
• Sample over time, not all at once. Clark runs every day and rotates through your questions, so each one comes back around roughly weekly, with no question re-asked sooner than seven days after its last check. That spacing is deliberate: twenty asks in one afternoon measures one afternoon.
• Separate a mention from a citation. If your domain appears in the source links but your name never appears in the answer text, that is a citation, not a mention. Conflating the two is how most AI-visibility tools over-report.
• Keep the raw answers. Every analyzed question should store the full answer per platform, which model produced it, and the source links the engine leaned on, filterable by the answers that named you. Verifiability is the only real counter to "how do I know this is real?"

Run across all four engines, too. ChatGPT, Claude, Gemini, and Perplexity have different variability profiles and different source habits, and a brand can be steady in one and invisible in another.

Why we refuse to publish a single "AI visibility score"

A composite score is the most comfortable way to lie about a noisy system. Blend mention rate, position, sentiment, and citations into one number, and you get something that moves smoothly, feels authoritative, and hides exactly the volatility you were hired to understand. Nobody can audit it, and nobody can act on it.

So there is no score in our product, on purpose. There is share of AI answers that mention you, average position when you are mentioned, how many answers were analyzed, and how many companies showed up. Four concrete numbers, each traceable back to an answer you can read.

What should a growth leader do with a noisy number?

Report ranges and trends, never single reads. Mention rate climbing from 0 to 30 percent over six weeks is a signal. A two-point wobble between Tuesday and Thursday is weather.

Set thresholds that reflect real movement. A 15-point swing in mention rate, or a two-place change in average position, is worth an alert and a look at which competitor took the ground. Anything smaller is the batch scheduler talking.

And treat visibility as the input, not the outcome. Mention rate tells you whether buyers are being pointed at you. Only attribution tells you whether it paid. Everyone shows you rank. We show you revenue.

The loop that actually matters runs past the mention. An engine names you in an answer, the buyer clicks the post that earned the citation, that session lands in your analytics tagged to the AI source it came from, and weeks later it closes as a deal with a number on it. Mention rate is the first step of four. A brand that tracks only step one will keep arguing about whether Tuesday's answer counted.

If you want to see the distribution instead of one draw from it, run a free AI visibility check against your own buyer questions, or read more on how an answer engine decides which brand to cite. One snapshot is a start. Clark runs it continuously.