Contact

Menu

Contact
, , ,

Why AI Gives Different Answers to the Same Question

AI assistants are built to vary their answers. The makers say so. Here is why one answer about your store proves little, and how to read the pattern.

Feature visual: Why AI Gives Different Answers to the Same Question

Have a question about your own website? We will get back to you.

Discuss your website

Ask ChatGPT the same question twice, and you can get two different answers. That is by design. The companies that build these assistants say so in their own documentation, so no single answer is the real one. One answer might say your store was named, or a competitor was named instead, and the next can give another.

OpenAI’s help centre says its model predicts “the next most likely word when generating a response, one word at a time”. It goes on to say “there is an inherent element of randomness in how the model responds. As a result, the same question may yield different answers across different queries.”

Table of five things that change an AI assistant's answer to the same question: sampling, live web search, memory and settings, location and wording, and model updates, with the companies whose own pages say so.

Variation is built into how these assistants write

When an assistant writes an answer, it has more than one good candidate for each next word. Valliappa Lakshmanan explains in Generative AI Design Patterns that a model which always took the single most likely word would write repetitive text. So models use “a sampling strategy in which all the possible continuation words have some nonzero probability of being selected”. In plain words, sampling means the model draws its next word from several good candidates, with some chance involved. Sinan Ozdemir describes the result in Building Agentic AI. With default settings, “you can often expect a different answer every time”, and while the gist may hold, the exact words change.

Some models give developers a setting called temperature, which Anthropic defines as the “amount of randomness injected into the response”. As of 14 October 2026, Anthropic says models released after Claude Opus 4.6 do not support setting it, and Google says Gemini 4 and Gemini 3.6 and later models ignore custom values. Where the setting remains, even its lowest value does not remove the variation, and Anthropic’s glossary says that “even with temperature set to 0, the results will not be fully deterministic”, meaning the same request will not always give the same result. So the variation sits outside any store owner’s control.

Live web search changes what the assistant reads

Live search means the assistant can read different pages each time you ask. OpenAI says ChatGPT “will automatically search the web if your question might benefit from information on the web”. Anthropic says Claude “searches the web when it helps”, and Perplexity says it searches “the internet in real-time” when you ask a question. When the assistant searches, its answer is written from the pages that search returned at that moment, and a search on another day can return different pages. Nathan Gotch counts retrieval changes among the shifts outside your control in AI SEO For Dummies.

Google adds another layer. Its Search Central guide says its two AI features in Search, AI Overviews and AI Mode, “may use different models and techniques, so the set of responses and links they show will vary”. That sentence compares the two features with each other, so a store linked in one feature need not be linked in the other for the same question.

Memory and personal settings change who it is answering

ChatGPT, Claude, Gemini and Perplexity can remember the person asking, so two people typing the same question can get answers shaped by different stored details. Memory, as a product feature, lets the assistant keep details from earlier chats and use them in later ones. OpenAI says ChatGPT “can also use information from your past chats to make future conversations more helpful”, and custom instructions, the standing notes a user gives ChatGPT, “are applied immediately to all chats”.

On Claude’s Free, Pro and Max plans, memory is on by default, according to Anthropic’s help page on Claude’s memory. Google says Gemini can remember “specific facts, interests, and preferences across conversations”, depending on eligibility and settings, and Perplexity says it “will call on memory and previous searches depending on the question”.

An owner who has spent months asking ChatGPT about their own products is asking from an account that already holds those chats. With chat history switched on, ChatGPT can draw on them, and a first-time shopper’s account has no such history.

Location, wording and model updates add more

Where you ask from counts. Google says that for prompts where place matters, Gemini by default uses “the general area from your IP address or the Home or Work addresses in your Google Account to give relevant responses”. Variation by place is not new. Google’s standard search already uses “your location and settings” alongside the query words and page quality.

The exact words you use count too. Google’s Gemini documentation says “using different words or phrasing in your prompts often yields different responses from the model even though they all mean the same thing”, a prompt being the question or instruction you type.

Then the models themselves change. A model version is a named, dated release of a model, and each company publishes new ones. OpenAI tells developers that “even different snapshots of models within the same family could produce different results”, a snapshot being one dated version, and Google says its “latest” model names get “hot-swapped with every new release” of a model. For a developer using one of Google’s “latest” names, the model answering this month need not be the one that answered last month.

Shopping results are picked for the shopper

Shopping features add the shopper’s own context on purpose. OpenAI says a product appears in ChatGPT’s shopping carousel “when ChatGPT perceives it to be relevant to your intent”, and that it considers “your query and context (such as Memory or Custom instructions)”. Shopify says a store selling through ChatGPT “must sell to customers in the United States”. When Perplexity launched shopping for “all US Perplexity users”, it said Perplexity “remembers your past searches and learns your patterns”. So the products one person sees are not a list every shopper sees.

What one answer can and cannot tell you

In one study, ChatGPT and Google’s AI almost never gave the same list of brands twice. Rand Fishkin of SparkToro and Patrick O’Donnell of Gumshoe.ai, an AI tracking company, had 600 volunteers run 12 brand and product prompts through ChatGPT, Claude and Google’s AI a combined 2,961 times, with each prompt run 60 to 100 times. They report under a 1 in 100 chance that ChatGPT or Google’s AI, asked 100 times, gives the same list of brands in any two responses. The study used US tools, collected data in November and December 2025 and has not been peer reviewed, and the authors say newer models may differ. So its figure describes those 12 prompts.

Gotch also separates a check from a test, writing that “a quick spot-check is fine for curiosity” and that an experiment is “a structured comparison”. A single answer that names your store just shows the store can be named. One that leaves it out shows it can be missed. We made the same point in our July 2026 article on measuring AI search visibility, where “the trend across a fixed set of prompts matters more than any one result.”

The official reports don’t provide a clear count of AI answers. Google says sites appearing in its AI features “are included in the overall search traffic in Search Console”, inside the Performance report, so that traffic is counted with ordinary search visits. Microsoft added an AI Performance report to Bing Webmaster Tools in public preview on 10 February 2026, showing how often a site’s pages are cited in Copilot and Bing’s AI summaries, and says “the data shown represents a sample of overall citation activity”. So your own count of a few fixed questions is the place to start.

How to read the pattern yourself

You can get a first reading in a week without any tool. The routine has four parts:

  • Pick three questions a buyer would type before choosing a store like yours. For an outdoor gear store (an invented example), one might be “Where can I buy a lightweight hiking tent in Australia?”
  • Keep the wording identical every time, because Google often gives a different reply for different wording.
  • Ask the same assistant all three questions on three different days.
  • Each time, write down whether your store is named and, if not, who is.

That gives you nine answers. Gotch tells you to “Record 1 if your brand appears” and 0 if it does not, then divide the answers that mention the brand by the total checked. Named in six of nine answers is a pattern. Being named in one of nine is a different pattern, and it tells you more than any single answer.

Worked example grid with invented results: three buyer questions asked of the same assistant on three days, scored 1 if the store is named and 0 if not. The store is named in 6 of 9 answers.

For a measurement, Fishkin recommends running each prompt 60 to 100 times as a usual minimum. Your nine answers show what your account sees, which may differ from what a first-time shopper in another city sees.

Why our report asks fifty questions

Everything above points the same way: one answer proves little, and a pattern across a fixed set of questions tells you more. Our free AI Search report is built on that. It asks fifty buyer questions of five AI tools and shows which questions name your store and which name a competitor instead. Asking the same questions again later is the last of our four steps, and it is what shows whether anything changed.

References

Nicholas Duell Avatar
Nicholas Duell
I run Duelling Pixels, a Melbourne studio that designs, develops, and looks after websites for businesses, and founder at Barking Owl AI, where we measure how AI models describe and recommend those businesses. The two jobs have become one. A website now has to load fast, rank in Google and AI chat interfaces, and be readable by the crawlers that feed AI answers, as well as by the person who finally clicks through to check. Most of my work is SEO in its wider sense, and delivering a website that has impact for the business. Site speed, technical SEO, the structured data that tells a machine what a business is, and the off-site mentions that decide whether an AI names it at all. I make those calls alongside the owner and stay accountable for them once the plan meets the trading week.

Related Articles

Recommended for you

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses cookies for analytics and advertising. Accept them all, decline them all, or choose which ones you allow. Learn more in our privacy policy.