AI shopping agents can't agree on a single best product

Three models, one catalogue, three different purchase recommendations. That is the finding Search Engine Journal reports from an experiment that pointed ChatGPT, Gemini and a third AI model at the same Shopify store and asked each to pick the best product to buy. No overlap. Each model reasoned its way to a different winner, using its own logic about what "best" meant, with no visibility into why.

This is the detail that should unsettle anyone running paid search or retail media budgets on the assumption that AI shopping agents will converge on something like a stable ranking, the way Google's organic results do. They will not. Google's algorithm is proprietary and frequently inscrutable, but it is at least one system, applied consistently, that SEOs have spent two decades reverse-engineering through rank trackers, SERP monitoring and leaked documentation. An AI shopping agent is not one system. It is a language model making a judgment call, informed by whatever subset of product data it retrieved, weighted by training priorities nobody outside the lab can audit, and liable to change its answer on the next query even against the identical catalogue.

The monitoring problem has no instrument yet

Search Engine Journal's framing is the sharpest part of the piece: Google rankings are a black box a store can still watch. Dashboards exist. Rank position is a number you can chart weekly. The AI shopping decision layer offers no equivalent. There is no "position one" for a ChatGPT shopping answer, because the answer is generated fresh each time from a blend of retrieval, model weights and conversational context that a retailer cannot see and cannot query directly. A brand can run the same prompt twice and get different reasoning both times.

That asymmetry matters more than it sounds. SEO built an entire measurement industry on the premise that visibility is trackable, even when the underlying algorithm is secret. Rank trackers work because Google's output is deterministic enough, for a given query and location, to be sampled and charted. AI shopping recommendations are not deterministic in that way. Three models disagreeing on the same catalogue is not a glitch to be patched. It is the structural consequence of each model running its own retrieval and reasoning pipeline, with its own notion of what counts as "best" (cheapest, most reviewed, best-described, most recently updated product copy), and no shared standard for resolving ties.

Why financial services and industrial brands should read this as more than a retail story

The Shopify experiment is about e-commerce, but the mechanism generalises to any sector where buyers are starting to ask AI tools to recommend a provider, product or policy framework rather than searching for one. A bank's wealth-management product, an industrial group's turbine maintenance contract, a multilateral's recommended reporting standard: all of these are now candidates for the same kind of single-shot AI judgment call that picked three different "best" products from one Shopify catalogue. If three mainstream consumer models can't agree on a toaster, there is no reason to expect convergence when the query is "which ESG reporting framework should I use" or "which asset manager has the strongest ten-year track record in emerging-market debt."

For financial services and industrial marketers, the lesson is that optimising for a single model's preferences is a wasted exercise. A brand that tunes its content and structured data to whatever ChatGPT currently seems to reward is optimising for one black box among several, each with a different and shifting internal logic. The practical response is not chasing a ranking signal that does not exist in any stable form. It is making the underlying facts, i.e. certifications, pricing, performance data, third-party validation, so unambiguous and well-structured across every surface that any model's retrieval step, regardless of its idiosyncrasies, lands on the same correct answer about who you are and what you offer.

What this means for the measurement budget

Search teams that have spent the past year building AI-visibility dashboards modelled on rank-tracking tools should treat this finding as a warning against false precision. A citation count in one model's answers today says little about tomorrow's answer, or about what a second or third model would say to the identical prompt. The sensible response is portfolio thinking: track presence across multiple models and multiple prompt phrasings, not a single score chasing a single "position." Expect divergence as the normal state, not the anomaly. The brands that adapt fastest will be the ones that stop asking "are we ranking" and start asking "are we legible," in the same way, everywhere, to a judge that keeps changing its mind.