Key takeaways
- Visa tested 22 AI models from 12 AI labs on 15 shopping tasks, like choosing a TV or a laptop.
- Overall, the models failed 42.8% of the test runs.
- Models that uncovered the most hidden customer needs passed 70.2% of the time. Those that uncovered the fewest passed 44.4%.
- In 54% of failed runs, the AI still bought something.
- Shopify stores face the same problem with vague searches. Baymard found 43% of ecommerce sites have issues with use-case searches like "wedding gift".
What is an AI shopping assistant?
An AI shopping assistant is software that helps a person choose a product through conversation. (We cover the basics in more detail in What is an AI shopping assistant?) Instead of filtering and scrolling, the shopper says what they want in normal words, and the assistant asks, explains and recommends. Some assistants work for the shopper across many stores (an "AI shopping agent" in agentic commerce). Others work on a single store and help its visitors, like a salesperson on the shop floor.
Both have the same core job. Find out what the person actually needs, then point them to the right product, or say honestly that nothing fits.
How did Visa's AI shopping test work?
Visa built a benchmark with 15 shopping scenarios, covering products like a TV, a laptop and a personal care appliance. In each one, an AI model played the shopping assistant for a customer.
The customer did not give every detail up front. In the TV scenario, for example, the customer asks for help choosing a TV and mentions a brand a relative recommended, but leaves out other needs such as the budget, support for a certain video format and gaming features. The AI had to find those out by talking to the customer. Then it had to make a decision: buy the product that fits, or buy nothing if nothing on offer fits.
A run only passed if the AI found every requirement and made the right buy or no-buy call. Each scenario was run three times per model. According to The Next Web's report, Visa did not name the models in its statement, and the coverage does not say who played the customer or whether purchases were real.
What did the test find?
The headline number is that the models passed 57.2% of runs and failed 42.8%. The spread between models was big: the best passed 82.2% of the time, the weakest 28.9%.
The more useful finding is why they failed.
Swipe to see the full table.
| Group of models | Pass rate |
|---|---|
| 5 models that uncovered the most hidden needs | 70.2% |
| 5 models that uncovered the fewest hidden needs | 44.4% |
| 5 most expensive models | 67.1% |
| 5 cheapest models | 50.2% |
Asking good questions made a bigger difference than price. And when the models failed, they often failed in the worst way for a shopper: in 54% of failed runs, the AI still bought something, either a product that didn't meet the needs or one chosen before it knew all of them.
Consistency was another problem. Only 34.2% of scenarios were passed in all three runs. Visa's global head of growth, Rubail Birwadker, said that AI agents could power the next era of commerce "but trust is key to making it work." Visa plans to add more product categories and extend testing to checkout, payment and ordering.
Why this matters for Shopify stores
You might think this is a story about ChatGPT-style agents and not about your store. I see it the other way around.
Before Talkom, I spent 11 years in enterprise site search. The pattern in Visa's test is the same one I saw in search logs for years. People rarely say everything they need. They type a word or two and expect the store to figure out the rest.
Baymard Institute's research on search query types shows how often stores struggle with exactly this. 43% of ecommerce sites have issues with use-case searches like "spring jacket" or "wedding gift". 39% have issues with feature searches like "leather jacket" or "$100-$200 backpacks". Those are the searches where the shopper knows the situation, not the product.
A classic search box doesn't ask anything back. It returns a list. A shopper who searched "wedding gift" and got 200 results is in the same spot as the TV buyer in Visa's test, just without anyone asking the budget.
At Talkom, about 70% of the conversations shoppers start are about products, not orders or shipping. People want help choosing. If your store only answers support questions, it misses most of what shoppers actually ask.
What can you do this week?
- Search your own store like a vague shopper. Try a use-case search ("gift for dad", "something for dry skin") and a feature search ("waterproof", "under $50"). Note what comes back.
- Read your search and chat logs. Look for searches with zero results and for questions that start with "which" or "what's the difference". Those are shoppers asking for advice.
- Write down the 3-5 questions a good salesperson would ask for your top categories. Budget, use, size or skin type are common ones. Make sure your product data actually contains those answers.
- Put those answers on the product page. Size, compatibility, who it's for and who it's not for. AI agents read product pages too, and they can only recommend what they can understand.
- Check your assistant or chat for one thing. Does it ever ask a question back when the request is vague, or does it always reply with a product list? If you're comparing tools, our guide on how to choose an AI shopping assistant has more questions to test on your own store.
How we handle this at Talkom
Talkom is an AI shopping assistant built natively for Shopify. It helps shoppers find, understand and choose the right products 24/7. Shoppers get suggested questions right on product pages with Ask About Products, and can search in plain language with AI Search Mode instead of a keyword search box. Visa's test is a good reminder of what we care about most: understanding the shopper before recommending anything. Across Talkom conversations, 36% lead to an add-to-cart and 15% to an order. You can see how it works on talkom.ai or install it from the Shopify App Store.
FAQ
What did Visa's AI shopping test measure?
Visa's test measured whether AI models acting as shopping assistants could find out what a customer really needed through conversation and then make the right call to buy a product or buy nothing. A test only passed if the AI found every requirement and made the correct buy or no-buy decision.
How many AI models did Visa test, and how did they do?
Visa tested 22 AI models from 12 AI labs on 15 shopping tasks, running each task three times. The models passed 57.2% of runs and failed 42.8%. The best model passed 82.2% of the time and the weakest 28.9%.
Why did AI shopping assistants fail in Visa's test?
Most failures came from not finding out what the customer actually needed. The five models that uncovered the most hidden needs passed 70.2% of the time, while the five that uncovered the fewest passed 44.4%. In 54% of failed runs, the AI still bought something.
Were real shoppers or real purchases used in Visa's test?
The published coverage does not say who played the customer or whether purchases were real. Visa said it plans to extend the test to checkout, payment and ordering in later rounds.
What does Visa's test mean for a Shopify store?
It shows that recommending the right product depends on understanding the shopper first. On a Shopify store, the same applies to site search and chat: if they never ask a follow-up question or understand use-case searches, shoppers with vague needs get long result lists instead of the right product.
Sources
- The Next Web, “Visa tested 22 AI models as shoppers, and they failed 42.8% of the time”, November 5, 2026
- Small Business Trends, “Visa Unveils AI Shopping Benchmark; 43% of Tests Fail”, November 2026
- Baymard Institute, “Ecommerce search query types”, last updated April 29, 2026
About the author
David Kis is the founder and CEO of Talkom, an AI shopping assistant built natively for Shopify. He spent 11 years in enterprise site search before founding the company. Follow David on LinkedIn.