NEW YORK — Artificial intelligence models, including OpenAI's ChatGPT and Google's Gemini, have recently been put to the test in a series of experiments designed to gauge their ability to analyze and recommend consumer products, specifically smartphones. The results, while showcasing the evolving sophistication of AI in product analysis, have also highlighted notable inconsistencies and raised questions about the trustworthiness of machine-generated advice for consumers.
One such experiment, reported by Android Authority on October 24, 2026, involved prompting ChatGPT and Gemini to debate which smartphone should be considered the best overall. TechKomo, also reporting on October 24, 2026, detailed that this "AI Debate Best Smartphone" experiment pitted the models against each other, with them "trading brutal arguments" while comparing the iPhone 16 Pro Max, Samsung Galaxy S24 Ultra, and Google Pixel 9 Pro XL. TechKomo alleged that the experiment concluded with a "surprising split consensus that challenges standard flagship buying advice," suggesting that AI logic might diverge from traditional human reviewer perspectives.
However, the debate was not without its criticisms. Android Authority alleged that Gemini's reasoning in the smartphone debate "sounded heavily influenced by vague marketing rather than concrete features and hardware." This observation points to a potential challenge in how AI models process and present information, possibly prioritizing promotional language over detailed technical specifications.
Beyond the specific smartphone debate, broader studies have scrutinized the consistency and accuracy of AI recommendations. A comprehensive analysis by marketing firm Suff Digital, published September 24, 2026, and reported on September 30, 2026, involved asking 300 "What is the best…?" questions 10 times each on both ChatGPT and Gemini, generating a substantial 6,000 answers. The Suff Digital analysis alleged a significant degree of inconsistency: ChatGPT changed its top pick in 56.3% of back-to-back answers to identical questions, while Gemini exhibited an even higher variability, changing its top pick in 82.7% of such instances. The firm presented its results as "one controlled comparison rather than a universal ranking of the two services," acknowledging the specific parameters of their test.
Further compounding concerns about AI reliability, a Product.ai study, published September 22, 2026, and reported on September 30, 2026, tested 220 shopping questions five times each across free and paid configurations of ChatGPT, Gemini, Claude, and Perplexity. This study alleged that a striking 86% of the questions produced at least one confirmed factual conflict among the AI models' responses. The Product.ai study also alleged that Gemini contradicted one of its own earlier answers on 29% of questions in its free configuration and 27% in its paid version, compared with 22% and 18% for ChatGPT. Moreover, Product.ai alleged at least one "costly error" on 56% of questions for Gemini free and 54% for paid, versus 19% and 17% for ChatGPT. It is important to note that the Product.ai tests used provider APIs rather than the consumer ChatGPT and Gemini interfaces, which could influence the results.
These findings emerge as the role of AI in consumer decisions is rapidly expanding. An unnamed outlet reporting on the Suff Digital and Product.ai studies forecasted that the reliability of AI shopping advice is "becoming more consequential as assistants move closer to purchases." This suggests a growing need for AI models to demonstrate greater consistency and factual accuracy as they become more integrated into the purchasing journey.
Several aspects of the AI-driven smartphone debate remain unconfirmed or were not specified in the experiments. Android Authority noted that the smartphone experiment did not guide the AI models on a specific year, price range, or operating system for their choices. While a market was not explicitly specified, it was "figured that the choices would lean towards those available to US consumers." TechKomo, for its part, posed the fundamental question of whether "machine logic truly evaluate personal taste better than seasoned human reviewers," highlighting the subjective nature of product preferences that AI may struggle to quantify.
In a broader context of AI capabilities, CNET on September 25, 2026, published a comparison of ChatGPT and Gemini's features and multimodal capabilities, including a "camera test." While not directly related to the product debate, such comparisons underscore the ongoing efforts to benchmark and understand the strengths and weaknesses of these advanced AI systems across various tasks.
The collective reporting indicates that while AI models like ChatGPT and Gemini can engage in sophisticated product analysis and debate, their current reliability and consistency in offering definitive "best" recommendations or factual information still present significant challenges. As these technologies continue to evolve, the balance between their analytical power and the need for trustworthy, consistent advice will be a critical area of focus for developers and consumers alike.