Thinking harder does not lead to better choices: increased reasoning effort raises LLM shopping agents' confidence rather than accuracy.
Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents
Researchers tested ten frontier models on three product-selection benchmarks built on WebShop, Amazon's SQID and Wayfair's WANDS platforms, and found that increasing reasoning effort did not statistically significantly improve the purchase quality of LLM shopping agents, but did statistically significantly raise their self-reported confidence; the paper says the extra thinking buys confidence rather than correctness. These agents process multimodal product information containing both visual and textual content and represent consumers in recommending purchases. The paper also measured model calibration, i.e., how well self-reported confidence matches the actual accuracy rate, and found that most models were poorly calibrated at all reasoning effort levels. Errors were highly correlated across models: when one model erred on a task, several other models often erred on the same task; on tasks where multiple models erred, a large number of models selected the same wrong product.